MOEA diagnostics for a simple test case (Part 2/3)

In part 1 of this tutorial we evaluated the DTLZ2 3 objective test problem with 100 parameter samples of NSGAII and Borg replicated for 15 random seeds.  We also generated the best known reference set across all parameter samples for these two algorithms which we named:  DTLZ2_3_combined.pf.    The following metrics will be calculated relative to this reference set.

1) Metric Calculation

Option 1. Calculate average metrics

Before you type the following command, remember to place a #  at the end of your DTLZ2_3_combined.pf file

$ for i in {1..15}; do java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.ResultFileEvaluator --b DTLZ2_3 --i NSGAII_DTLZ2_3_$i.obj --r DTLZ2_3_combined.pf --o NSGAII_DTLZ2_3_$i.metrics; done

Once you run the previous command, replace NSGAII  with Borg  in the –i flag.  In your .metrics files you will see 6 columns, one column per metric.  For this tutorial we will only be interested in the 1st, 2nd and 5th columns, corresponding to hypervolume, generational distance and epsilon-indicator.

The following command averages the metrics across the 15 random seeds:

$ java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.SimpleStatistics --mode average --output NSGAII_DTLZ2_3.average NSGAII_DTLZ2_3_*.metrics

Again, repeat the previous command by pressing the up arrow and replace NSGAII with Borg to obtain Borg’s average metrics.

Option 2.  Calculate reference set metrics for each parameterization.

In part 1 of this tutorial we generated a reference set for each parameter sample, in this step we will calculate the metrics of each of these sets relative to our reference set.

</pre>
 java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.ResultFileEvaluator --b DTLZ2_3 --i NSGAII_DTLZ2_3_LHS.reference --r DTLZ2_3_combined.pf --o NSGAII_DTLZ2_3.metrics

<pre>

2) Calculating reference set hypervolume

Copy the following script into a text editor and save it in your working directory as HypervolumeEval.java


/*Written by Dave Hadka */
import java.io.File;
import java.io.IOException;

import org.moeaframework.core.NondominatedPopulation;
import org.moeaframework.core.PopulationIO;
import org.moeaframework.core.indicator.Hypervolume;
import org.moeaframework.analysis.sensitivity.*;

public class HypervolumeEval {

public static void main(String[] args) throws IOException {
NondominatedPopulation refSet = new NondominatedPopulation(
PopulationIO.readObjectives(new File(args[0])));

System.out.println(new Hypervolume(
new ProblemStub(refSet.get(0).getNumberOfObjectives()), refSet)
.evaluate(refSet));
}

}


Compile it using the following command:

$ javac -cp MOEAFramework-2.0-Executable.jar HypervolumeEval.java

Once you have compiled the HypervolumeEval, you should have a new file on your directory called HypervolumeEval.class.  Now you can compute the reference set hypervolume with the following command:

$ java -cp MOEAFramework-2.0-Executable.jar HypervolumeEval DTLZ2_3_combined.pf

It can take a while to compute hypervolume, you can either stare at your Cygwin terminal until it outputs a value or you can do >  whatever.txt at the end of the previous command and the value will be printed in a separate file.  Either way,  keep track of this value because we will use it later.

Great, we finished calculating metrics.  Now we can generate our control maps, attainment plots and calculate our reference set contribution in part 3 of this tutorial.

Go to → MOEA diagnostics for a simple test case (Part 3/3)

Advertisements

4 thoughts on “MOEA diagnostics for a simple test case (Part 2/3)

  1. Pingback: MOEA diagnostics for a simple test case (Part 3/3) | Water Programming: A Collaborative Research Blog

  2. Thanks for this! A quick note on the calculation of reference set hypervolume. The code above works (the HyperVolumeEval.java). One item is that the new MOEAframework instructions suggest that you should use a different command to compile. Let’s assume you’re in the MOEAFramework-2.6 folder, then the command would be:

    javac -cp “examples;lib/*” .\examples\HyperVolumeEval.java

    And similarly I wanted to share the command that you’d call if you’re just using the code to calculate the hypervolume of any reference set. Assuming you have a file that only has objectives in it, and all objectives are minimized, you type:

    java -cp “examples;lib/*” HyperVolumeEval .\path-to\myrefset.txt

    Hope this helps!

  3. I ran into some trouble when trying to calculate the average metrics at the first step. It was working properly for Borg but not for NSGAII (I was getting a single row of metrics instead of 100). If anyone’s running into the same problem, I figured that the problem with mine was caused by some extra break lines between each of the 100 samples in my NSGAII_DTLZ2_3_$i.obj files. Removing those should fix the issue.

  4. Pingback: Water Programming Blog Guide (3) – Water Programming: A Collaborative Research Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s