MOEA diagnostics for a simple test case (Part 1/3)


In this tutorial you will learn how to do  the following:

ūüĎď Control¬†maps to evaluate the MOEA‚Äôs sensitivity to different combinations of crossover, mating and mutation parameters, and to test how quickly ¬†MOEAs can reach an acceptable approximation to the Pareto front.

ūüĎstAttainment plots¬†to evaluate the reliability of each MOEA (how likely can MOEAs attain high levels of performance) and how effectively they can approximate the best known Pareto front.

ūüĎď Reference set contribution to test the percent contribution of each MOEA when calculating the approximation to the best known Pareto front.

We will use three metrics: generational distance, epsilon-indicator and hypervolume to measure proximity, consitency and diversity, respectively (read more).

Prerequisites:

‚ústCygwin¬†terminal

‚úď ¬†Complete the basic Borg tutorial¬†part 1¬† and¬†part 2

‚ústThe MOEAFramework all in one executable available at¬†moeaframework.org

‚úď Java

Objective:

Perform a diagnostic study with an MOEAFramework algorithm (for this tutorial I selected NSGAII) and with the Borg MOEA  using the 3 objective DTLZ2 problem.


1) Parameter description files

Please refer to the parameter description files for a full explanation.  You can copy the corresponding parameters for NSGAII and Borg into a text editor and save it as  NSGAII_Params.txt and Borg_Params.txt.

2) Generating Latin Hypercube Samples

You will need a copy of the Borg_Params.txt, NSGAII_Params.txt and the MOEAFramework executable in your working directory.   From your terminal, navigate your working directory, in my case its something like this:


jz583@jz583-pc ~
$ cd C:

jz583@jz583-pc /cygdrive/c
$ cd Users/jz583/Desktop

jz583@jz583-pc /cygdrive/c/Users/jz583/Desktop
$ mkdir DTLZ2_3_example

jz583@jz583-pc /cygdrive/c/Users/jz583/Desktop
$ cd DTLZ2_3_example/

jz583@jz583-pc /cygdrive/c/Users/jz583/Desktop/DTLZ2_3_example
$ ls
MOEAD_Params.txt MOEAFramework-2.0-Executable.jar NSGAII_Params.txt


In lines 2 and 5,¬†I access my Desktop. ¬†In line 8,¬†I create¬†a directory called DTLZ2_3_example, where I will store all the files generated in this example. ¬†After I copy the required files into the DTLZ2_3_example folder, I use the cd or “change directory” command in line 11 to navigate the folder. ¬†I finally use the ¬†ls command in line 14 to list the folder’s contents and¬†verify that I have the correct files in it, and I do ;).

Now you can type the following command:


java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.SampleGenerator --method latin --numberOfSamples 100 --parameterFile NSGAII_Params.txt --output NSGAII_Latin

The previous command generates 100 latin hypercube samples (LHS) for the NSGAII algorithm. ¬†After you run the command,¬†simply press the¬†‚ÜĎ key¬†and ¬†replace NSGAII with Borg in the –parameterFile flag ¬†and in the –output flag.

Afterwards, there should be two new files in your directory, one named NSGAII_Latin and the other one named Borg_Latin.  Open the NSGAII_Latin file and verify that it has 100 rows, one row per parameter combination, and 6 columns, one column per parameter. Likewise, open the Borg_Latin file,  it should have 100 rows and 21 columns.

3) Running each parameter sample for multiple random seeds

The main difference between evaluating MOEAFramework algorithms and evaluating Borg, lies in this step.  The rest of the analysis is supported by the MOEAFramework and the commands will be the same for Borg and for NSGAII.

For NSGAII

The following command will evaluate the DTLZ2_3 problem for 100 parameterizations replicated for 15 seeds each using NSGAII.  A numerical precision of 0.01 is specified for each objective.   Since NSGAII is a point-dominance algorithm, we must specify an epsilon precision for consistency when comparing with Borg which is an epsilon-dominance algorithm.


for i in {1..15}; do java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.Evaluator --parameterFile NSGAII_Params.txt --input NSGAII_Latin --seed $i --problem DTLZ2_3 --algorithm NSGAII --epsilon 0.01,0.01,0.01 --output NSGAII_DTLZ2_3_$i.set; done

For Borg

Before running the following code, you may want to complete the borg basics tutorial.



for i in {1..15}; do ./borg.exe -v 12 -o 3 -l 0,0,0,0,0,0,0,0,0,0,0,0 -u 1,1,1,1,1,1,1,1,1,1,1,1 -e 0.01,0.01,0.01 -p Borg_params.txt -i Borg_Latin -s $i -f Borg_DTLZ2_3_$i.set python dtlz2.py; done

The -v flag indicates the number of variables, -o number of objectives, the -l and -u flags are the lower and upper bounds for the decision variables, respectively, -e is the epsilon precision, -p parameter description file, -i latin hypercube samples, -s indicates the random seed number and -f specifies the name of the output file.

4) Selecting objectives

The .set files generated in step 3 contain 15 columns.  The first 12 columns are the decision variables, and the last 3 columns are the objectives, we are interested on the latter.  To select the objectives, run the following command:


for i in {1..15}; do awk 'Begin{FS=" "};/^#/{print$0};/[-]?[0-9]/{printf("%s %s %s\n", $13,$14,$15)}' NSGAII_DTLZ2_3_$i.set > NSGAII_DTLZ2_3_$i.obj; done

To select the objectives for Borg, replace all the instances of NSGAII in the previous command with Borg.

5) Generating a reference set for each parameterization across all random seeds

The following command merges the results across the 15 random seeds for each parameterization:


java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.ResultFileSeedMerger -d 3 -e 0.01,0.01,0.01 -o Borg_DTLZ2_3_LHS.reference Borg*.obj

Once you run the previous command, press the ‚ÜĎ key and¬†replace all instances of Borg with NSGAII. ¬† The¬†files generated should¬†have 100 reference sets, one per parameterization, separated by a #. ¬†Keep track of these files, we will use them later in part 2 of this tutorial.

6) Generate reference set for each MOEA (across all parameters and seeds)

The following command generates a reference set for Borg across its 100 LHS parameterizations and 15 seeds.


java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.ResultFileMerger -d 3 -e 0.01,0.01,0.01 -o Borg_DTLZ2_3.reference Borg*.obj

Run the same command for NSGAII.

Once I¬†generated Borg_DTLZ2_3.reference and NSGAII_DTLZ2_3.reference files, I inserted a # at the end of each file. ¬†The reason is that I¬†combine these two sets in the following step using ¬†the MOEAFramework’s result file merger, which uses the # as an indicator that the file has reached to an end.

7) Generate a combined reference set across all MOEAs tested

The DTLZ2_3 problem has a well-known analytical solution but for learning purposes we will pretend that we don’t know this solution.¬† In real world applications, an analytical solution is hardly known; therefore,¬†the best¬†approximation to the Pareto front¬†is¬†generated across all algorithms, all seeds and all parameterizations. ¬†¬†In this case, we will generate a combined set merging NSGAII and Borg. ¬†Note that in the following command I used the Borg_DTLZ2_3.reference and NSGAII_DTLZ2_3.reference because these reference sets are already merged across their seeds and parameters. ¬†What we are doing now is merging across algorithms:


java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.ResultFileMerger --problem DTLZ2_3 --output DTLZ2_3_combined.pf Borg_DTLZ2_3.reference NSGAII_DTLZ2_3.reference

At this point, we generated the DTLZ2_3_combined.pf, which represents our best known Pareto front approximation, at least for this exercise. Now we are ready to calculate metrics!!

Advertisements

5 thoughts on “MOEA diagnostics for a simple test case (Part 1/3)

  1. Pingback: MOEA diagnostics for a simple test case (Part 2/3) | Water Programming: A Collaborative Research Blog

  2. Pingback: MOEA diagnostics for a simple test case (Part 3/3) | Water Programming: A Collaborative Research Blog

  3. If people are having trouble with ‘unable to parse solution, ignoring remaining entries in the file’ in step 5, I found that this could be prevented by removing white spaces/blank lines in the .obj files. Here is the alternative command for step 4 that does this:

    for i in {1..15}; do awk ‘Begin{FS=” “};/^#/{print$0};/^[0-9]/{printf(“%s %s %s\n”, $13,$14,$15)}’ NSGAII_DTLZ2_3_$i.set > NSGAII_DTLZ2_3_$i.obj; sed -i ‘/^[[:space:]]*$/d’ Borg_DTLZ2_3_$i.obj; done

  4. I’m going through this again to make sure I understand everything that I did.
    The first time I had already gone though the Borg basics tutorial, but this time I was using “fresh” files. In step 3, where it says you might want to go though the Borg tutorial first, if someone decides to skip it, make sure that in the dtlz2.py file you’re using lines 4 and 5 are set to nvars = 12 and nobjs = 3. Otherwise, Borg won’t be able to run.

  5. Pingback: Water Programming Blog Guide (3) – Water Programming: A Collaborative Research Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s