Random Seed Analysis for the Borg MOEA using DTLZ2, 3 objective instance

Why do we want to perform a Random Seed Analysis?

A random seed analysis will provide insight on how sensitive an algorithm is to its initial populations. Keep in mind that MOEAs are stochastic search tools that vary in their initial configuration and their probabilistic search operators and it is likely that two independent algorithmic runs yield different results.  We account  for this variability by running the algorithm for multiple seeds and then averaging them, or generating a reference set across all of them.  At the end of this exercise, you should be able to perform a random seed analysis  for the Borg MOEA using a simple test case.

Pre-requisites:

The Borg MOEA. For detailed instructions on how to request and compile the Borg MOEA,  read Basic Borg MOEA use.  I strongly suggest completing the DTLZ2 Exercise before performing this tutorial.  It will provide you with background on this test instance and will help you perform the initial setup.

tar –xzf  MOEAFramework-2.5.tar.gz


Follow the instructions in the MOEAFramework User Guide  to complete the setup.

Step 1.  Collect runtime dynamics for multiple seeds from the command line.

From your terminal, navigate the dmh309-serial-borg-moea-eb8f1ca9d713 directory (previously used in the DTLZ2 Exercise)  we’ll refer to this as “the borg directory” for later reference.   These are the files that you should see:

It is possible to run several seeds directly from your terminal with the following command:

for i in {1..10}; do ./borg.exe –v 12 –o 3 –s $i –e 0.01,0.01,0.01 –l 0,0,0,0,0,0,0,0,0,0,0,0 –u 1,1,1,1,1,1,1,1,1,1,1,1 –n 1000 –R dtlz2_3_borg_S$i.runtime –F 10 python dtlz2.py; done


This command is very similar to the one we used in the DTLZ2 exercise except here we specify the  seed flag -s, which denotes the  pseudorandom number.  For instance,  everytime we run  –s 1 , we should always get the same results for that seed number.  In this exercise we want to run 10 different random seeds, so we loop ten times, the $i will take the value of 1, 2 until 10. We also use the runtime flag –R to enable the runtime output, dtlz2_3_borg_$i.runtime is the name of the file in which the runtime output will be printed.  Finally, the  –F flag specifies the output frequency, here we used a frequency of 10.

Once you run the previous command, you should see ten files with the .runtime and .out extension in your borg directory, respectively.  Next, type the following:

mkdir runtime


This command created a directory named runtime.  You can move all your runtime output into this newly created folder with the following command:

mv *.runtime runtime


You should see inside your borg directory, a runtime directory with all your runtime files.  You can also create an output directory and move all your .out files into this folder to keep everything organized.

Step 2.  Select the objectives

Open one of your runtime files, you should see 15 columns. The first 12 column, represent the decision variables and the last three columns are the objectives.  In order to select the last three columns for all the ten runtime files at once.  From your terminal, navigate your runtime directory and run the following command:

for i in {1..10}; do awk ‘Begin {FS=” “}; /^#/  {print $0}; /^[-]?[0-9]/ {printf(“%s %s %s\n”,$13, $14,$15)}’ dtlz2_3_borg_S$i.runtime > dtlz2_3_runtime_S$i.obj; done


In your runtime directory, you should see 10 files with the .obj extension, each of these files should have only three columns. Hold on to these files, we’ll be using them later.

Step 3.  Generate the DTLZ2 3 objective analytical set using the MOEAFramework

The MOEAFramework supports the reference set for any problem with known analytical solution.  For simplicity, place the MOEAFramework-2.0-Executable.jar (or the most recent version) in your borg directory, and execute the following command:

 java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.SetGenerator -b DTLZ2_3 -n 1000 -o DTLZ2_3.pf -e 0.01,0.01,0.01


This command should generate the Pareto Front for the DTLZ2, 3 objective problem, using epsilon values of 0.01 for each objective.  We will use the DTLZ2.pf  for our metric calculations.

Step 4.  Calculate Metrics

For simplicity, copy the MOEAFramework-2.0-Executable.jar (or more recent version) and the DTLZ2.pf file generated in Step 3 into your runtime folder.  From your terminal access your runtime folder and execute the following command:

for i in {1..10}; do java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.ResultFileEvaluator --reference DTLZ2_3.pf --input dtlz2_3_runtime_S$i.obj --problem DTLZ2_3 --output dtlz2_3_borg_S$i.metrics; done


The previous command calculates metrics for each of the ten seeds, comparing it to the DTLZ2 analytical set.  Open one of your metrics files,  you can see that the file has 6 columns, one for each metric.  We usually care about hypervolume (first column), generational distance (second column) and epsilon indicator (fifth column).  Read more in MOEA Performance Metrics.

Subsequently, type the following command to average the metrics:

java -cp MOEAFramework-2.0-Executable.jar org.moeaframework.analysis.sensitivity.SimpleStatistics –mode average –output dtlz2_3_borg.average dtlz2_3_borg_*.metrics


We can now plot hypervolume, generational distance and epsilon indicator.   A python example is provided for plotting the average metrics.  You can also plot  the metrics for the ten seeds to see the traces.

import numpy as np
import matplotlib.pyplot as plt
f, (ax1, ax2, ax3) = plt.subplots(3, sharex=True, sharey=True)
ax1.plot(metrics[:,0]) #Hypervolume plot
ax1.set_ylabel('Hypervolume')
ax1.set_title('Runtime output')
ax2.plot(metrics[:,1]) #Generational distance plot
ax2.set_ylabel('Generational Distance')
ax3.plot(metrics[:,4], color='r') # Epsilon indicator plot
ax3.set_ylabel('Additive $\epsilon$-indicator')