Runtime metrics for MOEAFramework algorithms, extracting metadata from Borg runtime, and handling infinities

I have been working with runtime metrics for a variety of algorithms on the “Lake Problem” for max NFE of only 25,000 at an output frequency of 1,000. Here are a few things I learned along the way for which the group does not seem to have consolidated resources.

First, most of our recent runtime resources have been focused on Borg, so although it is extremely easy to get runtime metrics for MOEAFramework algorithms (in fact it’s one of the site’s examples), I was initially at a loss for where to look. What I ended up doing was copying the code from the 3rd example found here: http://moeaframework.org/Example3.java and making the following changes.

At the top of the code, I added the following line

import java.io.File;

just below the line that read:

import java.io.IOException;

First, clearly change the name in .withProblem to the name of the problem you defined (See examples 4 and 5 and the manual for more info). You can specify your desired output frequency in .withFrequency and include runtime info beyond that shown in the example. It only collects Elapsed Time and Generational Distance.  Because my problem was not built into the framework, I had to specify a reference set I had already made.  That is why I had to include the import java.io.File line.  Until I added it, I received compilation errors.

My section for setting up the instrumenter looks like this:

// setup the instrumenter to record metrics
Instrumenter instrumenter = new Instrumenter()
.withReferenceSet(new File("./5obj_2const_stoch/myLake5Obj2ConstStoch.reference"))
.withFrequency(1000)
.attachElapsedTimeCollector()
.attachGenerationalDistanceCollector()
.attachHypervolumeCollector()
.attachAdditiveEpsilonIndicatorCollector();

You will also need to specify the desired problem, algorithm and max NFE for the Executor. Further if you are using an epsilon dominance algorithm, you can specify the epsilons with the following line:

.withEpsilon(0.01,0.01)

I had a two objective problem with epsilons of 0.01 for both objectives.

Finally, I became frustrated with the runtime printing format included in the example as it didn’t have a consistent field separator when I ran it. This may not have actually been important, but below is what I used:

System.out.println("NFE"+"\t"+"Elapsed Time"+"\t"+
"Generational Distance"+"\t"+"Hypervolume"+"\t"+
"Additive Epsilon Indicator");

for(int ii=0; ii<accumulator.size("NFE"); ii++){
System.out.println(accumulator.get("NFE",ii)+"\t"+
accumulator.get("Elapsed Time",ii)+"\t"+
accumulator.get("GenerationalDistance",ii)+"\t"+
accumulator.get("Hypervolume",ii)+"\t"+
accumulator.get("AdditiveEpsilonIndicator",ii));
}

You could also use the code included in Matt’s blog post here: https://waterprogramming.wordpress.com/2013/02/05/matplotlib-part-i-borg-runtime-metrics-plots/ although I haven’t actually tried it yet.

Once this was done, I was excited to plot the results, but that is where I ran into another glitch.  At the extremely small intervals, the values for Additive Epsilon Indicator and Generational Distance were Infinity as the algorithms had yet to find any feasible solutions.  This was easily solved by the following command.  Thank you Jon for recommending it!

sed 's/Infinity/-9999.0/g' all_of_my_files.txt >> write_to_file.txt

It goes through the first file specified and replaces all of the “Infinity” values with -9999.0 before writing the output to the file after the sideways carrots.  This made it much easier to import the data to Matlab where I set the -9999.0 values to NaNs.  Since I had 300 files, I put this in a bash script that looped through my runtime files for all algorithms and seeds.

Finally, I wanted to plot Borg operator probabilities, but now that Borg is not in the framework, its initial output files are not very pretty.  There is a way to extract the data if one consults section 10.2.2 of the MOEAFramework manual.  Then, you get nice files that look kind of like the one’s in the blog post I referenced by Matt earlier.  This made it much easier to plot the data.  It also let me repeat the Python exercise in that post with my own files, which was fun.

March 27, 2015 edit: I just found this video, which provides an overview of calculating runtime metrics with the MOEAFramework.  It also includes an example of reading a seed from the command line at the beginning of the java code.

Advertisements

3 thoughts on “Runtime metrics for MOEAFramework algorithms, extracting metadata from Borg runtime, and handling infinities

  1. Hello Tori,

    I encountered a strange error when trying to use the analyzer. I get org.moeaframework.core.FrameworkException: java.lang.reflect.InvocationTargetException

    when trying to use the knapsack problem. I am able to use the executor and instrumentor but not the analyzer. Any ideas ? I did not stick my whole code here as it would be too long as a comment. If you want I can email you my code ?

    Many Thanks
    Ayush

    • Hi Ayush,

      Could you please include the full error message? It should include more lines similar to:
      at org.moeaframework.analysis.sensitivity.Analyzer (line 273)

      It would also be helpful to see the code. Please create a new ticket at http://sourceforge.net/p/moeaframework/bugs/ and you can click “Add attachments” to add files.

      Thanks,
      Dave

  2. Pingback: Water Programming Blog Guide (Part I) – Water Programming: A Collaborative Research Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s