Send all optimization algorithms the same model when comparing performance!

While performing diagnostics on Riddhi’s formulations for the environmental economics “Lake Problem”, we had had slightly different models for communication with Borg and the MOEAFramework algorithms. The only difference in the code was the part where the Borg/MOEA connection was made, so we did not think it would impact performance in any meaningful way, especially since command line calls outside of optimization with identical decision variables showed essentially identical results. The only difference was the truncation of a few digits way past the specified epsilon for the Borg model.  In spite of these minute differences, our preliminary results were incredibly disconcerting. Borg had a reference set contribution of zero, and when the combined set found by Borg was compared to that found by a Random search, the front found by the Random search dominated that found by Borg, which is not possible especially on such a simple problem as the 2 objective deterministic formulation with which we were working!  After many attempts to determine the reason for our bizarre results, Jon pointed out that we could probably call the model we had written for the MOEAFramework in the command line when we optimized with Borg. When I tried this, the results made sense!  Borg had a reference set contribution of 1.0 for the 2 objective deterministic formulation and its Pareto approximate front dominated that of the Random search.  Further investigation with decision variables returned by optimization is leading us to believe the format in which the two models read variables may differ slightly, and we are looking into that hypothesis.

For anyone else comparing performance of multi-objective evolutionary algorithms, make sure you send the exact same model to all algorithms!  Otherwise, the comparison will not be fair.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s