We have put together some training on using the Borg MOEA, and it is available here: Part I, Part II. This training goes over how to link one of the external C++ problems to the algorithm, and the focus is on using the algorithm itself, not on how to get a complicated model up and running. So I thought I would informally write some notes on how to get a model like the Lower Rio Grande Valley (LRGV) simulation running with Borg. If this post gets more developed, we may eventually merge it into the documentation. But let’s see how it goes!
This post is a work in progress! I am posting it incrementally in order to help the readers, but I will expand on it in the next few days. Last update: September 18, 2014
For this exercise you’ll need:
1. The LRGV source code. Get it from GitHub, or perhaps you already have a copy of it sitting around.
2. A copy of the Borg MOEA. Request it from Borgmoea.org, or perhaps you already have a copy of it sitting around.
3. Some way of compiling C/C++ code and running it. This can be either:
3a. A computer running Linux
3b. A computer running Cygwin, where you do all the computations on your local computer.
3c. Access to a computing cluster. This is probably the preferred method, because you can run longer jobs, and run the jobs simultaneously. Refer to our intro to using a cluster. Check out our video on submitting jobs to the cluster. Etc. In order to use the cluster you need:
3b.1 Ability to transfer files to the cluster. On all platforms (Mac, PC, Linux) you can use the web-based Globus Connect. Or on Windows you can use a program like WinSCP.
3b.2 Ability to perform interactive commands on the cluster. On Mac or Linux simply open your terminal software! On Windows, there’s a few options. PuTTY is a simple one. I like to use Cygwin, because it mimics a Linux environment on your PC and you can do things like compile code, run python, all as if you were running a Linux box even if you’re not.
So I will be running this example on a Windows PC with WinSCP and Cygwin. I already have a copy of Borg and the LRGV model, but nothing is on my cluster yet.
Connect to the cluster and transfer your files
Some of these instructions are going to be specific to your particular cluster. Since I’m in Colorado I’ll be writing specifically about the University of Colorado system, which I also discuss in this post over on our sister blog and a more recent post on this blog by my colleague Ben Livneh. But in general this should mostly apply to your own system too.
Start WinSCP and make sure you have selected the proper system you want to connect to. At Colorado, we have an Authenticator that will connect us to the system based on a one-time code that is refreshed every second. When WinSCP connects to the system, you’ll see your local files on the left, and the remote files on the right. Drag the folders that you want to copy to the system over to the right hand side. You should have Borg in one folder, and the LRGV code in the other.
In essence, you were given the source code for both Borg and LRGV. These are the fundamental instructions that tell the computer how to run the governing equations of the simulation and the optimization algorithm. For most code designs, though, you shouldn’t need to change the source code to change some parameters of how the research is carried out. For example, you can change the population size of Borg without changing the source code. In the LRGV simulation, you can change the volume of permanent rights without changing the source code, too. Now there’s different ways to arrange your files when you are doing research but here’s one option…
Keep the source code in their own separate folders. Then, when you compile, you can move the executable and project data (for the LRGV this is demand.txt, hydro.txt, etc.) into another folder. There’s several reasons for this:
- The source code may have dozens of files. If everything is in one folder, source code is going to be alongside project files, some of which may be extraneous outputs that you often want to delete. If you delete these outputs, you may inadvertently delete important source code. On Unix, there is no ‘undelete’ so be careful!
- It allows you to maintain multiple copies of the source code if you change something. Granted, source control using GitHub helps you do the same thing. But maybe you made a major change and you want to see what the ‘October 1’ versus the ‘November 1’ version of the code looks like. Keeping separate folders helps you see this.
- Because you don’t have to change source code that often, it allows you to have a ‘canonical’ version of the code that is used in the study. If the folder is called ‘LRGV_2013-10-08’, because I know the last time I changed the source code was back on 10/8. Then, I can move into my ‘project’ folders and start changing other parameters, variables, etc. that do not require re-compilation but allow me to explore what I’d like to explore.
The last task in this session is to connect to the cluster using your ‘terminal’ program.
On PCs with PuTTY, type your hostname in the opening screen. Then, the program will ask you what you want to login ‘as’, so type your username.
Use a ‘terminal’. On Mac or Linux, simply open up the program called ‘Terminal’. On PCs with Cygwin, find the ‘Cygwin Terminal’. Then, for any of these terminal systems, you’ll want to use ssh to connect. The command is something like, ssh [hostname] but consult your own documentation for the specific command. If you are rusty on your cluster commands please consult this post.
Congratulations! You’re connected to the cluster and are able to both transfer files and write commands. Time to…
Compile Borg
Source code by itself will not do anything, because it is a set of instructions written in programming language for your computer. But a computer doesn’t understand programming language — the program needs to be in executable form that the computer can actually use. Compilation is the process of converting source code into executables.
On Linux, the ‘makefile’ is a set of commands that does the compiling for you! It’s really easy. For a lot of programs, typing the command ‘make’ could be enough to compile all the files you need. However, sometimes you need to compile different sets of files, and thats why makefiles often have ‘targets’ that tell the computer the exact groups of files that the compiler needs, and what dependencies exist between the different files. More on makefiles here.
To begin compiling borg, cd into the Borg directory. You’ll notice there is the makefile, and all the source, in this directory. Use more makefile to examine what the makefile looks like. I see several targets in the makefile: all and clean. Clean is designed to delete all the outputs of your compilation. It’s helpful between compilations to make sure that all the code is ‘fresh’ when you use it. The other options are specific to Borg, but the one that I want is all, which is the default. Therefore, to compile borg simply type make. You should see commands that come up on the screen to indicate that the compilation is proceeding successfully! Note, if you need to compile again, or you’ve changed something, ‘make clean’ will clean up the intermediate files.
To take a look at the files in your folder, use the ls command. If you were successful, you should see some files that are highlighted in a different color on your screen, for example we have ‘borg.exe’, ‘dtlz1.exe’, and so forth. The program ‘test.exe’ looks intriguing. To run it, simply type ./test.exe on the command line. The output seems like it worked.
We’ll need to pull out those executable files in a minute, but for the time being leave them in the folder.
Compile the LRGV simulation model
Change into the LRGV directory, and you’ll see two folders. ‘lrgv_bin’ contains the executable, makefile, project data, etc. ‘lrgv_src’ contains the source code. Change into the ‘lrgv_bin’ folder. Notice that here, the makefile has a funky name. It is called ‘MakefileSerial’. The make command for a nonstandard makefile name looks like this: make -f MakefileSerial. Do this command, and you’ll see that the compilation begins. After it’s finished, when you ls the directory notice there is a new file called ‘lrgvForMOEAFramework.’ Voila!
To see the help file for this program, the command is ./lrgvForMOEAFramework -h. That provides the options to you.
If you don’t yet understand all the options, that’s ok. Try the following set of default options to make sure the program is working: ./lrgvForMOEAFramework -m std-io -b AllDecAll -c combined -r 1000. Basically this is saying to run the standard input/output version of the code that is designed to work with Borg; you are running the AllDecAll_control.txt control file, you are running the ‘combined’ calculation mode; and you want to run 1000 Monte Carlo replicates at each function evaluation.
You may think nothing has happened, and I guess that’s true! This is because any program designed for standard i/o to connect with Borg is designed to wait for you (or the algorithm, or the processing code) to input values, and it will output some objective function values and constraint violation values back to you. The cursor is waiting for you, the human, right now, so input some values! Enter them with spaces in between, and when you are done, hit enter.
For the LRGV, the decision variables are in a certain order depending on the model ‘case’ (see the De Novo paper). For case 3, the order is rights, options low, options high, xi (threshold for deciding between low and high option values), alpha2, beta2, alpha, and beta. These values have transformations applied to them. One of these days we will have to write some documentation for the LRGV code but for now take my word for it. Try these random numbers:
0.1 0.3 0.3 0.4 1.4 0.4 1.4 0.4
Hit enter, and you should see some objective function output.
Take a look at AllDecAll_control.txt to see the objectives. (In the LRGV, the list of objectives is actually set in the control file. Not all models are like this — but the good thing here is it allows us to change the problem formulation ‘on the fly’ so to say). The objectives are cost, critical reliability, surplus, dropped transfers, number of leases, and drought transactions cost.
You have now compiled Borg, and separately compiled the LRGV simulation model. Stay tuned for edits to this post, where we refine the procedure and get the algorithm to talk to the simulation model.
Arranging the files for your job
Above, we talked about how it’s a good idea to have all your executables and project data in one place in order to speed up the solution process. Navigate out to the main folder that contains Borg and LRGV. When you do a ‘ls’ command you should see those two folders listed on the screen. Now let’s make a new folder: ‘mkdir run01’.
Change into the Borg folder (‘cd Borg’, or whatever you are calling the folder that contains the Borg source code), and then copy the borg.exe file to your run directory (‘cp borg.exe ../run01/’).
Similarly, change into the LRGV folder and pull out the executable and the project data. First change into the directory. You can take advantage of some Linux tricks here. Levels of different folders are denoted with dots (see above). Also, you can use wildcards so you don’t have to type out the whole folder name. So something like this should work: ‘cd ../LR*/’. When you change into a new folder, the ‘pwd’ command lets you know what folder you are in, and you can always ‘ls’ to list the contents of the directory.
The LRGV project information is contained in the ‘lrgv_bin’ folder. Change into it (‘cd lrgv_bin’) and then copy the executable (‘cp ./lrgvForMOEAFramework ../../run01’). Note, when you copy the executable, you’re copying it two levels up, because you’re within a subfolder of your LRGV directory here. We’re not done yet! The LRGV model requires some other files in this folder that must be included in your run folder. They are demand.txt, hydro.txt, initrights.txt, and so forth. As a hint, you can just copy all the text files over: ‘cp *.txt ../../run01’.
Now change into the run01 folder and do a ‘ls’ to admire your handiwork! The output of ls should show you that you have (i) the borg executable, (ii) the LRGV executable, and (iii) the LRGV data files (hydro.txt etc.) and (iv) the ‘control’ file that has LRGV parameters ([something]_control.txt). If all of these files are not contained in the run folder, the program will not work.
Linking to the Borg ‘Command Line Interface’
The biggest lesson I learned here is that you need to make sure there are no extraneous fprintf, cerr, cout, (etc…) statements in the code, that give messages to the console. If you are specifying a socket for communication, this isn’t a problem, but without a specific socket you’ll run into an error. If you download the latest version of the LRGV code from github, this problem should be fixed!
Here is a sample command to run Borg with LRGV. Note that the backslashes simply split the command on multiple lines.
./borg.exe -v 8 -o 6 -c 4 -C E -R runtime.txt -F 100 \
-l 0,0,0,0.1,0,0,0,0 -u 1,1,1,0.4,3,1,3,1 \
-e 0.003,0.002,0.01,0.002,0.003,0.001 -n 1000 \
-- ./lrgvForMOEAFramework -m std-io -b AllDecAll -c combined -r 5000
You are running Borg with 8 decision variables, 6 objectives, and 4 constraints (as told by the v, o, and c flags). The -C flag tells Borg that constraints are feasible when the constraint violation variables equal zero. The l and u flags control the lower and upper bounds of the variables. Epsilon values are given by the e flag, and the n flag tells Borg to run for 1000 function evaluations. Runtime output is specified using the -R flag, with the output being printed in runtime.txt. The -F flag tells Borg to write runtime output every 100 function evaluations. Everything to the right of the two dashes is the set of commands for the LRGV model. Here we are specifying 5000 Monte Carlo simulations per evaluation.
This run should take a few minutes to do. The cool thing is you will actually get some good results from only running Borg for 1000 iterations.
Quick Plotting of Borg Results in AeroVis
The first Borg output you will see on the console window will look something like this:
BORG version: 1.4
Current time: Mon Feb 10 09:30:32 2014
Problem: ./lrgvForMOEAFramework -m std-io -b AllDecAll -c combined -r 5000
Number of variables: 8
Number of objectives: 6
Number of constraints: 4
Lower bounds: 0 0 0 0.1 0 0 0 0
Upper bounds: 1 1 1 0.4 3 1 3 1
Epsilons: 0.003 0.002 0.01 0.002 0.003 0.001
Directions: 0 0 0 0 0 0
Constraint mode: E
Seed: (time)
This is a nice way to verify that your flags all got parsed correctly. Everything looks like it is in order. Note that Borg will show you the statement, verbatim, that is used to invoke your model. It is up to the programmer of the simulation model to verify it is working successfully!. Also note that the default random seed is the system time. This means that every time you run Borg you will get different results! If you don’t want this to be the case, set the random seed with the -s command.
Another piece of advice: With the command I typed above, It should take Borg several minutes to get results! this means if the program terminates or exits right away, something is wrong! Unfortunately, some input/output errors can cause a crash and you won’t know it. You can verify that things are working by looking at the runtime.txt file periodically.
Alright, so I waited several minutes and Borg finished. The results are a teachable moment! When we look at what happened, it turned out that only one solution was generated. “Why?” you ask. Well, good question. There are three explanations — these explanations are important because no matter what problem you’re solving, a lot of real world problems will suffer from these traits:
- The LRGV problem is highly constrained. Constraints are given in the AllDecAll_control.txt file. Reliability, critical reliability, cost variability, and drought transactions cost are constrained. Therefore many solutions that are generated by Borg may be infeasible, limiting the output set.
- We didn’t run the algorithm for a long time. Given more time, we may have found more solutions.
- Epsilon values may have been too coarse. Epsilon values set the precision that the user wants out of their optimization process. The epsilon values used in the trial may have been too coarse, and because of that, there was only one non-dominated solution generated. This may not be a bad thing — it may show there aren’t strong tradeoffs between your objectives. Take a look at Josh Kollat et al’s calibration work for more details on that.
- The treatment of uncertainty may affect the process. This is more a comment for other applications and not so much the LRGV. The objectives and constraints were carefully formulated here to be fairly robust to different ensemble sizes. We’ve run the problem with 5000 replicates all the way down to 1000 replicates and it generally works pretty well. However, perhaps you may see a low number results because it’s hard to get a feasible point with a high number of Monte Carlo replicates. You could run into some issues when you formulate objectives — what if you did an objective like ‘the number of failures in a time series’? Then the number of time steps that you run would directly influence the feasible range of objectives that you could see. That sounds like another post all together!
Ok so let’s modify the run to get something that has more points. I’ll change two things. The first is, I will make the epsilon resolution much more fine. Second, I am going to change the number of Monte Carlo simulations — this is really not about generating more solutions, but rather just to speed up the process. Here’s the modified command:
./borg.exe -v 8 -o 6 -c 4 -C E -R runtime.txt \
-F 100 -l 0,0,0,0.1,0,0,0,0 -u 1,1,1,0.4,3,1,3,1 \
-e 0.0003,0.0002,0.001,0.002,0.0003,0.0001 -n 1000 \
-- ./lrgvForMOEAFramework -m std-io -b AllDecAll \
-c combined -r 1000
Now after you’re done you are going to see many solutions, a few of which are here:
0.66648339565528636 0.67476996888284801 0.043713294445470278 0.33181321009155323 0.91922773565526716 0.45571096741028849 0.26815793390110088 0.28307276249003427 0.12025764110925453 -0.99891666666666667 0.41695535434092823 0.00012229087558711678 0.00068000000000000005 0
0.65802436611603377 0.17040802931274276 0.89787749927831673 0.37145853665497164 0.73557567609940055 0.28699499841402953 0.66876681220805767 0.64095194612404671 0.11439486913000001 -0.99775000000000003 0.42406373201305358 3.2074893365397502e-06 0.00033 0
0.71927499163180475 0.47214204629854206 0.91211932640347826 0.28733751854346368 1.1157608185160335 0.38838605481614208 1.1937640440664763 0.4668697964761449 0.12171828747183734 -0.9996666666666667 0.48153734145487198 0.00012463055456501487 0.00024000000000000001 0
0.70988260669592018 0.66195116819530686 0.86137019607092546 0.22492238687148389 0.04267660529415044 0.83243266132074933 1.4422130367535617 0.12824331995291713 0.1230293014619977 -0.99916666666666665 0.46583663718635338 0 0.00013999999999999999 0
0.65514335046148675 0.72225758980317201 0 0.32678886191611856 0.018830764515940516 0.67001200457315935 0.16300338695838254 0.22887574793603069 0.1198738899999995 -0.99608333333333332 0.41045367650680376 0 0 0
Ok so now all we have to do is save these results into a file and view them in AeroVis. This should also be the subject of another post but in the interest of completeness I’ll briefly discuss it here. I should note: we probably should have specified -f as a flag to the Borg, which would have indicated that we wanted to save our approximation set output file, and then we could have specified a filename. But it’s basically equivalent to copying and pasting which is what I’m about to do. Also this is another teachable moment — you should really be looking at the Borg readme and Borg documentation for the full set of commands for using this algorithm!
Anyway, simply copy all the numeric data in between pound signs (or hash tags if you are a fan of #twitter) onto the clipboard. Modern versions of Excel make it really easy to create a comma separated value (CSV) file of pasted data. Basically when you use the ‘paste’ command it will prompt you to use the ‘Text Import Wizard’ (see Microsoft help here), where you should tell the program that the data is space delimited. (Many of my fellow blog mates will probably scream bloody murder about recommending that you process data in Excel. I suppose they are right! But this procedure is really easy especially in modern versions of Excel, and recent versions of AeroVis are designed to seamlessly integrate with Excel. If you are a fan of other platforms, skip this step and somehow parse the data into a comma separated format such that you have headers in the first line and then the data in subsequent lines.)
Once the data is in Excel, add a row with the appropriate header. Basically, Borg outputs the decision variables first, then the objectives. Each row is a different solution. So if you had S solutions, N decision variables, and M objectives, you would basically have a data table that had S rows by N+M columns. The header for our problem would be the following line. In the final CSV file, this will have commas in between each entry, but in Excel each entry is in its own cell (i.e. the first entry is in A1, the second entry is in B1, and so forth)
Rights,OptLow,OptHigh,Xi,Alpha2,Beta2,Alpha,Beta,Cost,CritRel,Surplus,Drop,NumLeases,DrTransCost
Save the file with some name, myfile.csv for example. Open AeroVis, and then in a separate window navigate to where the file is stored (in the ‘Explorer’ for example). Now simply drag the file into the main window of AeroVis! The ‘Data Import Wizard’ should come up. Click ‘Select All’ to select all the variables, and hit ok.
You should see something like the following [Insert image here]. You can kind of see a faint cone on the bottom left side, but it looks pretty transparent. If you use the ‘Plot Control Buttons’ tool, you can see that the program just assumes the first variable (Rights) is on the first axis (X), the second variable (OptLow) is on the second axis (Y) and so forth.
In the final update to this post, I will discuss good plotting settings to use and how to manipulate the AeroVis plot to learn more about the results you got. Please stay tuned!
Water Programming: A Collaborative Research Blog 