I’ve recently been using Bash shell scripts to improve the efficiency of my workflow when working on Linux systems and I thought I would share some of them here. I’m fairly new to Linux so this post is not meant to be a comprehensive guide on how to write shell scripts rather, I hope the scripts in this post can serve as examples for those who may also be learning Linux and unsure of where or how to start writing shell scripts. I didn’t write any of these from scratch, most of the scripts are based off files shared with me by group members Julie Quinn, Bernardo Trindade and Jazmin Zatarian Salazar. If you’re interested in learning more about any of the commands used in these scripts I’ve put some references I found useful at the end of this post. If you’re more experienced in writing shell scripts, please feel free to put tips or suggestions in the comments.
1. A simple script for making directories
For my research I’m processing results of a monte carlo simulation for several solutions found through multi-objective search and I needed to make folders in several locations to store the output from each solution. My first instinct was to make each directory separately using the mkdir command in the command line, but this quickly got tedious. Instead I used a bash script to loop through all the solution numbers and create a new directory for each. For more on using loops in Bash, check out this reference.
#!/bin/bash#!/bin/bash # This script will create directories named "Solution_*.txt" for # a set of numbered solutions # specify solution numbers SOLUTIONS=('162' '1077' '1713' '1725' '1939' '2191' '2290' '2360') # create a variable to store the string "Solution_" DIRECTORY="Solution_" # loop over solution numbers for i in ${SOLUTIONS[@]} do # create a separate directory for each solution mkdir $DIRECTORY${i} done
2. Calling a Java function and saving the output
The MOEA framework is a tool written in Java with all sorts of cool functions. I used it to generate 1024 latin hypercube samples across a given range for each of the 8 solutions mentioned above. Using a shell script allows for you to easily set up the arguments needed for the MOEA framework, call the Java function and save the output to your desired file format. The MOEA framework’s tool spits out a .txt file, but this script uses the “sed” command to save it as a .csv file. More on “sed” can be found in the reference at the end of this post.
#!/bin/bash#!/bin/bash # this shell script will call the MOEA framework's Latin Hypercube # Sampling tool to create 1024 samples from a set of # prespecified ranges for each of 8 solutions # create variables to store Java arguments JAVA_ARGS="-Xmx1g -classpath MOEAFramework-1.16-Executable.jar" NUM_SAMPLES=1024 METHOD=latin # these are the solutions we will create samples from SOLUTIONS=('162' '1077' '1713' '1725' '1939' '2191' '2290' '2360') # loop through solutions for i in ${SOLUTIONS[@]} do # define names for input (ranges) and output file names RANGES_FILENAME=${i}ranges.txt OUTPUT_FILENAME=Solution${i}_Samples.txt CSV_FILENAME=Solution${i}_Samples.csv # Call MOEA framework from JAVA using specified arguments to # create LHS Samples, specify OUTPUT_FILENAME as output java ${JAVA_ARGS} org.moeaframework.analysis.sensitivity.SampleGenerator -m ${METHOD} -n ${NUM_SAMPLES} -p ${RANGES_FILENAME} -o ${OUTPUT_FILENAME} # Use the sed command tocreate new comma separated values file # from original output .txt file sed 's/ /,/g' ${OUTPUT_FILENAME} > ${CSV_FILENAME} # remove .txt files rm $OUTPUT_FILENAME done
3. A piping example
Piping allows you to link together programs by making the output from one program or function the input to another. The script below was originally written by my friend Shrutarshi Basu for a class project we were working on together. This script is made to process the output from the Borg MOEA for 9 random seeds of the DTLZ2 benchmarking problem across several different algorithmic configurations, seen in the code as “masters” (for more on this see Jazmin’s post here). In addition to calling Java tools from the MOEAframework, Basu uses piping to link the Linux commands “tac”, “sed”, “grep” and “cut”. For more on each of these commands, see the links at the bottom of this post.
# loop over each of 9 seeds for i in {0..9} do obj=DTLZ2_S${i}.obj output=dtlz2.volume # loop over masters for m in $(seq 0 $1) do runtime=DTLZ2_S${i}_M${m}.runtime mobj=DTLZ2_S${i}_M${m}.obj # extract objectives from output echo "Extracting objectives" tac ${runtime} | sed -n '1,/\/\// p' | grep -v "//" | cut -d' ' -f15-19 | tac > ${mobj}; done # combine objectives into one file echo "Combining objectives" java -cp ../../moea.jar org.moeaframework.analysis.sensitivity.ResultFileSeedMerger \ -d 5 -e 0.01,0.01,0.01,0.01,0.01 \ -o ${obj} DTLZ2_S${i}_M*.obj # calculate the hypervolume echo "Finding final hypervolume" hvol=$(java -cp ../../moea.jar HypervolumeEval ${obj}) printf "%s %s\n" "$i" "$hvol" >> ${output} echo "Done with seed $i" done
Additional References and Links
- For background on basic Bash and defining variables, see this link.
- For a demonstration of useful Linux commands, see Bernardo’s post from 2015.
- For detail on linux commands used above use these links (click name for link): sed, tac, grep, cut, split.
- For a tutorial on piping and redirection in Linux see this reference.
- For information on using the MOEAframework see the documentation, found here.
Pingback: More on simple Bash Shell scripts (examples of “find” and “sed”) – Water Programming: A Collaborative Research Blog
Pingback: More on simple Bash Shell scripts (examples of “find” and “sed”) – Hydrogen Water