(Update: this, along with other examples of HPC submission scripts, can be found here: https://github.com/jdherman/hpc-submission-scripts).
Instead of using Bash to automate cluster submission scripts, you may be interested in trying Python instead. The end result is the same, but the process is a bit cleaner. I’ll paste an example below that iterates through a loop to submit a bunch of jobs. It follows the same outline as our usual Bash scripts (i.e., build a job submission string and then pipe it to qsub).
#!/usr/bin/python # Example PBS cluster job submission in Python from popen2 import popen2 import time # If you want to be emailed by the system, include these in job_string: #PBS -M your_email@address #PBS -m abe # (a = abort, b = begin, e = end) # Loop over your jobs for i in range(1, 10): # Open a pipe to the qsub command. output, input = popen2('qsub') # Customize your options here job_name = "my_job_%d" % i walltime = "1:00:00" processors = "nodes=1:ppn=1" command = "./my_program -n %d" % i job_string = """#!/bin/bash #PBS -N %s #PBS -l walltime=%s #PBS -l %s #PBS -o ./output/%s.out #PBS -e ./error/%s.err cd $PBS_O_WORKDIR %s""" % (job_name, walltime, processors, job_name, job_name, command) # Send job_string to qsub input.write(job_string) input.close() # Print your job and the system response to the screen as it's submitted print job_string print output.read() time.sleep(0.1)
Make sure you give executable permissions (chmod +x thisFile.py) and then just run ./thisFile.py from the command line.
Pingback: Using linux “split” « Pat Reed Group Research Tips Blog
Pingback: Python Data Analysis Part 2: Pandas / Matplotlib Live Demo | Pat Reed Group Research Tips Blog
Pingback: Re-evaluating solutions using Python subprocesses | Pat Reed Group Research Tips Blog
Pingback: Python for automating cluster tasks: Part 1, Getting started | Water Programming: A Collaborative Research Blog
Pingback: Page not found | Water Programming: A Collaborative Research Blog
Thank you! I have been trying to submit jobs with Python for a while now, but none of the usual shell command functions (e.g., subprocess.call, os.system) have worked. Implementing your solution worked great! Many thanks!
Is there an updated version of this? I get the following message:
DeprecationWarning: The popen2 module is deprecated. Use the subprocess module.
Hi Bob,
You can try using the subprocess.call() function instead of popen2. Here’s a tutorial:
http://www.pythonforbeginners.com/os/subprocess-for-system-administrators
subprocess.call() receives the full command, so instead of opening a pipe to qsub you would just do:
subprocess.call(‘qsub whatever …’, shell=True)
and it should capture the return value from the shell. If you get this to work, please post a comment so we have a record of it 🙂
Beautiful, thank you for posting this.
Pingback: Water Programming Blog Guide (Part I) – Water Programming: A Collaborative Research Blog
Pingback: Quick guide to distributed computing system TORQUE – Chris Gahnström
Pingback: Coarse-Graining Routine – Baso's Brain Bucket
I have a script file bh1.py. I am going to run it with qsub. how can I run my python script file with your code?I copied this file into my directory in cluster. and also copied bh1.py file to directory in cluster. but I don’t know how to put bh1.py in your code.
hi. I have a python script bh1.py . can you tell me how can I use your code exactly? I am new with python.