PBS Job Submission with Python

(Update: this, along with other examples of HPC submission scripts, can be found here: https://github.com/jdherman/hpc-submission-scripts).

Instead of using Bash to automate cluster submission scripts, you may be interested in trying Python instead. The end result is the same, but the process is a bit cleaner. I’ll paste an example below that iterates through a loop to submit a bunch of jobs. It follows the same outline as our usual Bash scripts (i.e., build a job submission string and then pipe it to qsub).

#!/usr/bin/python
# Example PBS cluster job submission in Python

from popen2 import popen2
import time

# If you want to be emailed by the system, include these in job_string:
#PBS -M your_email@address
#PBS -m abe  # (a = abort, b = begin, e = end)

# Loop over your jobs
for i in range(1, 10):

    # Open a pipe to the qsub command.
    output, input = popen2('qsub')
    
    # Customize your options here
    job_name = "my_job_%d" % i
    walltime = "1:00:00"
    processors = "nodes=1:ppn=1"
    command = "./my_program -n %d" % i

    job_string = """#!/bin/bash
    #PBS -N %s
    #PBS -l walltime=%s
    #PBS -l %s
    #PBS -o ./output/%s.out
    #PBS -e ./error/%s.err
    cd $PBS_O_WORKDIR
    %s""" % (job_name, walltime, processors, job_name, job_name, command)
    
    # Send job_string to qsub
    input.write(job_string)
    input.close()
    
    # Print your job and the system response to the screen as it's submitted
    print job_string
    print output.read()
    
    time.sleep(0.1)

Make sure you give executable permissions (chmod +x thisFile.py) and then just run ./thisFile.py from the command line.

Advertisements

10 thoughts on “PBS Job Submission with Python

  1. Pingback: Using linux “split” « Pat Reed Group Research Tips Blog

  2. Pingback: Python Data Analysis Part 2: Pandas / Matplotlib Live Demo | Pat Reed Group Research Tips Blog

  3. Pingback: Re-evaluating solutions using Python subprocesses | Pat Reed Group Research Tips Blog

  4. Pingback: Python for automating cluster tasks: Part 1, Getting started | Water Programming: A Collaborative Research Blog

  5. Pingback: Page not found | Water Programming: A Collaborative Research Blog

  6. Thank you! I have been trying to submit jobs with Python for a while now, but none of the usual shell command functions (e.g., subprocess.call, os.system) have worked. Implementing your solution worked great! Many thanks!

  7. Is there an updated version of this? I get the following message:

    DeprecationWarning: The popen2 module is deprecated. Use the subprocess module.

  8. Hi Bob,

    You can try using the subprocess.call() function instead of popen2. Here’s a tutorial:
    http://www.pythonforbeginners.com/os/subprocess-for-system-administrators

    subprocess.call() receives the full command, so instead of opening a pipe to qsub you would just do:
    subprocess.call(‘qsub whatever …’, shell=True)

    and it should capture the return value from the shell. If you get this to work, please post a comment so we have a record of it 🙂

  9. Pingback: Water Programming Blog Guide (Part I) – Water Programming: A Collaborative Research Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s