PBS Job Submission with Python

(Update: this, along with other examples of HPC submission scripts, can be found here: https://github.com/jdherman/hpc-submission-scripts).

Instead of using Bash to automate cluster submission scripts, you may be interested in trying Python instead. The end result is the same, but the process is a bit cleaner. I’ll paste an example below that iterates through a loop to submit a bunch of jobs. It follows the same outline as our usual Bash scripts (i.e., build a job submission string and then pipe it to qsub).

#!/usr/bin/python
# Example PBS cluster job submission in Python

from popen2 import popen2
import time

# If you want to be emailed by the system, include these in job_string:
#PBS -M your_email@address
#PBS -m abe  # (a = abort, b = begin, e = end)

# Loop over your jobs
for i in range(1, 10):

    # Open a pipe to the qsub command.
    output, input = popen2('qsub')
    
    # Customize your options here
    job_name = "my_job_%d" % i
    walltime = "1:00:00"
    processors = "nodes=1:ppn=1"
    command = "./my_program -n %d" % i

    job_string = """#!/bin/bash
    #PBS -N %s
    #PBS -l walltime=%s
    #PBS -l %s
    #PBS -o ./output/%s.out
    #PBS -e ./error/%s.err
    cd $PBS_O_WORKDIR
    %s""" % (job_name, walltime, processors, job_name, job_name, command)
    
    # Send job_string to qsub
    input.write(job_string)
    input.close()
    
    # Print your job and the system response to the screen as it's submitted
    print job_string
    print output.read()
    
    time.sleep(0.1)

Make sure you give executable permissions (chmod +x thisFile.py) and then just run ./thisFile.py from the command line.