Multiple Run Job Submission

There might be times when you would like to run multiple executions of you job and have the output sort to separate directories so they don't overwrite each other.

Simply put, the sample script provided will copy input files (including program executable) to a directory based on the provide "Jab Name).  Therefore, by simply changing the job Name, multiple instances can be "concurrently running", without affect each other.

The "job-submission.pbs" script file can be found on the HPC system at /apps/samples/job-submission directory, in which can be copied to a directory of your choosing.  Some general notes:

• The variable $PBS_O_WORKDIR indicates the directory where the submission script file is located and launched from (qsub'ed from).  This means you can place this file in any directory from you choosing and the value of $PBS_O_WORKDIR will be automatically defined.
• The program executable "my_program.exe" (found within the input directory) is a simple "Hello World" program and can be replaced with your own executable and input files.
• The defining of an email address section within the submission script has been purposely removed.  This will be an optional command line argument that can be added if desired.
• The "jobname" will be defined as a command line argument, which will allow the value to be changed quickly and efficiently.  Note that PBS has a jobname limit of 10 characters.

The details for this script (job.submit.pbs) can be found below:

###### Select resources #####
#### Job Name will be defined upon using qsub -N <jobname> <this-script file>
#PBS -l ncpus=1
#PBS -l mem=1gb

##### Queue #####
#pbs -q workq

##### Mail Options #####
#PBS -m abe

### Set up the run
JOB_DIR=$PBS_O_WORKDIR/$PBS_JOBNAME
INPUT_DIR=$PBS_O_WORKDIR/input

mkdir -p $JOB_DIR
cp -pr $PBS_O_WORKDIR/input/* $JOB_DIR

##### Change to current working directory #####
cd $JOB_DIR

##### Execute Program #####
./my_program.exe 2> $PBS_JOBNAME.err > $PBS_JOBNAME.out

With the above example, you may wish to change the values for a number of PBS arguments within the script file, these include:

#PBS -l ncpus=1 [this value can be changed to the number of CPU's your program requires (maximum 64)]
#PBS -l mem=1gb [this value can be changed to the amount for memory program requires, examples include 5gb, 10gb, 50gb up to 500gb]

You may wish to change the program being executed, or run something else like Matlab or R.  The line containing

./my_program.exe 2> $PBS_JOBNAME.err > $PBS_JOBNAME.out

Would be changed to some like the following (depending on the requirements):

./<program executable name> 2> $PBS_JOBNAME.err > $PBS_JOBNAME.out
R --vanilla < [Your R file].R 2> $PBS_JOBNAME.err > $PBS_JOBNAME.out
matlab -nodisplay -nodesktop -nojvm -nosplash < <matlab-program.m> 2> $PBS_JOBNAME.err > $PBS_JOBNAME.out

To submit your program, you will need to use the following syntax:

qsub –N <jobname> job-submission.pbs

If you also want the system to send you an email when your job starts and finishes, use the following syntax:

qsub –N <jobname> -M <email address> job-submission.pbs

Real case example

If we leave the PBS arguments the default values (Memory, CPU and the program executable will be my_program.exe[hello world program]), I can execute the following commands:

bellj@isaac:~/job_submit> pwd
/home/bellj/job_submit
bellj@isaac:~/job_submit> ls input/
my_program.exe

The above commands simply provide information on the files contained within the input directory and where this PBS submission was executed from.

bellj@isaac:~/job_submit> qsub -N run1 -M j.bell@cqu.edu.au job_submit.pbs
144951.pbsserver
bellj@isaac:~/job_submit> qsub -N run2 -M j.bell@cqu.edu.au job_submit.pbs
144952.pbsserver
bellj@isaac:~/job_submit> qsub -N run3 -M j.bell@cqu.edu.au job_submit.pbs
144953.pbsserver
bellj@isaac:~/job_submit> qsub -N run4 -M j.bell@cqu.edu.au job_submit.pbs
144954.pbsserver
bellj@isaac:~/job_submit> qsub -N run5 -M j.bell@cqu.edu.au job_submit.pbs
144955.pbsserver

bellj@isaac:~/job_submit> ll
drwxr-xr-x 2 bellj users  27 Apr 30 16:24 input
-rw-r--r-- 1 bellj users 529 May  3 09:59 job_submit.pbs
drwx------ 2 bellj users  57 May  3 15:12 run1
-rw------- 1 bellj users   0 May  3 15:12 run1.e144951
-rw------- 1 bellj users   0 May  3 15:12 run1.o144951
drwx------ 2 bellj users  57 May  3 15:12 run2
-rw------- 1 bellj users   0 May  3 15:12 run2.e144952
-rw------- 1 bellj users   0 May  3 15:12 run2.o144952
drwx------ 2 bellj users  57 May  3 15:12 run3
-rw------- 1 bellj users   0 May  3 15:12 run3.e144953
-rw------- 1 bellj users   0 May  3 15:12 run3.o144953
drwx------ 2 bellj users  57 May  3 15:12 run4
-rw------- 1 bellj users   0 May  3 15:12 run4.e144954
-rw------- 1 bellj users   0 May  3 15:12 run4.o144954
drwx------ 2 bellj users  57 May  3 15:12 run5
-rw------- 1 bellj users   0 May  3 15:12 run5.e144955
-rw------- 1 bellj users   0 May  3 15:12 run5.o144955

The files $JOBNAME.o#### and $JOBNAME.e#### which are listed in the default folder, should be empty, unless some PBS error information is provided.

The directories run1, run2, run3, run4 and run5 in this example will contain the following:

bellj@isaac:~/job_submit> ll run1
-rwx------ 1 bellj users 31583 Apr 30 16:24 my_program.exe
-rw------- 1 bellj users     0 May  3 15:12 run1.err
-rw------- 1 bellj users    26 May  3 15:12 run1.out

As you can see, the "Error" file run1.err is empty and the output file run1.out contains:

bellj@isaac:~/job_submit> cat run1/run1.out
Hello world - Jason here

As you can see, using this method, by simply changing the "JobName", you can run many programs and the output for each program execution will be located in separate directories.