OpenPBS (Portable Batch System) HOW TO


Our Linux cluster gibbs has OpenPBS installed to facilitate the computation job submissions and managements.
Jobs are not allowed to run on the master node, witch is  used for interactive editing,
compiling and testing  your programs. All your  jobs have to be submitted on to  slave nodes. Any jobs running on master node and have accumulate more than 30 min. of CPU time will be killed without noticing the owners.



Batch Commands

qsub [ option(s) ] [ script-file ]

All options to qsub can be specified either on the command line or inside the script  file. Command line options will overwrite the options inside the script file. Scripts can be submitted using qsub [list of qsub options] script_name.

qsub scripts examples (jobs just require one or two processors on the same node):


a very easy script

#!/bin/csh
###join both output and error files
#PBS -j oe
### Run your executable
/home/people/yuan/fortran_codes/test_pgf77.exe



A more complete example

#!/bin/csh
### Job name job (default is name of pbs script file)
#PBS -N test
###set u id, check it with id command
###PBS -u 1153(yuan)
#PBS -u yuan
#
### Number and which of nodes as well as number of CPUs:

#PBS -l nodes="node9:ppn=1"
#
### Declare job non-rerunable

#PBS -r n
### resource limits: amount of memory and CPU time ([[h:]m:]s).
#PBS -l mem=213mb
#PBS -l walltime=2:00:00
#PBS -l cput=1:00:00
#
### send me email when job begins

#PBS -m b
### send me email when job end
#PBS -m e
### send me email when job aborts (with an error)
#PBS –m a 
#
### Output files

#PBS -e /home/people/yuan/fortran_codes/test.err
#PBS -o /home/people/yuan/fortran_codes/test.log
### Queue name (small, medium, long, verylong)
#PBS -q default
#
#############
### This job's working directory
echo Working directory is $PBS_O_WORKDIR
cd $PBS_O_WORKDIR    
echo Running on host `hostname`
echo Time is `date`
echo Directory is `pwd`
# Run your executable
/home/people/yuan/fortran_codes/test_pgf77.exe

Notice:

If a user's job was aborted, you can check the standarnd error file of the job you submitted. For example in above script:
/home/people/yuan/fortran_codes/test.err




Real world example (NOAH/DIAMOD PBS script)


Notice: You can not run any shell script before #PBS scripts were finished, which means
#PBS -q default is the last line of PBS scripting and you have to run normall shell scripting after it.



OpenPBS User Manuals

An easy handy reference manual please click here (PDF file)
A complete manual written for PBS_Pro please click here (PDF file). There are many differences between
OpenPBS and PBS_Pro.



qsub script example (MPICH parallel jobs):



Some useful PBS preset enviromental varibales

When your batch job starts running, some environment variables are predefined, which include the
variables defined on the node where your job is running and the variables exported from the
submission node with -v (selected variables) and -V (all variables) .
To export PBS variables just simply add: #PBS -V for all or #PBS -v for


Variables predefined by PBS
$PBS_HOME          The home dirrectory Where you ran qsub command
$PBS_O_LOGNAME     Your UID
$PBS_O_HOST        Your host or node name from where you ran qsub
$PBS_O_QUEUE       The queue name from which you submitted your job
$PBS_QUEUE         The queue name where your job is running
$PBS_JOBID         The job ID
$PBS_JOBNAME       The job name
$PBS_ENVIRONMENT   The PBS enviromental variables
$PBS_O_WORKDIR     The working directory from where you ran qsub



PBS Commands for users

qsub shellscript
submit a shellscript to PBS batch queue
qsub -I
submit an interactive-batch job
qsub -q queue
submit job directly to specified queue
qstat
gives information about queues and jobs
qstat -q
gives all queues status on gibbs
qstat -Q
gives queue limits for all queues
qstat -a
gives the status of all jobs on gibbs
qstat -au userid
gives all jobs owned by user userid
qstat -s
gives all jobs' status
qstat -r
gives all running jobs' status
qstat -f
gives detailed information on all job
qstat -f jobid
gives detailed information on a particular job
qstat -Qf queue gives detailed information about specified queue
qstat -B gives summary information about the PBS server
qdel jobid
delete a specified job
qalter job
modify the attributes of a job
qhold
stops a running job and holds it in the queue
(manager  only)
qrls
release a hold job back to queue
(manager only)
qmove
move a batch job to a different queue
(manager only)
xpbs -admin
GUI PBS interface  (manager  only)
xpbs
Graphical User Interface of PBS commands
xpbsmon
node monitoring via GUI
pbsnodes -a
gives the status of all pbs nodes