SEARCH
TOOLBOX
LANGUAGES
modified on 6 October 2014 at 11:40 ••• 5,629 views

Code:SGE

From Predictive Chemistry

Jump to: navigation, search

User environment

Zeroth, set up your own system (for linux), by adding a host def for circe to your $HOME/.ssh/config file. Then you don't have to keep typing in circe's full path and your username when running ssh from linux.

Host circe
  User <username on circe>
  Hostname circe.rc.usf.edu
  ServerAliveInterval 30
  ServerAliveCountMax 120
  ForwardX11 yes


You may need:

 chmod 600 $HOME/.ssh/config

On circe, first, setup your user environment by creating/editing a $HOME/.bashrc

# .bashrc
 
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
 
# User specific aliases and functions
alias mystat="qstat -u $USER -t"

Then create $HOME/.bash_profile, which is run for interactive, login sessions on circe.

# .bash_profile
 
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
 
# User specific environment and startup programs
 
PATH=$PATH:$HOME/bin
#module load compilers/intel/11.1.064 mpi/openmpi-1.4.1/intel-11.1.064
module purge
module load compilers/intel/11.1.064 mpi/openmpi-1.4.1/intel-11.1.064 apps/cuda/5.0.35
 
# apps/nwchem/6.1.1
#module load apps/namd/2.7 apps/iso2mesh/0.8.0 apps/netcdf/4.2.0 apps/gaussian/03e01_em64t apps/gromacs/4.5.5a
 
export PATH

Queue

Here's a basic template for queuing a job using MPI (here 4 cpus and 6h max run-time).

#$ -N a-test-job
#$ -cwd
#$ -o job-$JOB_ID.out
#$ -e job-$JOB_ID.err
#$ -l pcpus=4,h_rt=06:00:00
 
# Example minimum memory request:
#$ -l mpj=10G
# Max run-time sets queue:
# queue         max run-time
# devel 	1 hour
# short 	6 hours
# medium 	2 day
# long 	1 week 
 
mpirun parallel-executable

Submit with

qsub job.sh

For execution and status, use qstat, or the mystat command, defined as an alias in .bashrc.

For more info, see the CIRCE SGE guide.

Here's a run-down on the environment variables available during running (for scripting):

# Environment Menu:
#    $JOB_ID: The job number assigned by the scheduler to your job
# 1844896
#    $JOBDIR: The directory your job is currently running in
# <blank>
#    $USER: The username of the person currently running the job
# anyuser
#    $TMPDIR: The directory containing informational files about the job e.g. the machines allocated to run the job, etc.
# /tmp/1844896.1.default
#    $SGE_O_WORKDIR: Similar to $JOBDIR
# /home/a/anyuser/test
#    $HOME: User's home directory on execution machine
# /home/d/anyuser
#    $JOB_NAME: The job name specified by -N option
# a-test-job
#    $HOSTNAME: Name of execution host
# wh-520-7-5.rc.usf.edu
#    $TASK_ID: You can submit array jobs. This would be the TASK number in the array job of the current task
# <blank>
#    $SGE_CELL: SGE cell that the job is currently running in
# default
#    $SGE_O_SHELL: Invoking shell of the current SGE job
# /bin/bash
#    $QUEUE: Current running queue
# default