Zeroth, set up your own system (for linux), by adding a host def for circe to your $HOME/.ssh/config file. Then you don't have to keep typing in circe's full path and your username when running ssh from linux.
Host circe User <username on circe> Hostname circe.rc.usf.edu ServerAliveInterval 30 ServerAliveCountMax 120 ForwardX11 yes
You may need:
chmod 600 $HOME/.ssh/config
On circe, first, setup your user environment by creating/editing a $HOME/.bashrc
- Source global definitions
if [ -f /etc/bashrc ]; then
- User specific aliases and functions
alias mystat="qstat -u $USER -t" </source>
Then create $HOME/.bash_profile, which is run for interactive, login sessions on circe.
- Get the aliases and functions
if [ -f ~/.bashrc ]; then
- User specific environment and startup programs
- module load compilers/intel/11.1.064 mpi/openmpi-1.4.1/intel-11.1.064
module purge module load compilers/intel/11.1.064 mpi/openmpi-1.4.1/intel-11.1.064 apps/cuda/5.0.35
- module load apps/namd/2.7 apps/iso2mesh/0.8.0 apps/netcdf/4.2.0 apps/gaussian/03e01_em64t apps/gromacs/4.5.5a
export PATH </source>
Here's a basic template for queuing a job using MPI (here 4 cpus and 6h max run-time).
- $ -N a-test-job
- $ -cwd
- $ -o job-$JOB_ID.out
- $ -e job-$JOB_ID.err
- $ -l pcpus=4,h_rt=06:00:00
- Example minimum memory request:
- $ -l mpj=10G
- Max run-time sets queue:
- queue max run-time
- devel 1 hour
- short 6 hours
- medium 2 day
- long 1 week
mpirun parallel-executable </source>
Submit with <source lang="bash"> qsub job.sh </source>
For execution and status, use qstat, or the mystat command, defined as an alias in .bashrc.
For more info, see the CIRCE SGE guide.
Here's a run-down on the environment variables available during running (for scripting): <source lang="bash">
- Environment Menu:
- $JOB_ID: The job number assigned by the scheduler to your job
- $JOBDIR: The directory your job is currently running in
- $USER: The username of the person currently running the job
- $TMPDIR: The directory containing informational files about the job e.g. the machines allocated to run the job, etc.
- $SGE_O_WORKDIR: Similar to $JOBDIR
- $HOME: User's home directory on execution machine
- $JOB_NAME: The job name specified by -N option
- $HOSTNAME: Name of execution host
- $TASK_ID: You can submit array jobs. This would be the TASK number in the array job of the current task
- $SGE_CELL: SGE cell that the job is currently running in
- $SGE_O_SHELL: Invoking shell of the current SGE job
- $QUEUE: Current running queue