Code:SGE

User environment

Zeroth, set up your own system (for linux), by adding a host def for circe to your $HOME/.ssh/config file. Then you don't have to keep typing in circe's full path and your username when running ssh from linux.

Host circe
  User <username on circe>
  Hostname circe.rc.usf.edu
  ServerAliveInterval 30
  ServerAliveCountMax 120
  ForwardX11 yes

You may need:

 chmod 600 $HOME/.ssh/config

On circe, first, setup your user environment by creating/editing a $HOME/.bashrc

<source lang="bash">

1. .bashrc

1. Source global definitions

if [ -f /etc/bashrc ]; then

       . /etc/bashrc

fi

1. User specific aliases and functions

alias mystat="qstat -u $USER -t" </source>

Then create $HOME/.bash_profile, which is run for interactive, login sessions on circe.

<source lang="bash">

1. .bash_profile

1. Get the aliases and functions

if [ -f ~/.bashrc ]; then

       . ~/.bashrc

fi

1. User specific environment and startup programs

PATH=$PATH:$HOME/bin

1. module load compilers/intel/11.1.064 mpi/openmpi-1.4.1/intel-11.1.064

module purge module load compilers/intel/11.1.064 mpi/openmpi-1.4.1/intel-11.1.064 apps/cuda/5.0.35

1. apps/nwchem/6.1.1
2. module load apps/namd/2.7 apps/iso2mesh/0.8.0 apps/netcdf/4.2.0 apps/gaussian/03e01_em64t apps/gromacs/4.5.5a

export PATH </source>

Queue

Here's a basic template for queuing a job using MPI (here 4 cpus and 6h max run-time).

<source lang="bash">

1. $ -N a-test-job
2. $ -cwd
3. $ -o job-$JOB_ID.out
4. $ -e job-$JOB_ID.err
5. $ -l pcpus=4,h_rt=06:00:00

1. Example minimum memory request:
2. $ -l mpj=10G
3. Max run-time sets queue:
4. queue max run-time
5. devel 1 hour
6. short 6 hours
7. medium 2 day
8. long 1 week

mpirun parallel-executable </source>

Submit with <source lang="bash"> qsub job.sh </source>

For execution and status, use qstat, or the mystat command, defined as an alias in .bashrc.

For more info, see the CIRCE SGE guide.

Here's a run-down on the environment variables available during running (for scripting): <source lang="bash">

1. Environment Menu:
2. $JOB_ID: The job number assigned by the scheduler to your job
3. 1844896
4. $JOBDIR: The directory your job is currently running in
5. <blank>
6. $USER: The username of the person currently running the job
7. anyuser
8. $TMPDIR: The directory containing informational files about the job e.g. the machines allocated to run the job, etc.
9. /tmp/1844896.1.default
10. $SGE_O_WORKDIR: Similar to $JOBDIR
11. /home/a/anyuser/test
12. $HOME: User's home directory on execution machine
13. /home/d/anyuser
14. $JOB_NAME: The job name specified by -N option
15. a-test-job
16. $HOSTNAME: Name of execution host
17. wh-520-7-5.rc.usf.edu
18. $TASK_ID: You can submit array jobs. This would be the TASK number in the array job of the current task
19. <blank>
20. $SGE_CELL: SGE cell that the job is currently running in
21. default
22. $SGE_O_SHELL: Invoking shell of the current SGE job
23. /bin/bash
24. $QUEUE: Current running queue
25. default

</source>