Code:SGE

From Predictive Chemistry
Jump to: navigation, search

User environment

Zeroth, set up your own system (for linux), by adding a host def for circe to your $HOME/.ssh/config file. Then you don't have to keep typing in circe's full path and your username when running ssh from linux.

Host circe
  User <username on circe>
  Hostname circe.rc.usf.edu
  ServerAliveInterval 30
  ServerAliveCountMax 120
  ForwardX11 yes


You may need:

 chmod 600 $HOME/.ssh/config

On circe, first, setup your user environment by creating/editing a $HOME/.bashrc

<source lang="bash">

  1. .bashrc
  1. Source global definitions

if [ -f /etc/bashrc ]; then

       . /etc/bashrc

fi

  1. User specific aliases and functions

alias mystat="qstat -u $USER -t" </source>

Then create $HOME/.bash_profile, which is run for interactive, login sessions on circe.

<source lang="bash">

  1. .bash_profile
  1. Get the aliases and functions

if [ -f ~/.bashrc ]; then

       . ~/.bashrc

fi

  1. User specific environment and startup programs

PATH=$PATH:$HOME/bin

  1. module load compilers/intel/11.1.064 mpi/openmpi-1.4.1/intel-11.1.064

module purge module load compilers/intel/11.1.064 mpi/openmpi-1.4.1/intel-11.1.064 apps/cuda/5.0.35

  1. apps/nwchem/6.1.1
  2. module load apps/namd/2.7 apps/iso2mesh/0.8.0 apps/netcdf/4.2.0 apps/gaussian/03e01_em64t apps/gromacs/4.5.5a

export PATH </source>

Queue

Here's a basic template for queuing a job using MPI (here 4 cpus and 6h max run-time).

<source lang="bash">

  1. $ -N a-test-job
  2. $ -cwd
  3. $ -o job-$JOB_ID.out
  4. $ -e job-$JOB_ID.err
  5. $ -l pcpus=4,h_rt=06:00:00
  1. Example minimum memory request:
  2. $ -l mpj=10G
  3. Max run-time sets queue:
  4. queue max run-time
  5. devel 1 hour
  6. short 6 hours
  7. medium 2 day
  8. long 1 week

mpirun parallel-executable </source>

Submit with <source lang="bash"> qsub job.sh </source>

For execution and status, use qstat, or the mystat command, defined as an alias in .bashrc.

For more info, see the CIRCE SGE guide.

Here's a run-down on the environment variables available during running (for scripting): <source lang="bash">

  1. Environment Menu:
  2. $JOB_ID: The job number assigned by the scheduler to your job
  3. 1844896
  4. $JOBDIR: The directory your job is currently running in
  5. <blank>
  6. $USER: The username of the person currently running the job
  7. anyuser
  8. $TMPDIR: The directory containing informational files about the job e.g. the machines allocated to run the job, etc.
  9. /tmp/1844896.1.default
  10. $SGE_O_WORKDIR: Similar to $JOBDIR
  11. /home/a/anyuser/test
  12. $HOME: User's home directory on execution machine
  13. /home/d/anyuser
  14. $JOB_NAME: The job name specified by -N option
  15. a-test-job
  16. $HOSTNAME: Name of execution host
  17. wh-520-7-5.rc.usf.edu
  18. $TASK_ID: You can submit array jobs. This would be the TASK number in the array job of the current task
  19. <blank>
  20. $SGE_CELL: SGE cell that the job is currently running in
  21. default
  22. $SGE_O_SHELL: Invoking shell of the current SGE job
  23. /bin/bash
  24. $QUEUE: Current running queue
  25. default

</source>