modified on 13 October 2014 at 13:38 ••• 4,498 views


From Predictive Chemistry

Jump to: navigation, search

User environment

Zeroth, set up your own system (for linux), by adding a host def for circe to your $HOME/.ssh/config file. Then you don't have to keep typing in circe's full path and your username when running ssh from linux.

Host rc
  User <username on circe>
  ServerAliveInterval 30
  ServerAliveCountMax 120
  ForwardX11 yes

You may need:

 mkdir -p $HOME/.ssh && chmod 700 $HOME/.ssh
 vi $HOME/.ssh/config # ':q' to exit
 chmod 600 $HOME/.ssh/config

The slurm job status command is squeue. A helpful alias to monitor your own jobs is,

 alias myq="squeue -u $USER"


Here's a basic template for queuing a job using MPI (here 2 whole nodes and 6h max run-time).

#SBATCH -J test
#SBATCH -N 2 -t 6:00:00
module load mpi/openmpi/1.4.5  compilers/intel/11.1.064
start=`date +%s`
mpirun parallel-executable
end=`date +%s`
echo "Job completed in $((end-start)) seconds."

By default, slurm jobs start in the same directory that sbatch was invoked.

A few more useful options are:

#SBATCH -o output_log_name.log
#SBATCH --mem=2000

These will specify the name of the output log file instead of the default (names auto-generated from the job number, e.g. slurm-22425.out). By default, the log files include both standard output and standard error from the job.

Submit with

sbatch -p saturn

The -p saturn selects the default queue and can be left out. Other possible values select different node partitions, and are:

  • "jupiter": 444 cores; preemptable by "deadline" QOS (currently inactive).
  • "saturn": 280 cores; default; preemptable by "deadline" QOS (currently inactive).
  • "neptune": 168 cores; preemptable by "deadline" QOS (currently inactive).
  • "hii_broad": 80 cores; testing "contributor" hardware pool; preemptable by "hii_broad" QOS (active).
  • "titan": 16 cores; no preemption; 128 GB RAM for large memory jobs.
  • "pluto": 8 cores; no preemption.

For execution and status, use squeue, or the myq command, defined above as an alias in .bashrc.

For more info, see LLNL's Slurm Quickstart Guide.

Here's a run-down on some of the environment variables available during running (for scripting) see sbatch's manpage for more:

  • SLURM_JOB_NAME - Name of the job.
  • SLURM_JOB_ID - The ID of the job allocation.
  • SLURM_CPUS_ON_NODE - Number of CPUS on the allocated node.
  • SLURM_JOB_NODELIST - List of nodes allocated to the job in a compressed format.
  • SLURM_JOB_NUM_NODES - Total number of nodes in the job’s resource allocation.
  • SLURM_JOB_CPUS_PER_NODE - Count of processors available to the job on this node.
  • SLURM_SUBMIT_DIR - The directory from which sbatch was invoked.
  • SLURM_JOB_PARTITION - Name of the partition in which the job is running.
  • SLURM_LOCALID - Node local task ID for the process within a job.
  • SLURM_GTIDS - Global task IDs running on this node. Zero origin and comma separated.