Migrating from Colossus to Deep Thought

Migrating from Colossus HPC to Deep Thought HPC means migrating workload management from SGE to Slurm.

SGE was Sun Grid Engine, and is now Oracle Grid Engine.

Slurm is the Simple LinUx Resource Manager, and increasingly used as a de facto standard. The principles remain the same, just changes in commands and some syntax.

The Slurm quick start guide is very helpful to acquaint yourself with the most used commands. Slurm has also produced the Rosetta Stone - a document that shows equivalents between several workload managers.

We have compiled a list of the most basic job commands in SGE, and the Slurm equivalents below.

Command differences - SGE to Slurm

Differences between job scripts in SGE and SLURM

Quick links

Using Slurm

Slurm quick start guide

Rosetta Stone

Command differences - SGE to Slurm

Basic user commands

User commands	SGE	SLURM
Interactive login	qlogin	srun --pty bash or srun (-p "partition name"--time=4:0:0)
Job submission	qsub [script_file]	sbatch [script_file]
Job deletion	qdel [job_id]	scancel [job_id]
Job status by job	qstat -u \* [-j job_id]	squeue [job_id]
Job status by user	qstat [-u user_name]	squeue -u [user_name]
Job hold	qhold [job_id]	scontrol hold [job_id]
Job release	qrls [job_id]	scontrol release [job_id]
Queue list	qconf -sql	squeue
List nodes	qhost	sinfo -N OR scontrol show nodes
Cluster status	qhost -q	sinfo
GUI	qmon	sview

Environmental commands

User commands	SGE	SLURM
Job ID	$JOB_ID	$SLURM_JOBID
Submit directory	$SGE_O_WORKDIR	$SLURM_SUBMIT_DIR
Submit host	$SGE_O_HOST	$SLURM_SUBMIT_HOST
Node list	$PE_HOSTFILE	$SLURM_JOB_NODELIST
Job Array Index	$SGE_TASK_ID	$SLURM_ARRAY_TASK_ID

Job specification commands

User commands	SGE	SLURM
Script directive	#$	#SBATCH
queue	-q [queue]	-p [queue]
count of nodes	N/A	-N [min[-max]]
CPU count	-pe [PE] [count]	-n [count]
Wall clock limit	-l h_rt=[seconds]	-t [min] OR -t [days-hh:mm:ss]
Standard out file	-o [file_name]	-o [file_name]
Standard error file	-e [file_name]	e [file_name]
Combine STDOUT & STDERR files	-j yes	(use -o without -e)
Copy environment	-V	--export=[ALL \| NONE \| variables]
Event notification	-m abe	--mail-type=[events]
send notification email	-M [address]	--mail-user=[address]
Job name	-N [name]	--job-name=[name]
Restart job	-r [yes\|no]	--requeue OR --no-requeue (NOTE: configurable default)
Set working directory	-wd [directory]	--workdir=[dir_name]
Resource sharing	-l exclusive	--exclusive OR --shared
Memory size	-l mem_free=[memory][K\|M\|G]	--mem=[mem][M\|G\|T] OR --mem-per-cpu= [mem][M\|G\|T]
Charge to an account	-A [account]	--account=[account]
Tasks per node	(Fixed allocation_rule in PE)	--tasks-per-node=[count]
		--cpus-per-task=[count]
Job dependancy	-hold_jid [job_id \| job_name]	--depend=[state:job_id]
Job project	-P [name]	--wckey=[name]
Job host preference	-q [queue]@[node] OR -q [queue]@@[hostgroup]	--nodelist=[nodes] AND/OR --exclude= [nodes]

qstat and squeue comparison commands

User commands	SGE	SLURM
	qstat	squeue
	qstat -u username	squeue -J jobname
	qstat -f	squeue -al
	qsub	sbatch
	qsub -N jobname	sbatch -J jobname
	qsub -m beas	sbatch --mail-type=ALL
	qsub -M fan@flinders.edu.au	sbatch --mail-user=fan@flinders.edu.au
	qsub -l h_rt=24:00:00	sbatch -t 24:00:00
	qsub -pe dmp4 16	sbatch -p node -n 16
	qsub -l mem=4G	sbatch --mem=4000
	qsub -P projectname	sbatch -A projectname
	qsub -o filename	sbatch -o filename
	qsub -e filename	sbatch -e filename
	qsub -l scratch_free=20G	sbatch --tmp=20480

Examples of differences between job scripts in SGE and SLURM

Single core application

SGE	SLURM
#!/bin/bash #$ -N test #$ -j y #$ -o test.output #$ -cwd #$ -M FAN@flinders.edu.au #$ -m bea # Request 5 hours run time #$ -l h_rt=5:0:0 # #$ -l mem=4G # your_application	#!/bin/bash # #SBATCH -J test #SBATCH -o test.output #SBATCH -e test.output # Default in slurm #SBATCH --mail-user FAN@flinders.edu.au #SBATCH --mail-type=ALL # Request 5 hours run time #SBATCH -t 5:0:0 #SBATCH --mem=4000 #SBATCH -p core -n 1 your_application

MPI application

SGE	SLURM
#!/bin/bash # #$ -N test #$ -j y #$ -o test.output #$ -cwd #$ -M FAN@flinders.edu.au #$ -m bea # Request 5 hours run time #$ -l h_rt=5:0:0 #$ -R y #$ -pe dmp4 16 #$ -l mem=2G # memory is counted per process on node module load module1 module2 ... mpirun your_application	#!/bin/bash # #SBATCH -J test #SBATCH -o test.output #SBATCH -e test.output # Default in slurm #SBATCH --mail-user FAN@flinders.edu.au #SBATCH --mail-type=ALL # Request 5 hours run time #SBATCH -t 5:0:0 # #SBATCH -p node -n 16 # module load module1 module2 ... mpirun your_application