New User Guide

HIBU (Slurm HPC)

Cluster details

HIBU is a high-performance computing (HPC) system consisting of four compute nodes organized into two partitions. Job scheduling is managed by the Slurm scheduler. A dedicated head (login) node allows users to prepare scripts, submit/cancel jobs, and transfer data. HIBU uses a NAS with a total capacity of up to 42 TB.

Partitions

long short smallhm
  • Number of nodes: 3
  • CPU cores per node: 24
  • Threads per core: 2
  • Available RAM per node: 256GB
  • List of node hostnames: node1,  node2,  node3
  • Number of nodes: 1
  • CPU cores per node: 20
  • Threads per core: 2
  • Available RAM per node: 126GB
  • List of node hostnames: node4
  • Number of nodes: 1
  • CPU cores per node: 16
  • Threads per core: 2
  • Available RAM per node: 256GB
  • List of node hostnames: node5

HIBU in a nutshell

1. Access & First Login

If you do not own an account:
  1. Request an account: Send an e-mail to request a new account. Provide affiliation, intended workloads, software needs.
  2. Set up VPN: Connect to IMBB VPN to be able to access HIBU through SSH.
  3. Download Google Authenticator on your smartphone and scan the QR code you have received from the administrator upon your account creation.

If you already own an account:

Windows users

  1. Download PuTTY on your personal computer,
  2. Connect to IMBB VPN (when you are out of office)
  3. Fill in your username, correct hostname/IP and port to PuTTY (as shown in the figure).
  4. When you are connected, you will be asked to type your password and a verification code from Google Authenticator on your smartphone.

Figure 1

Linux/iOS users

For Ubuntu/Debian/IOS: Open the console application

Type ssh [my-username]@139.91.75.141

When you are connected, you will be asked to type a code from Google Authenticator on your smartphone.

Figure 1

You are currently connected to the cluster’s head node. This node is not for running analyses; use it only to prepare jobs, data, and scripts. Running intensive workloads here can disrupt or crash the gateway that all users rely on to access the cluster. Submit analyses through the scheduler from compute nodes only.

Figure 1

2. Filesystems & Data Storage

!ATTENTION Do not run analyses that read input data from, or write output to, your $HOME directory. Use /data instead. !

Data transfers

3. SLURM Job Scheduler

3.1 Introduction

Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.

3.2 Commands you will use a lot

User tools include srun to initiate jobs, scancel to terminate queued or running jobs, sinfo to report system status, squeue to report the status of jobs, and sacct to get information about jobs and job steps that are running or have completed. There is an administrative tool scontrol available to monitor and/or modify configuration and state information on the cluster.

Arguments:

You can find more documentation for SLURM scheduler at: https://slurm.schedmd.com/documentation.html

3.3 Basic types of jobs

A) Single Node: serial (1 core) or multicore (>1 cores)

You can request only a 'slice' of a node, leaving space for other users to utilize the remaining resources of a node. To achieve this, you should specify both the number of cores you need and the corresponding memory (RAM). See example below:

#!/bin/bash
#SBATCH --name=”hello”   
#SBATCH --partition=long
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1       	# binds only 1 core out of the available
#SBATCH --mem=2G		# binds only 2GB out of the available
#SBATCH -o slurm-%j.out

python myscript.py --input /data/{{USER}}/data/input.dat --out  /data/{{USER}}/results/out.txt

If you need to bind the entire node, you could omit flags in bold. SLURM will automatically bind the entire node to your job.

B) Multinode: Job array

If your type of analysis allows to split your input data into chunks and process them separately, you can use the following example to create multiple jobs at once either in the same node or across different nodes of the same partition.

#!/bin/bash
#SBATCH --name="jobarray"
#SBATCH --partition=long
#SBATCH --nodes=1			# or more
#SBATCH --cpus-per-task=1	# binds 1 core out of the available for each chunk
#SBATCH --mem=2G		# binds 2GB of RAM for each chunk
#SBATCH --array=1-10		# 10 chunks
#SBATCH -o logs/array_%A_%a.out

PARAMS=$(sed -n "${SLURM_ARRAY_TASK_ID}p" params.txt)

python simulate.py $PARAMS --seed ${SLURM_ARRAY_TASK_ID} --out results/${SLURM_ARRAY_TASK_ID}.json

C) Dependencies (run B after A finishes OK)

jid1=$(sbatch jobA.slurm | awk '{print $4}')
sbatch --dependency=afterok:${jid1} jobB.slurm

3.4 How to request the right resources

Pro tip: Avoid --exclusive unless you truly need the whole node; it reduces throughput and priority.

3.5 Monitoring & Logs

History & usage:

sacct -j <jobid> --format=JobID,JobName%20,Partition,State,Elapsed,Timelimit,AllocTRES%30,ReqMem,MaxRSS,ExitCode

Logs

Stdout/stderr go to slurm-%j.out by default; customize with -o and -e.

3.7 Best Practices & Etiquette

3.8 Troubleshooting

You can send us an e-mail for troubleshooting if you face issues running your job at: hpc-support@imbb.forth.gr, including: jobID(s), script(s), slurm-<jobid>.out, pathways to your scripts, and a minimal reproducer if possible.

3.9 Policy Highlights

3.10 Quick Reference (Cheat Sheet)

Task Command
Submit job 
sbatch job.slurm
Live queue 
squeue -u $USER
Job details 
scontrol show job <jobid>
Cancel job 
scancel <jobid>
Node/partition info 
sinfo -o '%P %a %l %D %C %G %F'
Interactive alloc 
salloc -p {{DEFAULT_PARTITION}} -t 02:00:00 --cpus-per-task=4 --mem=8G
Array job 
#SBATCH --array=1-100%10
Account/QoS 
#SBATCH -A {{SLURM_ACCOUNT}} / #SBATCH --qos={{DEFAULT_QOS}}
Efficiency 
seff <jobid> (if available)

3.11 Templates (copy & adapt)

Minimal job script

#!/bin/bash
#SBATCH --name="hello"   
#SBATCH --partition=long
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1       	# binds only 1 core out of the available
#SBATCH --mem=2G		# binds only 2GB out of the available
#SBATCH -o slurm-%j.out

python myscript.py --input /data/{{USER}}/data/input.dat --out  /data/{{USER}}/results/out.txt

Conda in job

If you have installed any tool on your account through CONDA environments

#!/bin/bash
#SBATCH -J py
#SBATCH -A {{SLURM_ACCOUNT}}
#SBATCH -p {{DEFAULT_PARTITION}}
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G

conda activate ENVIRONMENT_NAME	# env should always be reactivated 
mafft --help 	# then call the tool installed within the env

Singularity

To download and convert Docker images into Singularity container:

#!/bin/bash
#SBATCH --name=”containers”
#SBATCH --partition=short
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=32G

singularity pull docker://staphb/mafft:latest # this creates a file named "mafft_latest.sif"

To download and convert Docker images into Singularity container:

#!/bin/bash
#SBATCH --name=”containers”
#SBATCH --partition=short
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=32G

singularity exec ./mafft_latest.sif mafft --help # To execute a command from the container
singularity run ./mafft_latest.sif # To run the container script

For more information on how to use Singularity you can see https://docs.sylabs.io/guides/latest/user-guide/.