This is a quick start to using Singularity on the Sherlock cluster. You should have familiarity with how to log in, and generally use a command line. If you run into trouble, please ask us for help.

Login

Here are the official “How to Connect to Sherlock” docs. Generally you set up kerberos and do this:

ssh <username>@login.sherlock.stanford.edu 

I’m lazy, so my preference is to define this setup in my ~/.ssh/config file. By setting the login node to be a specific one I can maintain a session over time. Let’ say my username is “tacos”

 Host sherlock User tacos Hostname sh-ln05.stanford.edu GSSAPIDelegateCredentials yes GSSAPIAuthentication yes ControlMaster auto ControlPersist yes ControlPath ~/.ssh/%l%r@%h:%p 

Then to login I don’t need to type the longer string, I can just type:

ssh sherlock 

Interactive Node

You generally shouldn’t run anything computationally intensive on the login nodes! It’s not just to be courteous, it’s because the processes will be killed and you have to start over. Don’t waste your time doing this, grab an interactive node to start off:

# interactive node sdev # same thing, ask for different memory or time srun --time 8:00:00 --mem 32000 pty bash 

If your PI has purchased nodes and you have a partition for your lab, this means that your jobs will run a lot faster (you get priority as a member of the group!) and you should use that advantage:

srun --time 8:00:00 --mem 32000 --partition mygroup --pty bash 

If you don’t have any specific partition, the implied one is --partition normal.

Load Singularity

Let’s get Singularity set up! You may want to add this to your bash profile (in HOME) under $HOME/.bashrc (or similar depending on your shell) so that you don’t need to manually do it.

 module use system module load singularity 

A very import variable to export is the SINGULARITY_CACHEDIR. This is where Singularity stores pulled images, built images, and image layers (e.g., when you pull a Docker image it first pulls the .tar.gz layers before assembling into a container binary). What happens if you don’t export this variable? The cache defaults to your HOME, your HOME quickly gets filled up (containers are large files!) and then you are locked out.

don’t do that.

 export SINGULARITY_CACHEDIR=$SCRATCH/.singularity mkdir -p $SINGULARITY_CACHEDIR 

Also note for the above that we are creating the directory, which is something that needs to be done once (and then never again!).

Examples

Now let’s jump into examples. I’m just going to show you, because many of the commands speak for themselves, and I’ll add further note if needed. What you should know that we are going to reference a container with a “uri” (unique resource identifier) and it can be in reference to (the most common forms):

• docker://: a container on Docker Hub
• shub://: a container on Singularity Hub
• container.simg: a container image file

Pull

You could just run or execute a command to a container reference, but I recommend that you pull the container first.

 singularity pull shub://vsoch/hello-world singularity run $SCRATCH/.singularity/vsoch-hello-world-master-latest.simg 

It’s common that you might want to name a container based on it’s Github commit (for Singularity Hub), it’s image file hash, or a custom name that you really like.

 singularity pull --name meatballs.simg shub://vsoch/hello-world singularity pull --hash shub://vsoch/hello-world singularity pull --commit shub://vsoch/hello-world 

Exec

The “exec” command will execute a command to a container.

 singularity pull docker://vanessa/salad singularity exec $SCRATCH/.singularity/salad.simg /code/salad spoon singularity exec $SCRATCH/.singularity/salad.simg /code/salad fork 

 # Other options for what you can do (without sudo) singularity build container.simg docker://ubuntu singularity exec container.simg echo "Custom commands" 

Run

The run command will run the container’s runscript, which is the executable (or ENTRYPOINT/CMD in Docker speak) that the creator intended to be used.

 singularity pull docker://vanessa/pokemon singularity run $SCRATCH/.singularity/pokemon.simg run catch 

Options

And here is a quick listing of other useful commands! This is by no means an exaustive list, but I’ve found it helpful to debug and understand a container. First, inspect things!

 singularity inspect -l $SCRATCH/.singularity/pokemon.simg # labels singularity inspect -e $SCRATCH/.singularity/pokemon.simg # environment singularity inspect -r $SCRATCH/.singularity/pokemon.simg # runscript singularity inspect -d $SCRATCH/.singularity/pokemon.simg # Singularity recipe definition 

Using --pwd below might be necessary if the working directory is important. Singularity doesn’t respect Docker’s WORKDIR Dockerfile command, as we usually use the present working directory.

 # Set the present working directory with --pwd when you run the container singularity run --pwd /code container.simg 

Singularity is great because most of the time, you don’t need to think about binds. The paths that you use most often (e.g., your home and scratch and tmp) are “inside” the container. I hesitate to use that word because the boundry really is seamless. Thus, if your host supports overlayfs and the configuration allows it, your container will by default see all the bind mounts on the host. You can specify a custom mount (again if the administrative configuration allows it) with -B or --bind.

 # scratch, home, tmp, are already mounted :) But use --bind/-B to do custom singularity run --bind $HOME/pancakes:/opt/pancakes container.simg 

I many times have conflicts with my PYTHONPATH and need to unset it, or even just clean the environment entirely.

 PYTHONPATH= singularity run docker://continuumio/miniconda3 python singularity run --cleanenv docker://continuumio/miniconda3 python 

What does writing a script look like? You are going to treat singularity as you would any other executable! It’s good practice to load it in your SBATCH scripts too.

 module use system module load singularity singularity run --cleanenv docker://continuumio/miniconda3 python $SCRATCH/analysis/script.py 

Build

If you build your own container, you have a few options!

• Created an automated build for Docker Hub then pull the container via the docker:// uri.
• add a Singularity recipe file to a Github repository and build it automatically on Singularity Hub.
• Ask for @vsoch help to build a custom container!
• Check out all the example containers to get you started.
• Browse the Containershare and use the Forward tool for interactive jupyter (and other) notebooks.
• Build your container locally, and then use scp to copy it to Sherlock:

scp container.simg <username>@login.sherlock.stanford.edu:/scratch/users/<username>/container.simg 

Have a question? Or want to make these notes better? Please open an issue.