Use JANA2 + GPU in singularity container

Here are some instructions for building a Singularity container that can access NVidia GPU hardware.

These instructions for building the image were developed on a JLab CUE desktop that has /apps mounted. In principle, they can also be used on the ifarm machines, but some of the image files can be multiple GB in size which tends to be more of an issue. There are also some system packages that may need to be installed so having sudo privilege is useful. Once the image file is created, it can be moved to other computers and run there without issues.

First, make sure the squashfs tools are installed since singularity will needs these to create the image files:

 sudo yum install squashfs-tools

Next, setup to use singularity from the CUE via the /apps network mounted directory:

 module use /apps/modulefiles
 module load singularity/3.9.5

Create a singularity image from the official nvidia/cuda images. Here, I chose 11.4.2 because it is closest to the CUDA version on gluon200, which I will be targeting for the test. Note that the devel version is semi-large (~3GB) but includes the gcc 9.3.0 compiler.

 singularity pull docker://nvidia/cuda:11.4.2-devel-ubuntu20.04

This should leave you with a file named something like cuda_11.4.2-devel-ubuntu20.04.sif.

Copy the file to the computer with available GPUs and CUDA drivers installed and test it like this:

singularity run -c --nv tensorflow_latest-gpu.sif