SPACK Mirror on JLab CUE

Overview

SPACK is a package manager used to maintain multiple versions of software compiled at JLab. The spack manager takes care of many stages of managing the software packages used for the scientific program. It keeps track of multiple software versions built using multiple compilers and even with multiple dependency lists. For example, you can have a version of gemc that uses root v6.18/00, GEANT 10.1.02, and was built with the gcc 9.2.0 compiler. You can also build another version which changes any or all of those version numbers and spack will happily organize it.

Mostly, we want to use spack to centrally manage some standard builds of commonly used software packages. This avoids every researcher from having to build their own copies which can be costly in storage, computing, and their own time. This will include external, 3rd party packages like CLHEP and internal software like gemc.

There are three primary use cases for the software built with the spack system:

1. Users on the JLab CUE want to use the pre-built binary versions on JLab computers
2. Users running offsite want to access the binaries through /cvmfs
3. Users want to install the pre-built binaries on their local computer so they can run untethered

Instructions for using the software in each of these modes are given in the sections below.

Spack repository management

Organizational Overview

The organization of the spack binaries is as follows:

1. Packages are built using singularity containers
   • Containers bind the /scigroup/cvmfs subdirectory to be at /cvmfs/oasis.opensciencegrid.org/jlab inside the container
   • This allows absolute paths that start with /cvmfs to be used in the build/install process
   • The /scigroup/cvmfs/epsci directory is exported to CVMFS so it can be mounted read-only from anywhere
2. The CUE mounts CVMFS (under /cvmfs as is standard) so that CUE users can access the software there (i.e. not through /scigroup/cvmfs/epsci)
3. The packages are exported to a build cache accessible from https://spack.jlab.org/mirror
   • They can also be accessed from file:///scigroup/spack/mirror if on a computer that mounts /scigroup

Creating a new Singularity Image

For the purposes of this system, the Singularity images used for building packages are derived from Docker images. This ensures that either Docker or Singularity can be used to build packages with spack. Thus, if someone needs to build another package, they can choose the container system most convenient for them. Docker images are posted on Docker Hub where Singularity can easily pull them. (Docker images cannot be easily created from Singularity images.)

The Dockerfiles used to create the Docker images are kept in the git-hub repository "epsci-containers". They are also copied into the image itself so one can always access the Dockerfile used to create an image via /container/Dockerfile.*. The Docker images are created with a few system software packages installed. Mainly a C++ compiler, version control tools (e.g. git and svn), python, and a couple of other tools needed for building packages. Below is an example of a Dockerfile (click right-hand side to view).

EXAMPLE Dockerfile. (Click "Expand" to the right for details -->):

#--------------------------------------------------------------------------
# ubuntu build environment
# 
# This Dockerfile will produce an image based on the one used for running
# at NERSC, PSC, and the OSG, but which can also be used to mount CVMFS
# using any computer. The main use case is to provide a simple way to
# mount and run software from /group/halld/Software on you local laptop
# or desktop.
#
# To use this most effectively:
#
#      docker run -it --rm jeffersonlab/epsci-ubuntu cat /container/dsh | tr -d "\r" > dsh
#      chmod +x ./dsh
#      ./dsh jeffersonlab/epsci-ubuntu
#
#--------------------------------------------------------------------------
#
#   docker build -t epsci-ubuntu:21.04 -t jeffersonlab/epsci-ubuntu:21.04 .
#   docker push jeffersonlab/epsci-ubuntu:21.04
#
#--------------------------------------------------------------------------   

FROM ubuntu:21.04
 
# Python3 requires the timezone be set and will try and prompt for it.
ENV TZ=US/Eastern
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

# Install compiler and code management tools
RUN apt -y update \
	&& apt -y install build-essential libssl-dev libffi-dev python-dev \
	&& apt -y install python python3 git subversion cvs curl

COPY dsh /container/dsh
COPY Dockerfile /container/Dockerfile
RUN ln -s /root /home/root
RUN ln -s /root /home/0

CMD ["/bin/bash"]

To create a singularity image, one first needs to create a Docker image. Thus, one needs access to a computer with Docker installed. This generally needs to be a personal desktop or laptop since Docker requires root access and is therefore not available on the public machines like ifarm. (Incidentally, singularity also requires root privileges in order to build an image from a recipe, but not if just pulling from an existing Docker image). Here is example of the steps you might go through if creating an image for a new version of ubuntu. This assumes you are starting on a computer with Docker installed and running.

1. git clone https://github.com/JeffersonLab/epsci-containers
2. cd epsci-containers/base
3. cp Dockerfile.ubuntu.21.04 Dockerfile.ubuntu.18.04
4. edit Dockerfile.ubuntu.18.04 to replace the version numbers with the new ones. They appear in a lot of places so better to do global replace
5. docker build -t epsci-ubuntu:18.04 -t jeffersonlab/epsci-ubuntu:18.04 -f Dockerfile.ubuntu.18.04 .
6. docker push jeffersonlab/epsci-ubuntu:18.04
7. ssh ifarm
8. module use /apps/modulefiles
9. module load singularity
10. cd /scigroup/spack/mirror/singularity/images
11. singularity build epsci-ubuntu-18.04.img docker://jeffersonlab/epsci-ubuntu:18.04
12. git clone https://github.com/spack/spack.git /scigroup/cvmfs/epsci/ubuntu/18.04

The last step above will clone a new spack instance that corresponds to the new image.

Building a spack package with a Singularity (or Docker) container

The preferred method of building new packages is to use one of the ifarm computers with a singularity container from the /scigroup/spack/mirror/singularity/images directory. Any packages built should also be exported to the build cache so they are accessible for offsite installations. Below is an example recipe that builds zlib for the ubuntu 21.04 platform using the native gcc10.2.1 compiler:

1. ssh ifarm1901
2. module use /apps/modulefiles
3. module load singularity
4. singularity shell -B /scigroup/cvmfs:/cvmfs/oasis.opensciencegrid.org/jlab -B /scigroup:/scigroup /scigroup/spack/mirror/singularity/images/epsci-ubuntu-21.04.img
5. source /cvmfs/oasis.opensciencegrid.org/jlab/epsci/ubuntu/21.04/share/spack/set-env.sh
6. spack compiler find
7. spack install zlib%gcc@10.2.1 target=x86_64
8. cd /scigroup/spack/mirror
9. spack buildcache create -r -a -u -d . zlib%gcc@10.2.1
10. spack buildcache update-index -k -d /scigroup/spack/mirror

Be careful that the singularity image you use matches the spack root directory (i.e. where you source the set-env.sh script).

You also want to specify the x86_64 target so generic binaries are built that do not contain optimizations for specific processors.

Finally, don't forget to run the last two commands above to add the package to the build cache and to update the index.

Mac OS X

Some binaries are available for the macosx platform. One issue here is that multiple versions of the Apple supplied compiler are available. This complicates things since one would need to maintain a complete set of builds for multiple compilers in order to support the multiple OS versions. To simplify things, we instead use a compiler installed by spack itself to build the packages. This gives end users access to the compiler which can be used consistently regardless of the exact Mac OS X system version you are using. Note that we also compile the packages with the generic x86_64 target for a similar reason: to be independent of the exact flavor of CPU being used. Thus, the packages are built with:

compiler: gcc 10.2.0 target: x86_64

package	compiler	notes
curl@7.74.0	apple-clang@12.0.0	spack install curl%apple-clang@12.0.0 target=x86_64
clhep@2.4.4.0	apple-clang@12.0.0	spack install clhep%apple-clang@12.0.0 target=x86_64
xerces-c@3.2.3	apple-clang@12.0.0	spack install xerces-c%apple-clang@12.0.0 target=x86_64
gcc@10.2.0	apple-clang@12.0.0	spack install gcc@10.2.0%apple-clang@12.0.0 target=x86_64 n.b. Only some packages will build using this compiler
xerces-c@3.2.3	apple-clang@12.0.0	spack install xerces-c%apple-clang@12.0.0 target=x86_64