Revision as of 16:49, 20 December 2020

Overview

SPACK is a package manager used to maintain multiple versions of software compiled at JLab. The spack manager takes care of many stages of managing the software packages used for the scientific program. It keeps track of multiple software versions built using multiple compilers and even with multiple dependency lists. For example, you can have a version of gemc that uses root v6.18/00, GEANT 10.1.02, and was built with the gcc 9.2.0 compiler. You can also build another version which changes any or all of those version numbers and spack will happily organize it.

Mostly, we want to use spack to centrally manage some standard builds of commonly used software packages. This avoids every researcher from having to build their own copies which can be costly in storage, computing, and their own time. This will include external, 3rd party packages like CLHEP and internal software like gemc.

There are three primary use cases for the software built with the spack system:

Users on the JLab CUE want to use the pre-built binary versions on JLab computers
Users running offsite want to access the binaries through /cvmfs
Users want to install the pre-built binaries on their local computer so they can run untethered

Instructions for using the software in each of these modes are given in the sections below.

Using spack packages on the JLab CUE

Using spack packages offsite via CVMFS

Installing the spack binary packages on your local computer

Spack repository management notes

Organizational Overview

The organization of the spack binaries is as follows:

Packages are built using singularity containers
- Containers bind the /scigroup/spack/cvmfs subdirectory to be at /cvmfs inside the container
- This allows absolute paths that start with /cvmfs to be used in the build/install process
- The /scigroup/spack/cvmfs is exported to CVMFS so it can be mounted read-only from anywhere
The CUE mounts CVMFS (under /cvmfs as is standard) so that CUE users can access the software there (i.e. not through /scigroup/spack)

git clone https://github.com/spack/spack.git --branch v0.16.0 v0.16.0

Mac OS X

Some binaries are available for the macosx platform. One issue here is that multiple versions of the Apple supplied compiler are available. This complicates things since one would need to maintain a complete set of builds for multiple compilers in order to support the multiple OS versions. To simplify things, we instead use a compiler installed by spack itself to build the packages. This gives end users access to the compiler which can be used consistently regardless of the exact Mac OS X system version you are using. Note that we also compile the packages with the generic x86_64 target for a similar reason: to be independent of the exact flavor of CPU being used. Thus, the packages are built with:

compiler: gcc 10.2.0 target: x86_64

package	compiler	notes
curl@7.74.0	apple-clang@12.0.0	spack install curl%apple-clang@12.0.0 target=x86_64
clhep@2.4.4.0	apple-clang@12.0.0	spack install clhep%apple-clang@12.0.0 target=x86_64
xerces-c@3.2.3	apple-clang@12.0.0	spack install xerces-c%apple-clang@12.0.0 target=x86_64
gcc@10.2.0	apple-clang@12.0.0	spack install gcc@10.2.0%apple-clang@12.0.0 target=x86_64 n.b. Only some packages will build using this compiler
xerces-c@3.2.3	apple-clang@12.0.0	spack install xerces-c%apple-clang@12.0.0 target=x86_64

@@ Line 1: / Line 1: @@
 = Overview =
-[https://spack.io/ SPACK] is a package manager used to maintain multiple versions of software compiled at JLab. The spack manager takes care of many stages of managing the software packages used for the scientific program at the lab. It keeps track of multiple software versions built using multiple compilers and even with multiple dependency lists. For example, you can have a version of gemc that uses root v6.18/00, GEANT 10.1.02, and was built with the gcc 9.2.0 compiler. You can also build another version which changes any or all of those version numbers and spack will happily organize it.
+[https://spack.io/ SPACK] is a package manager used to maintain multiple versions of software compiled at JLab. The spack manager takes care of many stages of managing the software packages used for the scientific program. It keeps track of multiple software versions built using multiple compilers and even with multiple dependency lists. For example, you can have a version of gemc that uses root v6.18/00, GEANT 10.1.02, and was built with the gcc 9.2.0 compiler. You can also build another version which changes any or all of those version numbers and spack will happily organize it.
-Mostly, we want to use spack to centrally manage some standard builds of commonly used software packages. This avoids every researcher from having to build their own copies which can be costly in storage, computing, and time. This will include external, 3rd party packages like CLHEP and internal software like gemc.
+Mostly, we want to use spack to centrally manage some standard builds of commonly used software packages. This avoids every researcher from having to build their own copies which can be costly in storage, computing, and their own time. This will include external, 3rd party packages like CLHEP and internal software like gemc.
+There are three primary use cases for the software built with the spack system:
+# Users on the JLab CUE want to use the pre-built binary versions on JLab computers
+# Users running offsite want to access the binaries through [https://cernvm.cern.ch/fs/ /cvmfs]
+# Users want to install the pre-built binaries on their local computer so they can run untethered
+Instructions for using the software in each of these modes are given in the sections below.
+== Using spack packages on the JLab CUE ==
+== Using spack packages offsite via CVMFS ==
+== Installing the spack binary packages on your local computer ==
+= Spack repository management notes =
+== Organizational Overview ==
+The organization of the spack binaries is as follows:
+# Packages are built using singularity containers
+#* Containers bind the /scigroup/spack/cvmfs subdirectory to be at /cvmfs inside the container
+#* This allows absolute paths that start with /cvmfs to be used in the build/install process
+#* The /scigroup/spack/cvmfs is exported to CVMFS so it can be mounted read-only from anywhere
+# The CUE mounts CVMFS (under /cvmfs as is standard) so that CUE users can access the software there (i.e. '''not''' through /scigroup/spack)
+git clone https://github.com/spack/spack.git --branch v0.16.0 v0.16.0
 == Mac OS X ==
@@ Line 38: / Line 66: @@
 | spack install xerces-c%apple-clang@12.0.0 target=x86_64
 |}
-=== Using the prebuilt binaries ===
-The binaries can be accessed in two different ways: via the mirror interface on the web or via CVMFS.

Difference between revisions of "SPACK Mirror on JLab CUE"