Getting Started - Farm Jobs

Using the Jefferson Lab Farm

Running Production Replays on the Farm

• Navigate to your hallc_replay_XEM group location. If you do not have local version of the hallc_replay_XEM go to the hallc_replay_XEM page and follow the setup instructions (not created yet).
• In a separate terminal, ssh to cdaq and go_analysis. You will need to be inside the firewall or have access to the hallcgw via 2FA.
  • secure copy the DBASE/SHMS/standard.kinematics and DBASE/HMS/standard.kinematics to your local hallc_replay_XEM directory. You will over-write the existing standard.kinematics files in your group location. Do not screw this up as you could overwrite the counting house files and there is likely no backup. This will give the angle, momentum, and target to hcana to run on the new data. Without this parameter file you will likely get an error 'no gpbeam in database!'.
• Once in the standard.kinematics files have been added, we need to TAR the hallc_replay_XEM directories to be copied to the local farm node via hcswif. Use the following command from the directory containing hallc_replay_XEM: cd hallc_replay_XEM/ && tar -czf ../hallc_replay_XEM.tar.gz . && cd -
  • ls and you will see the hallc_replay_XEM.tar.gz file in your current group directory. hcswif currently assumes the tar file is in your group directory as: /group/c-xem2/$USER/

You have now created the hallc_replay_XEM tar file with all the relevant replay parameters. This will be copied to the farm node along with the raw EVIO file. We need to specify which runs to replay. Currently, this is done by checking which replays have been completed and running all the replays that are available on /mss/ that come after said replay. For example, if run /mss/hallc/xem2/analysis/ONLINE/REPLAYS/SHMS/PRODUCTION/shms_replay_production_19076_-1.root is the last run replayed, we want the runs that come after this run to be replayed. One can ls the /mss to see which runs have been copied from the CDAQ computers to /mss. ls -1 /mss/hallc/xem2/raw/shms_all_*.dat / This will list the files in order and lets say the last one available is 19136, then we can replay runs 19077 - 19136. The next section discusses the quick guide to creating a job and submitting it in swif2.

hcswif.py can take a space separated list of run number and file size where each line is a new shms or hms run to replay on the farm. The file size is important to allocate enough local disk space on the farm node. Additionally, with the large variation in SHMS file size changes how long it takes to replay the SHMS data. hcswif.py will use the file size to specify the walltime given to the job on the farm. For the HMS you will need to specify the time parameter to hcswif.py The common hcswif script is located in: /group/c-xem2/software/XEM_v2.0.0/hcswif/ For more guidance on hcswif use hcswif.py --help.

To get a space separated file with file size and run numbers, I will use that stat command. stat --print="%n %s\n" /cache/hallc/xem2/raw/shms_all_*.dat You cannot get the true file size from the /mss/ tape stub unless you look inside the stub file and extract the "size=" row.


Save the file under run-lists/ in the hcswif directory. Make sure you add the following directories to your /farm_out/ location: hallc_replay_XEM_STDERR hallc_replay_XEM_STDOUT Note, you must run this on an ifarm computer. Now, we can submit a farm job. ./hcswif.py --mode REPLAY --spectrometer SHMS_PROD --run file run-lists/shms_on_cache_12_17_22.dat --name SHMS_PROD_12_17_22 --events -1 --account hallc This will create a json file under the jsons directory. jsons/SHMS_PROD_12_17_22.json We can tell the farm what to run with the following swif2 command: swif2 import -file jsons/SHMS_PROD_12_17_22.json This will create the workflow, we will the run the job with the command: swif2 run SHMS_PROD_12_17_22

Now we will wait until the job finishes or fails! If the job fails ask Casey for help.




Documentation

This wiki is to be used as a conglomerate of resource links and practice. The documents here are not necessarily the most up-to-date, but it serves as a starting point for new users to get familiar with the JLab HPC environment and get some hands-on practice. Here is a list of useful information:

Farm Usage

Brad's famous JLab Compute Resources "How-to"

Farm Users Guide

Analyzer Information

Ole's 2019 Hall A/C Analyzer Software Overview

2018 Joint A/C Analysis Workshop

hcana docs

Overview

All current tasks in the XEM2 group require submitting many single-core jobs to the farm using either SWIF or AUGER. hcswif is used to submit replay jobs run-by-run to the farm nodes to run in parallel using SWIF (Outlined in the Farm Users Guide). Auger is used to submit multiple single-core jobs that do not need to access the tape library. This includes running multiple mc-single-arm instances, or running rc-externals with multiple cores. The following example(s) are in support of the XEM2 use case.

AUGER

• Practice submitting stuff...

SWIF

• Practice submitting stuff...

Using hcswif

hcswif is used to submit many analysis jobs based on run-number.

1. cache.sh in u/group/c-xem2/software/hcswif/run-lists/tools to pull replays from tape to cache (using jcache)
   1. Sample usage: cache.sh

Troubleshooting

Common commands and difficulties with jobs

Common Failure Modes