Scientific Computing

From Computer Center Documentation

Jump to: navigation, search

The Scientific Computing group in JLab's Information Technology (IT) Division supports the experimental and theoretical Physics computing environments. The group develops and operates JASMine and Auger- the lab's mass storage system and offline experimental physics computing environment, and the Lattice QCD parallel clusters LQCD wiki pages.


The CentOS 5 interactive and batch farm have reached end of life and will be retired on Monday October 14. Users requiring further access need the approval of their Hall Computing Coordinator, with only a few nodes to remain up to allow final close-out. All CentOS 5 resources will be turned off at the end of December 2013.

Want to stay informed? Check out the newest documentation. Also, subscribe to the jlab-scicomp-briefs email.


Experimental Physics Computing Overview Documentation

Fall 2012 Software Upgrade

New Jasmine

  • New Jasmine - View documents and new features of the new Jasmine.

Disk Storage and Management

Cache system and volatile disk pool commands

  • Jcache - Get files from the tape to the cache disks.
  • Jvolatile - Get volatile information of a specified project.

Modified Auger System

Auger is the software that manages the batch farm. It utilizes PBS (Portable Batch System) as underlying batch queuing system. PBS is an open source resource manager providing control over batch jobs and computing nodes. Auger is a front end for the batch system that enables users to submit jobs to the batch system and the ability to stage input files to and from the compute nodes.

Current auger system has been modified in order to utilize the new cache management system. In the modified auger system, all the commands such as jsub have the same syntax as the current Auger system. However, the behavior of jsub has been changed for submitting multiple jobs. In the current Auger, jsub reports status of each submitted job one after another until all submitted jobs are processed. In the modified Auger, jsub will report status of all submitted jobs in one XML block, which contains detailed information for every jobs, after all jobs are processed by the Auger system. This increases the performance of submitting large number of jobs dramatically.


Auger is the software that manages the batch farm. It utilizes PBS (Portable Batch System) as underlying batch queuing system. PBS is an open source resource manager providing control over batch jobs and computing nodes. Auger is a front end for the batch system that enables users to submit jobs to the batch system and the ability to stage input files to and from the compute nodes.

The auger commands should be in any CUE system's path which is /site/bin.

Interactive Nodes

The "ifarm" alias (host ifarm1102) provides interactive user access to the CentOS 6.2 environment for compiling code to be submitted to the CentOS 6.2 farm nodes. Use the OS: centos62 tag with jsub to submit jobs to the CentOS 6.2 farm.

END-OF-LIFE 30-SEP-13: The "ifarml64" alias (hosts ifarm1101) provides interactive user access to the CentOS 5.3 environment for compiling code to be submitted to the CentOS 5.3 farm nodes.

Batch Farm Cluster

The Batch Farm cluster currently contains 112 nodes running either CentOS 5.3 or CentOS 6.2 (80 Centos 5.3, 32 CentOS 6.2).

We have set the maximum job cap to 300 for regular user and 450 for group users (like clasg13, claseg4, etc.). This means that a maximum of 300 or 450 jobs depend on the type of user can be run per user at one time. There are three queues, prod64, priority, and longJob, set up for this cluster.

prod64 Queue

Jobs with all tracks such as analysis, simulation, etc., will go to production queue. The default and maximum walltime are 1 days (24 hours) and 3 days (72 hours).

priority Queue

Jobs submitted using debug and test tracks will go to priority queue. The default cpu time is 30 minutes and default walltime is 4 hours for the priority queue. Only use debug or test track when submitting a short job. There is a maximum of 32 concurrent priority jobs that can run at one time. There is a limit of 16 queued jobs at any one time per user. Expect strange failures if submitting over 16 at one time.

longJob Queue

Jobs submitted using theory track will go to longJob queue. The maximum walltime for this queue is 7 days. At this moment, max 50 jobs per system and 20 jobs per user are configured for longJob queue (we can increase these numbers if necessary). Use theory track only when the job can't finish in 3 days.

Auger Commands

  • jobstat - A summary of jobs status.
  • jsub - Submit jobs to the batch farm (no change).
  • jkill - Delete queued or stop executing jobs.
  • farmhosts - Query the status of the batch farm nodes.
  • farmqueues - Query the status of the batch farm queues.
  • qstat - Query the PBS batch system status directly.
  • tracejob - List detailed information about a job (including completed jobs).

Input/Output from the Auger/PBS System

Default behaviors of Auger-PBS system is that the owner of a job will get a email when the job abort no matter email option is used or not. User can turn on/off the job end email (request level) from xml version of jsub file as well as non xml jsub file (new in auger-pbs). To turn the email notification, just add "MAIL:" in the jsub file. The notification email will be different with Auger-LSF. It only contains the job information (see below).

PBS Job Id: 2666.farml210
Job Name:   TestJob1
Execution terminated

After a job finish, auger-pbs will copy a JOB_NAME.JOB_ID.out file to user's $HOME/.farm_out directory. It will contains any stdout from user's script (not captured by user) and auger's preamble and postamble. On the bottom of this file, there is a job resource usage summary (see below).

Resource usage summary:

If (only if) there is any stderr from user's job (not captured by user), user will find a JOB_NAME.JOB_ID.err file under the $HOME/.farm_out directory. Both these files will be cut in the middle if they are larger than 2MB.

The fairshare of a job is determined by account (clas, halla, hallb, etc) that is specified in jsub file by the PROJECT directive. All of the projects are mapped to one account. Please use correct project name when submit the job. Project's weight is set higher than user's weight (10 to 1). And the priority of production group account user is set higher than regular user. This means if use same project, the production group account user will have higher priority and/or the user who used less process hour will have higher priority. Turing the fare share behave to the best state is a on going process (there are many parameters in the Maui scheduler are available to change). We will continue to adjust these parameters as needed during first few months to a year.

Time Limit
The maximum time limit on the batch farm is 72 hours and the default is 24 hours, the Time: xxxx option (in minutes) must been used to use a time limit other than the default. Any number larger than the maximum time (currently it is 4320 minutes) is included in the Time option, jsub will fail with this error message:

qsub: Job exceeds queue resource limits MSG=cannot satisfy queue max walltime requirement

Batch Farm


Do I have to clean the .farm_out directory periodically?

Yes, Auger will always put a file named JOB_NAME.JOB_ID.out in .farm_out directory under user's home after a job finish. Although these files are small (normall size is a few KB and maxmum size is 2MB), after thousands and ten thousands jobs these .out files will add up to a significant size, which may cause some unexpected problems. Cleaning .farm_out directory once a month or after a massive of job is recommended.

Why my job is killed by PBS server?

PBS sever will kill the job if it exceed the resource limit. Most common case is exceed time limit (walltime). In this case following error message will appear in the file $HOME/.farm_out/JOBNAME.JOBID.err file.

=>> PBS: job killed: walltime xxx exceeded limit xxxx

The default walltime limit is set to 24 hours and maximum time limit is 72 hours (3 days). The TIME option must be used if the job will run more than one day. The default memory allocation is 160MB. If your job needs more memory, you have to use MEMORY option in the jsub file. Otherwise it will been killed by server and you can find this message in the $HOME/.farm_out/JOBNAME.JOBID.err file.

=>> PBS: job killed: vmem xxxxx exceeded limit xxxxxxx
Disk quota exceeded error

Why can I not remove a file and get "Disk quota exceeded" errors

Because of a bug in the Solaris zfs (underlying filesystem) quota system the only way to remove files when the disc quota exceeded message show up is to first truncate the file, then remove it. To simplify this process there is a zrm command located in /site/bin. Usage is zrm file1 ... fileN.

How do I submit a multithread job ?

Using CPU: X tag to request X cores in a job. See Auger Examples for more information.


Jasmine is the software that manages data stored on tapes in the IBM TS3500 tape library. It also interfaces directly with users and disk management systems.

Jasmine Commands

  • jcancel - Cancel a tape library request.
  • jget/jput - Copies files into and out of the tape library.
  • jremove - Mark files for deletion from the tape library.
  • jcert - Manage scientific computing certificate.

Mass Storage System Documentation

  • Cache Disk - Cache Disk information, use, and layout.
  • Work Disk - Work Disk information, use, and layout.
  • Jasmine Terms - Common terms and definitions used in these documents.
  • Data Paths - How data is moved through the network to and from tape

Experimental Program Computing and Storage Requirements

The experimental program at Jefferson Lab uses the mass storage system and data analysis cluster for storing and processing experimental data. Each experimental hall has a Hall Computing Coordinator who works with both the Experiment Computing Coordinators or PI and the Scientific Computing group to plan, acquire, and allocate required resources.

The Hall Computing Coordinators are

  • Hall A - Ole Hansen, Bob Michaels
  • Hall B - Dennis Weygand
  • Hall C - Steve Wood, Brad Sawatzky
  • Hall D - Mark Ito

Additionally, Graham Heyes oversees all of Experimental Physics' offline computing and is the interface to the Information Technology division.

The basic requirements experiments need to provide are

  • Data Storage: expected raw, simulated, and production data to be stored in the mass storage system, and online disk requirements,in Terabytes
  • Computing: SPEC CINT2000 hours requested for simulation, and production (replay, cooking, and analysis)
  • Data to be imported/exported to/from JLab, in Terabytes
  • Timeline that resources are needed

These requirements are used to form the lab's experimental offline computing plan and budgeting.

News Archives

Technical Computing Seminars

Please forward requests/ideas/suggestions for upcoming talks to

  • Wed May 5-6, 2009 9am - Intel compiler, vTune training
  • Thu Jan 15, 2009 10:30am - A Cross-Input Adaptive Framework for Optimizing GPU Programs - Xipeng Shen, W&M
  • Wed Nov 14, 2008 noon - Lattice QCD and Nuclear Physics from Jaguar - Balint Joo
  • Wed Oct 17, 2007 1:30pm - Java Agent based Control System - Vardan Gyurjyan
  • Wed Sep 12, 2007 2pm - Multi-Threading Performance on Commodity Multi-core Processors - Jie Chen
  • Wed May 9, 2007 2pm - Web Services and Ajax - Bobby Lawrence
  • Wed Apr 11, 2007 2pm - Intermediate C++ - Elliott Wolin
  • Wed Mar 14, 2007 2pm - Programming Contests - Michael Haddox-Schatz
Personal tools