Difference between revisions of "Run Group F"

From clas12-run
Jump to: navigation, search
(Remote Shifts)
(Schedule)
Line 134: Line 134:
  
 
===<font color=blue>''* Requests''</font>===
 
===<font color=blue>''* Requests''</font>===
<!-- # Monitor Hurrican Isaias - reduce gas flow to RTPC to 200 sccm if outside pressure drops drastically.
+
*
-->
+
  
 
===<font color=blue>'' * For every run (~2 hours)''</font>===
 
===<font color=blue>'' * For every run (~2 hours)''</font>===
Line 148: Line 147:
 
# Run clas12mon over '''at most''' 20-30 min '''with good beam(!)''' (reset first) and submit the occupancy plots (at least '''once''' per run, '''early in the run'''). Compare with previous monitoring plots and make a log entry about any changes you observe. Make sure to reset monitoring plots regularly.
 
# Run clas12mon over '''at most''' 20-30 min '''with good beam(!)''' (reset first) and submit the occupancy plots (at least '''once''' per run, '''early in the run'''). Compare with previous monitoring plots and make a log entry about any changes you observe. Make sure to reset monitoring plots regularly.
 
# Take at least one screen shot of RTPC CED with beam per run and post in logbook. De-select "Auto-event" and use Cntrl-N to browse for an event with tracks in it.
 
# Take at least one screen shot of RTPC CED with beam per run and post in logbook. De-select "Auto-event" and use Cntrl-N to browse for an event with tracks in it.
 
  
 
===<font color=blue>''* Serious note about setting the trigger rates alarm limits)''</font>===
 
===<font color=blue>''* Serious note about setting the trigger rates alarm limits)''</font>===
Line 160: Line 158:
 
... You are done.
 
... You are done.
 
Detailed procedures with pictures can be found here [https://logbooks.jlab.org/entry/3819963]
 
Detailed procedures with pictures can be found here [https://logbooks.jlab.org/entry/3819963]
 
 
  
 
===<font color=blue>''I. ESTABLISH BEAM FOR THE FIRST TIME AT 10.4 GeV (5 pass) ''</font>===
 
===<font color=blue>''I. ESTABLISH BEAM FOR THE FIRST TIME AT 10.4 GeV (5 pass) ''</font>===

Revision as of 03:51, 8 August 2020

[edit]

Shift Schedule

Shift Checklist

Hot Checkout

Beam Time Accounting

  • See guidance for BTA in

Beam Time Accounting User Guide.

Manuals

Procedures

JLab Logbooks


RC schedule Weekly form

  • Feb 12 - Feb 18: Mohammad Hattawy
  • Feb 19 - Feb 25: Florian Hauenstein
  • Feb 26 - Mar 3: Eric Christy
  • Mar 4 - Mar 19: Sebastian Kuhn
  • Mar 18 - Mar 23: Carlos Ayerbe

  • RC & RTPC expert list [1]
  • 07/31 - 08/04 : Narbe Kalantarian
  • 08/05 - 08/11 : Stephen Bueltmann
  • 08/12 - 08/18 : Carlos Ayerbe Gayoso
  • 08/20 - 08/21 : Mohammad Hattawy
  • 08/22 - 08/25 : Sebastian Kuhn
  • 08/26 - 09/01 : Mohammad Hattawy
  • 09/02 - 09/08 : Sebastian Kuhn
  • 09/09 - 09/23 : Mohammad Hattawy


Daily Run Group F Coordination Meetings Daily at 10:00 a.m. Bluejeans = 757 683 5804

If calling local numbers (with area code prefix 757) from the counting room dial 9+[last 7 digits]. To call other area codes dial 91+[10 digit number]

Person Phone Number
Run Coordinator 9-757 575-7540 (cell)
Physics Division Liaison 9-757 876-1789 (cell)
Shift Expert 5244 or 9-757 329-4846 (cell)
Shift Worker 5126
MCC-OPS 7048
Crew Chief 7045
Crew Chief 9-757 876-3367 (cell)
Program Deputy 9-757 876-7997 (cell)
Engineering on-call 9-757 748-5048 (cell)
Engineering on-call secondary 9-757 897-9060 (cell)
Radcon (needed for target cell swaps) 9-757 876-1743 (cell)
Gate House Guard 5822
Location Phone Number
Hall-B Floor 5165
Hall-B Space Frame 5170, 5171
Hall-B Forward Carriage 5371
Hall-B Counting Room 5245 (Shift1), 5126 (Shift2)
Hall-B Counting Room 5244 (Expert)
Hall-B Gas Shed 7115
System Phone Number On-Call Person
DC (757) 748-5048 (cell) Engineering On-Call
SVT + MM (757) 541-7539 (cell) Yuri Gotra Primary
SVT + MM (757) 753-7769 (cell) Rafo Paremuzyan (secondary)
ECAL (757) 810-1489 (cell) Cole Smith
FTOF/CTOF (757) 344-7204 (cell) Jose Carvajal
CND (757) 344-7204 (cell) Jose Carvajal
HTCC-LTCC (757) 344-7174 (cell) Youri Sharabian
FT (757) 344-1848 (cell) Raffaella De Vita
BAND (757) 310-7198 (cell) Florian Hauenstein
RICH (757) 344-3235 (cell) Valery Kubarovsky
DAQ (757) 232-6221 (cell) Sergey Boiarinov
Slow Controls (757) 748-6922 (cell) Nathan Baltzell
Beamline (757) 303-3996 (cell) Eugene Pasyuk
Pol. Target On-Call (757) 218 2266 (cell) James Maxwell
Polarized Target (757) 871-5374 (cell) James Brock
Target (not used) (757) 822-9586 (cell) Bob Miller Primary
Target (not used) (757) 897-9060 (cell) Denny Insley Secondary
Counting House C x6000
Counting House D x5504

In case of a problem with superconducting magnets call Engineering on-call first, particularly if it is related to cryo.
For all other superconducting magnet problems call magnet group.
Their schedule and phone numbers are posted on the wall next to the Saclay target computer.




Click Here to edit Phone Numbers. Note, you then also have to edit this page to force a refresh.


In case you need to call in any electronics help, call in the following order

Person Phone Number
Chris Cuevas (757) 865-0461
Mark Taylor (757) 846-7322
Armen Stepayan (757) 814-5622



CLAS12 Run Group F
Part I (Spring 2020): Beam energy 2.18 GeV (1 pass) and10.38 GeV (5 pass)
Part II (Summer 2020): Beam energy 2.18 GeV (1 pass) and10.38 GeV (5 pass)

Important: Document all your work in the logbook! For every run, make comment at run start and run end (including "ignore" if warranted), enter RELEVANT screen shots *), and take about 20-30 min data with good beam, then post monitoring plots (reset if beam is bad and wait another 20-30 min). Also take screenshot of RTPC in CED at least once per run with beam


RC: Stephen Bueltmann

  • 757-575-7540
  • 9 757-575-7540 from Counting Room.
  • RC meeting held at 10am in 2nd Floor Counting House. Bluejeans 757 683 5804

PDL: Stepan Stepanyan

  • (757) 876-1789
  • 9 876-1789 from Counting Room

NEW SHIFT INFORMATION - PLEASE READ

Due to travel restrictions, only local people will be able to come to Jefferson Lab for their shifts. To deal with this situation, Jefferson Lab leadership has developed a new POLICY that requires only an expert shift taker during Day and Swing shifts (Owl shifts will still be covered by 2 people). There will be remote backup, at least partial presence by the Run Coordinators, and all counting houses have been equipped with a direct camera feed so that shift takers for ANY hall can monitor the counting houses of Hall B and Hall D when only 1 person is present. Shift Experts will carry a cell phone that can be used to call in help. IF YOU ARE ALONE ON SHIFT, you may not go into the Hall - call the relevant experts instead. Shift experts during Day and Swing are expected, in addition to their regular duties, to also check monitoring histograms and CED and log them (as well as standard screen shots) to the logbook, and to respond to any alarms, as well as fill out shift checklists. These new rules take effect on Tuesday, March 24.

Starting immediately, please follow these rules: Keep a distance of 6 ft from all other people AT ALL TIMES. In particular, expert and worker (when both present) should sit 6 feet apart, facing in different directions. In addition, only the RC OR an RTPC expert should be inside the counting house B at any time - a maximum room occupancy of THREE has to be obeyed at all times. The RC, PDL and shift expert have the authority to ask non-essential people to leave if overcrowding occurs. Any work that requires 2 people to work in closer proximity than 6' must be first approved.

Until further notice, we expect the experimental run to continue until September 6, 2020. Obviously, this puts extra burden on many people, and we are grateful especially for those local volunteers who have signed up for shifts. Please also read the relevant information on COVID-19 and MEDCON5 operations at Jefferson Lab: [2]

Run Plan (see also initial run plan)

(Last updated: August 6, 2020 - S. Bueltmann)

* Requests

* For every run (~2 hours)

  1. In the begin run dialog, enter the requested beam current and any conditions changed since the last run (target, Bfields, HVs, DAQ config, trigger, ...). In the end run dialog, check boxes ("junk" etc.) and add to the dialog any significant changes during the run.
  2. Near the beginning of the run, submit the following screenshots (can be all in one elog entry) while beam is on. Sample plots for reference:
    1. Beamline overview,
    2. DAQvnc (select "statistics" tab) - [3] (NOT the default window with all the green ROC terminals),
    3. CLAS12 VTP Trigger GUI - [4],
    4. ROC Busies [5],
    5. RTPC overview [6],
    6. BONuS HV [7].
  3. Run clas12mon over at most 20-30 min with good beam(!) (reset first) and submit the occupancy plots (at least once per run, early in the run). Compare with previous monitoring plots and make a log entry about any changes you observe. Make sure to reset monitoring plots regularly.
  4. Take at least one screen shot of RTPC CED with beam per run and post in logbook. De-select "Auto-event" and use Cntrl-N to browse for an event with tracks in it.

* Serious note about setting the trigger rates alarm limits)

When the data taking conditions are changing (going from one target to another, H2,D2,4He, empty, or changing the beam current or something else) the shift has to adjust the trigger rates limits that wile be used during new data taking. The procedure is shown at the pictures:

  1. Press MENU buttom at the trigger GUI
  2. Choose ALARMS SETTINGS - Expert
  3. When the beam is stable press SET ALL NORMALIZED RATE LIMITS

... You are done. Detailed procedures with pictures can be found here [8]

I. ESTABLISH BEAM FOR THE FIRST TIME AT 10.4 GeV (5 pass)

  1. Make sure that the solenoid is at 1800 A (~3.78T) and the torus is at full in-bending field (3770 A). If it is not the case, call Eugene (Beamline Expert: (757) 303-3996).
  2. In parallel to 1, call RTPC expert to set target to “empty” (= 1 atm H2).
  3. Restore beam to the hall, first to the tagger dump, following the procedures described in [9].
  4. After doing the harp scans at 2C21 and 2C24, while the beam is at the tagger, call MCC and inform them that we want to take a Moller run.
  5. Follow the procedures for taking a Moller run [10].
  6. Check beam quality using harp scans 2C21 and 2C24, following the procedures described in [11]
  7. When the beam quality is acceptable, call MCC and ask for a low-intensity (~5nA) beam to FC, Check the procedures below, complete harp scans (Performing Harp Scans). We require less than 0.3 mm (1σ) beam widths in x and y (As measured by 2H01A harp) and halo below 10^−4 (As measured by 2H01A harp).
  8. Center beam on target, using rates in BOM and downstream beam monitors with empty target and low (few nA) beam:
    1. Call MCC and keep the phone line open as you will have to ask for many small displacements of the beam on 2H01
    2. Move the beam in 1 mm steps in x on 2H01, until the rates in BOM, downstream beam monitors and TOF counters start increasing noticeably
    3. Go another 0.5 mm to make sure you are starting to interact with the wall (rates keep increasing). Note x position on 2H01
    4. Go back to the original position, then repeat in the opposite x-direction until you start seeing wall again
    5. Take mean value between 2 edges and record both on this page and the whiteboard in the CH - this will be the nominal x-position on 2H01 from now on
    6. Repeat the exact same procedure in y to establish range where beam starts directly interacting with target walls and take middle. Again, note nominal y-position on 2H01.
  9. Ask MCC to update their orbit locks with the nominal position on 2H01 (as well as the earlier established position on 2C21 and 2C24). Engage orbit locks and ascertain that they are stable.
  10. Set alarms and interlocks.
  11. Call the MCC and tell them that the beam is good and must keep the beam current at 5 nA.
  12. TURN HV ON for all the detectors.


II. Production: Electron-inbending Configuration, D2 target @ 10.4 GeV beam

DAQ config: PROD66; Trigger file:rgf_inbending_v3_4.trg

  1. Call the RTPC expert to fill the target with D2.
  2. Make sure beam blocker is in front of Faraday Cup
  3. After a longer beam-off period (assuming no major retuning of the accelerator), no harp scans are necessary. However, follow these steps to bring beam back:
    1. Ask for 10 nA. Check that all BPMs are within 0.1 mm of their nominal values.
    2. Ask for 50 nA. Check BMPs again and ask for orbit locks to be activated.
    3. Once orbit locks are running, ask for 240 nA.
  4. Ask MCC for 240 nA beam current (NOTE that 2C21A BPM will show 260nA. This is a known issue.)
  5. Please write the target pressure and comments in the dialogue box when starting/stopping a run
  6. Start a run and check that live time is above 85%. If not decrease current and log it.
  7. Restart a new run every 2h OR 10M events. END A RUN if beam will be off > 10 min, RTPC HV trip, or whenever half-wave plate is changed!
  8. Keep BTA (beam time accounting) up to date, including reasons for any entries with experiment not ready
  9. Submit the list of screenshots at the beginning and at the end of every run.
  1. When checking the ECAL response, make sure to look at the histograms for each sector by using the sector buttons at the bottom of the clas12mon window - if adcecal* or/and adcpcal* ROC crashes, there will be no alarm and the histograms are the way to notice this issue.
  2. For trigger alarms check Note 14 first!


Beam Restoration (last updated Feb 19)

Updated Manual is at [12]

  1. IF YOU GO DIRECTLY TO FARADAY CUP!!!: Quick beam recovery procedure including Harp Scan
    1. All HV are off (including FTOF)
    2. Check that 20 mm collimator is in
    3. Call MCC to unmask U/M/D counters but keep BOM masked or get it masked in case it isn't
    4. Call MCC to put 5nA to the FC.
    5. Compare the 2c21, 2c24, 2H01 positions and beam overview screen to the reference below
    6. Put thresholds on U/M/D as 1M/1M/1M and dwell time 50 ms.
    7. follow the procedures for the harp scans at 2H01 [13] (optionally at 2C21 and 2C24)
    8. Compare the results with the references scans below (be aware to use the correct analysis counter)
    9. if what you get is similar to the references, call MCC and tell them the the beam is acceptable and set FSD thresholds and dwelltimes as written below and continue with production running (see above)
    10. if what you get isn't similar to the references, do the other harp scans. If you are not sure if a tune of the beam is needed, consult beam line expert and/or RC.

At the end of each run, follow the standard DAQ restart sequence
(Note, this might be different from the instructions coming up on RunControl.)

  • "end run", "cancel", "reset"
  • if the run ended correctly:
    • "download", "prestart", "go"
  • if the run did not end correctly or if any ROCs had to be rebooted:
    • "configure", "download", "prestart", "go"

References and Standards:

(last update Aug 2nd, 2020 @ 8:40 PM.)

Restoration of Beam to Hall B


Default solenoid current and B field 1800 A (~3.78 T)

Nominal Beam Positions

  • 2H01, X: -0.1 mm, Y: 0.9 mm
  • 2C24, X: -1.0 mm, Y: 0.8 mm
  • 2C21, X: -0.2 mm, Y: 0.1 mm


FSD Thresholds TEMPORARY UNTIL BOM REPARATIONS

  • Upstream: 2000 Hz
  • Midstream: 2000 Hz
  • Downstream: 20000 Hz
  • BOM: 100000

FSD Dwell time

  • Upstream: 5 ms
  • Midstream: 5 ms
  • Downstream: 5 ms
  • BOM: 5 ms

BONuS HV setting

Reference Harp Scans for Beam on Tagger Dump: 2C21 [14], 2C24 [15]

Reference Harp Scans for Beam on Faraday Cup: 2C21 [16], 2C24 [17], 2H01 [18].

Reference Monitoring Histograms 5 nA production trigger [19]

Reference Scalers and Halo-Counter Rates:

  • Well-tuned CW beam on FC, Full LD2 and 4He target, CLAS12 ON: 5 nA [20], 50 nA [21], 240nA [22]
  • Well-tuned CW beam on FC, Empty Target, CLAS12 OFF: 5 nA [23]
  • Well-tuned beam during Moller run [24]
  • Well-tuned beam on Tagger Dump at 5 nA during harp scans: [25]

General Instructions:

  • The main lights in the Hall (dome lights) and the Forward Carriage lights are being kept off because of light leaks affecting some of the detectors. If these lights are switched on during an access, they should be switched off when leaving the Hall. Note that the dome lights when switched off cannot be turned back on immediately because they require 10-15 min to cooldown.
  • Get familiar with the shift crew responsibilities for the RTPC (see manual [26]). Especially on how to switch on the HV after a trip.
  • Do not run more than 30 minutes above 30 nA with 5-pass beam without the beam blocker in front of Faraday cup. Put beam blocker in for long running at high currents for 5 pass operations.
  • Turn DC HV off only for beam tuning; if no beam is available or when beam is stable, keep them on even if you are not taking data.
  • With any issue contact On-Call Experts or RC - do not spend more than 15-20 min trying to fix the problem.
  • Check for and read any comments to log-book entries made during your shift.
  • During Day shift, check once per day with RC or RTPC expert whether they have purged the target.

Once Per Day (At 13:00):

  • Complete RICH recovery procedure - see Note: 11 below

Every Shift:

  • Follow run plan as outlined by RC
  • If any concern about beam stability, ask MCC if orbit locks are on (they should be).
  • Keep shift summary up to date in HBLOG. Record all that happens.
  • Document any beam condition change and send scaler GUIs to HBLOG
  • Fill out BTA hourly. Click "Load from EPICS" to automatically fill the left side.
  • Upload spectra from clas12mon at beginning of each run and cross-check with reference spectra
    • Reset clas12mon every ~2 hours or after extensive beam down and inspect spectra
  • Fill and submit the shift checklist in the logbook


  • Note 1: FTOF HVs: The goal is to minimize the number of power cycles of the dividers.
    • should be turned off during the initial beam tuning down to the Faraday Cup after CLAS12 has been off for a long shutdown, when doing a Moller run, when doing harp scans, or if there is tuned/pulsed beam in the upstream beamline.
    • should be left on after an initial beam tune has been established and if there are only minor steering adjustments and “tweaks” being made.
    • if shift workers have doubts what to do with the HVs, they can always contact the TOF on-call expert for advice.
  • Note 2: In case of a Torus and/or Solenoid Fast Dump do the following:
    1. Notify MCC to request beam OFF and to drop Hall B status to Power Permit
    2. Call Engineering on-call
    3. Make separate log entry with copies to HBTORUS and HBSOLENOID logbooks. In the "Notify" field add Ruben Fair, Probir Goshal, Dave Kashy and esr-users@jlab.org
    4. Notify Run Coordinator
    5. Turn off all detectors
  • Note 3: When beam is being delivered to the Faraday Cup:
    1. the Fast Shut Down elements: Upstream, Midstream, Downstream, BOM, and Solenoid should always be in the state UNMASKED
    2. No changes to the FSD threshold should be made without RC or beamline expert approval
  • Note 4: Any request for a special run or change of configuration has to be approved by the RC & documented
  • Note 5: Carefully check the BTA every hour and run the script btaGet.py to print for you what HAS TO BE in BTA for this hour. Edit BTA if it is incorrect.
  • Note 6: Reset CLAS12MON frequently to avoid histogram saturation. If CLAS12MON is restarted change MAX scale to 10 (default is 5) for DC normalized occupncy histograms. This would prevent them from saturation. See log entry [27]
  • Note 7: Shift workers must check the occupancies! Use this tool to compare to previous runs: [28]
  • Note 8: Check the vacuum periodically, make sure vacuum is not higher than 5e-5
  • Note 9: Always reset the CFD threshold after all power off/on on the CND CAMAC crate

After the CAMAC crate (camac1) holding the CND CFD boards is switched off for any reason, it is mandatory to reset the associated thresholds typing the following command from any clon machine terminal: $CODA/src/rol/Linux_x86_64/bin/cnd_cfd_thresh -w 0 If this command is failing and the crate is not responding, reboot it as follows: roc_reboot camac1

  • Note 10: Shift workers should anyway check routinely scalers to verify they update correctly and make a logbook entry if anomalies are observed after starting a new run.
  • Note 11:

For RICH recovery procedures, please see Log entry https://logbooks.jlab.org/entry/3562273. This would apply in the cases of 1. DAQ crash: rich4 is not responding or 2. RICH alarms (LV,missing tile, temperature etc). If it does not work or you are uncertain about what to do, contact the RICH expert on call. Please, note that missing tiles typically occur due to lost communication. Keep in mind that the recovery procedure will kill DAQ. If DAQ is running for other purpose, rather than data taking (for which the RICH acceptance would be important) do not initiate the recovery procedure.

  • The most critical parameter for RICH is the temperature of the photosensors. If the temperature rises above limits, an interlock will automatically turn the RICH HV and LV. If this happens, notify the expert on call and keep taking data without RICH.
  • Note 12: If ROCs mvt1 and/or mvt2 are 100% busy: stop the run, power cycle FMT/BONUS FEU LV (accessible from RTPC Overview or FMT Overview, FMT/BONUS - All LV OFF, FMT/BONUS - All LV ON, HV stays ON), then roc_reboot mvt1, roc_reboot mvt2, wait until it reboots ('ssh mvt1' and 'ssh mvt2' to verify), press 'cancel', 'reset', in the run control GUI, proceed with a new run from 'configure'. If the first LV recycle / roc_reboot doesn’t work call RTPC/BONUS expert.
  • Note 13: If ROC mmft1 is 100% busy: power cycle FTT LV (FT->FTT Overview, HV stays ON), roc_reboot mmft1, wait until it reboots (ssh mmft1 to verify). If the first LV recycle/mmft1 reboot doesn’t work call FT expert
  • Note 14: Trigger Troubleshooting: If you get trigger alarms check first if the trigger bit of a sector alarms both, for the DC roads and non DC roads, in the trigger expert screen (DAQ --> CLAS1 Trigger --> Menu --> Table - Expert). If this is the case (i.e. sector 3 and bits 3 and 10), there might be a problem with HTCC, PCAL or ECAL rocs and you should check the rocs in the runcontrol. If the non DC trigger bit is working and only the one with DC alarms, you would need to check the DC trigger rocs (possibly the one from the alarming sector). If you are not sure, call RC or trigger expert.

Webcams

Hall Webcams

(the first time you open any camera in any browser, accept the security exception)

Manuals

Epics on the web

Fall 2018 Run

fall2018_charge.jpg

Moller rg a fall.jpeg

Spring 2018 Run

chart.jpeg

Moller measurement rg-a.jpg

Remote Shift Schedule and Signup

Bluejeans

  • Permanent connection for communication and screen sharing between remote and local shift workers.
  • https://bluejeans.com/950041022
  • Or, on a clonpc## as user=clasrun, in a terminal run "bluejeans"
  • For audio in the counting house, for now we can use one of the rarely-used phones.

WebOPI

Strip Charts

Monitoring Histograms

For Shift Expert

  • Bluejeans is running on clonpc14 - DO NOT CLOSE THE SESSION
  • The bluejeans session should always share the screen, it is the lower monitor of clonpc14. The alarm handler must be displayed on that monitor to allow remote shifter to keep eye on it.
  • Audio for the bluejeans session goes through the phone, 269-5247. This is the phone attached on the second row of tables (former RC station), behind the shift expert station. DO NOT HANG UP or MUTE the phone, if you have to mute then do not forget to unmute.


Clas12design.png