Difference between revisions of "Run Group F"
| Line 120: | Line 120: | ||
| − | == RC:Sebastian Kuhn == | + | == RC: Sebastian Kuhn == |
=== RC phone: === | === RC phone: === | ||
Revision as of 15:04, 2 September 2020
Shift ScheduleShift ChecklistHot CheckoutBeam Time Accounting
|
Manuals |
Procedures |
JLab Logbooks
|
RC schedule Weekly form
|
If calling local numbers (with area code prefix 757) from the counting room dial 9+[last 7 digits]. To call other area codes dial 91+[10 digit number]
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
- Note, all non-JLab numbers must be dialed with an area code. When calling from a counting-house landline, dial "9" first.
- To call JLab phones from outside the lab, all 4-digit numbers must be preceded by 757-269
- Click Here to edit Phone Numbers. Note, you then also have to edit the current page to force a refresh.
Click Here to edit Phone Numbers. Note, you then also have to edit this page to force a refresh.
In case you need to call in any electronics help, call in the following order
| Person | Phone Number |
|---|---|
| Chris Cuevas | (757) 865-0461 |
| Mark Taylor | (757) 846-7322 |
| Armen Stepayan | (757) 814-5622 |
CLAS12 Run Group F
Part I (Spring 2020): Beam energy 2.18 GeV (1 pass) and10.38 GeV (5 pass)
Part II (Summer 2020): Beam energy 2.18 GeV (1 pass) and10.38 GeV (5 pass)
Important: Document all your work in the logbook! For every run, make comment at run start and run end (including "ignore" if warranted), enter RELEVANT screen shots *), reset monitoring plots and accumulate monitoring data for about 10-20 min with good beam (on!), then post monitoring plots (reset if beam is bad and wait another 10-20 min). Also take screenshot of RTPC in CED at least once per run with beam
RC: Sebastian Kuhn
RC phone:
- 757-575-7540
- 9 757-575-7540 from Counting Room.
- If you can't reach the RC on the RC phone, use the personal phone numbers (Mohammad: (423) 596-8352, Sebastian: (757) 639-6640)
- RC meeting held at 12 pm online. Bluejeans 757 683 5804
- Number of Events Spreadsheet RIght now manually, until somebody with the knowledge and the time write an script to get the number of events from SQL --> RCDB
PDL: Stepan Stepanyan
- (757) 876-1789
- 9 876-1789 from Counting Room
NEW SHIFT INFORMATION - PLEASE READ
Due to travel restrictions, only local people will be able to come to Jefferson Lab for their shifts. To deal with this situation, Jefferson Lab leadership has developed a new POLICY that requires only an expert shift taker during Day and Swing shifts (Owl shifts will still be covered by 2 people). There will be remote backup, at least partial presence by the Run Coordinators, and all counting houses have been equipped with a direct camera feed so that shift takers for ANY hall can monitor the counting houses of Hall B and Hall D when only 1 person is present. Shift Experts will carry a cell phone that can be used to call in help. IF YOU ARE ALONE ON SHIFT, you may not go into the Hall - call the relevant experts instead. Shift experts during Day and Swing are expected, in addition to their regular duties, to also check monitoring histograms and CED and log them (as well as standard screen shots) to the logbook, and to respond to any alarms, as well as fill out shift checklists. These new rules take effect on Tuesday, March 24.
Starting immediately, please follow these rules: Keep a distance of 6 ft from all other people AT ALL TIMES. In particular, expert and worker (when both present) should sit 6 feet apart, facing in different directions. In addition, only the RC OR an RTPC expert should be inside the counting house B at any time - a maximum room occupancy of THREE has to be obeyed at all times. The RC, PDL and shift expert have the authority to ask non-essential people to leave if overcrowding occurs. Any work that requires 2 people to work in closer proximity than 6' must be first approved.
At present, we expect the experimental run to continue until September 21, 2020. Obviously, this puts extra burden on many people, and we are grateful especially for those local volunteers who have signed up for shifts. (It's still not too late to sign up for some empty slots!!!) Please also read the relevant information on COVID-19 and MEDCON5 operations at Jefferson Lab: [2]
======================================================================================================================
A. Requests
1. Please end a run whenever the status of the 1/2 waveplate changes
MCC should announce any changes, but sometimes they won't do that. The Hall requesting the 1/2 waveplate change should contact us first with a time estimate - at that point, please watch the status of the 1/2 waveplate on the Beam Overview screen and stop the run as soon as it changes.
There will be an audible Beamline alarm when the half-wave plate (HWP) state changes. Please end the ongoing run and start a new run.
2. For every run (~2 hours)
- In the begin run dialog, enter the requested beam current, the state of the half-wave plate (IN or OUT) and any conditions changed since the last run (target, B-fields, HVs, DAQ config, trigger, ...). In the end run dialog, check boxes ("junk" etc.) and add to the dialog any significant changes during the run.
- *) Near the beginning of the run, submit the following screenshots (can be all in one elog entry) while beam is on. Sample plots for reference:
- Run clas12mon over at most 10-20 min with good beam(!) (reset first) and submit the occupancy plots (at least once per run, early in the run). Whenever you restart clas12mon (and if you haven't checked it yet), go to the TriggerBits menu and select "ElectronOr". Compare with previous monitoring plots and the reference WITHOUT FT and make a log entry about any changes you observe. Make sure to reset monitoring plots regularly.
- Take at least one screen shot of RTPC CED with beam per run and post in logbook. De-select "Auto-event" and use Cntrl-N to browse for an event with tracks in it.
If you change the current, make sure that...
- The total trigger rate remains below 2 kHz
- The MVT1 and MVT2 ROCs stay below 5-6% busy
- The deadtime stays below 10%
- The DC occupancy for RI in all 6 sectors stays below 4% (detailed monitoring plots)
- RTPC HV remains stable (no trips).
3. Serious note about setting the trigger rates alarm limits
When the data taking conditions are changing (going from one target to another, H2,D2,4He, empty, or changing the beam current or something else) the shift has to adjust the trigger rates limits that wile be used during new data taking. The procedure is shown at the pictures:
- Press MENU buttom at the trigger GUI
- Choose ALARMS SETTINGS - Expert
- When the beam is stable press SET ALL NORMALIZED RATE LIMITS
... You are done. Detailed procedures with pictures can be found here [8]
4. PLEASE DO NOT IGNORE ANY ALARM and follow the guidance for each alarm notification.
================================================================================================================================================
B. Run Plan (see also initial run plan)
General plan: Follow a sequence of 6 Days D2 (240 nA), 2 shifts H2 (240 nA), 1 2-hour run empty (250 nA) and 10M events on 4He (240 nA).
* Continue taking data on D2
DAQ config: PROD66 , Trigger file: rgf_inbending_v4_1.trg (.cnf)
- Continue taking data on Full D2 target (~ 67 psig) and 240 nA beam current.
- Please write the target pressure and comments in the dialogue box when starting/stopping a run
- Start a run and check that live time is above 85%. If not decrease current and log it.
- Restart a new run every 2h OR 10M events. END A RUN if beam will be off > 15 min, RTPC HV trip, or whenever half-wave plate is changed!
- Submit the list of screenshots at the beginning and at the end of every run.
- Keep BTA (beam time accounting) up to date, including reasons for any entries with experiment not ready.
- When checking the ECAL response, make sure to look at the histograms for each sector by using the sector buttons at the bottom of the clas12mon window - if adcecal* or/and adcpcal* ROC crashes, there will be no alarm and the histograms are the way to notice this issue.
- For trigger alarms check Note 14 first!
* At the end of each run, follow the standard DAQ restart sequence:
- -> "end run", "cancel", "reset"
- a. if the run ended correctly: -> "download", "prestart", "go"
- b. if the run did not end correctly or if any ROCs had to be rebooted:-> "configure", "download", "prestart", "go"
======================================================================================================================
C. References and Standards:
* Establish beam in the Hall ( CALL RC )
- ALL the detectors must be off.
- Ask MCC to energize the tagger magnet.
- Ask MCC for 5 nA and check the beam quality using harp scans 2C21 and 2C24, following the procedures described in [9]
- When the beam quality is acceptable, Call MCC and ask for a low-intensity (~5nA) beam to FC, complete harp scans. Procedures described in [10]. We require less than 0.3 mm (1σ) beam widths in x and y (As measured by 2H01A harp) and halo below 10^−4 (As measured by 2H01A harp).
- Center beam on target, using rates in BOM and downstream beam monitors. Establish range where beam starts directly interacting with target walls and take middle.
- Engage orbit locks and ascertain that they are stable.
- Set alarms and interlocks.
- Call the MCC and tell them that the beam is good and must keep the beam current at 5 nA.
- TURN HV ON for all the detectors.
Checking the beam profiles and positions
- Doing harp scans on FC:
- All HV are off (including FTOF)
- Check that 20 mm collimator is in
- Call MCC to unmask U/M/D counters but keep BOM masked or get it masked in case it isn't
- Put thresholds on U/M/D as 1M/1M/1M and dwell time 50 ms.
- Call MCC to put 5nA to the FC.
- Compare the 2c21, 2c24, 2H01 positions and beam overview screen to the reference below
- follow the procedures for the harp scans at 2H01 [11] (also do harp scans on 2C24)
- Compare the results with the references scans below (be aware to use the correct analysis counter)
- if what you get is similar to the references, call MCC and tell them the the beam is acceptable and set FSD thresholds and dwelltimes as written below and continue with production running
- if what you get isn't similar to the references or If you are not sure if a tune of the beam is needed, consult beamline expert and/or RC.
Default solenoid current and B field 1800 A (~3.78 T)
Nominal Beam Positions (last update Aug 29th, 2020 @ 02:20 AM)
- 2H01, X: +0.2 mm, Y: -1.3 mm
- 2C24, X: -3.7 mm, Y: -1.7 mm
- 2C21, X: -0.4 mm, Y: 0.3 mm
FSD Thresholds
- Upstream: 2,000 Hz
- Midstream: 2,000 Hz
- Downstream: 500,000 Hz
- BOM: 2,600 Hz
FSD Dwell (Integration) Time
- Upstream: 5 ms
- Midstream: 5 ms
- Downstream: 5 ms
- BOM: 5 ms
BONuS HV setting
- Reference Harp Scans for Beam on Tagger Dump: 2C21 [12], 2C24 [13]
- Reference Harp Scans for Beam on Faraday Cup: 2C21 [14], 2C24 [15][16], 2H01 [17].
Reference Monitoring Histograms 5 nA production trigger [18]
Reference Scalers and Halo-Counter Rates:
- Well-tuned CW beam on FC, Full LD2 and 4He target, CLAS12 ON: 5 nA [19], 50 nA [20], 240nA [21]
- Well-tuned CW beam on FC, Empty Target, CLAS12 OFF: 5 nA [22]
- Well-tuned beam during Moller run [23]
- Well-tuned beam on Tagger Dump at 5 nA during harp scans: [24]
======================================================================================================================
D. Beam Restoration after major retuning
Updated Manual is at [25]
======================================================================================================================
F. General Instructions:
- The main lights in the Hall (dome lights) and the Forward Carriage lights are being kept off because of light leaks affecting some of the detectors. If these lights are switched on during an access, they should be switched off when leaving the Hall. Note that the dome lights when switched off cannot be turned back on immediately because they require 10-15 min to cooldown.
- Get familiar with the shift crew responsibilities for the RTPC (see manual [26]). Especially on how to switch on the HV after a trip.
- Do not run more than 30 minutes above 30 nA with 5-pass beam without the beam blocker in front of Faraday cup. Put beam blocker in for long running at high currents for 5 pass operations
- Turn DC HV off only for beam tuning; if no beam is available or when beam is stable, keep them on even if you are not taking data.
- With any issue contact On-Call Experts or RC - do not spend more than 15-20 min trying to fix the problem.
- Check for and read any comments to log-book entries made during your shift.
- During Day shift, check once per day with RC or RTPC expert whether they have purged the target.
Once Per Day (At 13:00):
- Complete RICH recovery procedure - see Note: 11 below
Every Shift:
- Follow run plan as outlined by RC
- If any concern about beam stability, ask MCC if orbit locks are on (they should be).
- Keep shift summary up to date in HBLOG. Record all that happens.
- Document any beam condition change and send scaler GUIs to HBLOG
- Fill out BTA hourly. Click "Load from EPICS" to automatically fill the left side.
- Upload spectra from clas12mon at beginning of each run and cross-check with reference spectra
- Reset clas12mon every ~2 hours or after extensive beam down and inspect spectra
- Fill and submit the shift checklist in the logbook
- Note 1: FTOF HVs: The goal is to minimize the number of power cycles of the dividers.
- should be turned off during the initial beam tuning down to the Faraday Cup after CLAS12 has been off for a long shutdown, when doing a Moller run, when doing harp scans, or if there is tuned/pulsed beam in the upstream beamline.
- should be left on after an initial beam tune has been established and if there are only minor steering adjustments and “tweaks” being made.
- if shift workers have doubts what to do with the HVs, they can always contact the TOF on-call expert for advice.
- Note 2: In case of a Torus and/or Solenoid Fast Dump do the following:
- Notify MCC to request beam OFF and to drop Hall B status to Power Permit
- Call Engineering on-call
- Make separate log entry with copies to HBTORUS and HBSOLENOID logbooks. In the "Notify" field add Ruben Fair, Probir Goshal, Dave Kashy and esr-users@jlab.org
- Notify Run Coordinator
- Turn off all detectors
- Note 3: When beam is being delivered to the Faraday Cup:
- the Fast Shut Down elements: Upstream, Midstream, Downstream, BOM, and Solenoid should always be in the state UNMASKED
- No changes to the FSD threshold should be made without RC or beamline expert approval
- Note 4: Any request for a special run or change of configuration has to be approved by the RC & documented
- Note 5: Carefully check the BTA every hour and run the script btaGet.py to print for you what HAS TO BE in BTA for this hour. Edit BTA if it is incorrect.
- Note 6: Reset CLAS12MON frequently to avoid histogram saturation. If CLAS12MON is restarted change MAX scale to 10 (default is 5) for DC normalized occupncy histograms. This would prevent them from saturation. See log entry [27]
- Note 7: Shift workers must check the occupancies! Here is the standard sets of monitoring histograms WITHOUT the FT. Between August 13 and August 18 7 a.m., please use THIS link for the proper reference WITH FT.
- Note 8: Check the vacuum periodically, make sure vacuum is not higher than 5e-5
- Note 9: Always reset the CFD threshold after all power off/on on the CND CAMAC crate
After the CAMAC crate (camac1) holding the CND CFD boards is switched off for any reason, it is mandatory to reset the associated thresholds typing the following command from any clon machine terminal: $CODA/src/rol/Linux_x86_64/bin/cnd_cfd_thresh -w 0 If this command is failing and the crate is not responding, reboot it as follows: roc_reboot camac1
- Note 10: Shift workers should anyway check routinely scalers to verify they update correctly and make a logbook entry if anomalies are observed after starting a new run.
- Note 11:
For RICH recovery procedures, please see Log entry https://logbooks.jlab.org/entry/3562273. This would apply in the cases of 1. DAQ crash: rich4 is not responding or 2. RICH alarms (LV,missing tile, temperature etc). If it does not work or you are uncertain about what to do, contact the RICH expert on call. Please, note that missing tiles typically occur due to lost communication. Keep in mind that the recovery procedure will kill DAQ. If DAQ is running for other purpose, rather than data taking (for which the RICH acceptance would be important) do not initiate the recovery procedure.
- The most critical parameter for RICH is the temperature of the photosensors. If the temperature rises above limits, an interlock will automatically turn the RICH HV and LV. If this happens, notify the expert on call and keep taking data without RICH.
- Note 12: If ROCs mvt1 and/or mvt2 are 100% busy: stop the run, power cycle FMT/BONUS FEU LV (accessible from RTPC Overview or FMT Overview, FMT/BONUS - All LV OFF, FMT/BONUS - All LV ON, HV stays ON), then roc_reboot mvt1, roc_reboot mvt2, wait until it reboots ('ssh mvt1' and 'ssh mvt2' to verify), press 'cancel', 'reset', in the run control GUI, proceed with a new run from 'configure'. If the first LV recycle / roc_reboot doesn’t work call RTPC/BONUS expert.
- Note 13: If ROC mmft1 is 100% busy: power cycle FTT LV (FT->FTT Overview, HV stays ON), roc_reboot mmft1, wait until it reboots (ssh mmft1 to verify). If the first LV recycle/mmft1 reboot doesn’t work call FT expert
- Note 14: Trigger Troubleshooting: If you get trigger alarms check first if the trigger bit of a sector alarms both, for the DC roads and non DC roads, in the trigger expert screen (DAQ --> CLAS1 Trigger --> Menu --> Table - Expert). If this is the case (i.e. sector 3 and bits 3 and 10), there might be a problem with HTCC, PCAL or ECAL rocs and you should check the rocs in the runcontrol. If the non DC trigger bit is working and only the one with DC alarms, you would need to check the DC trigger rocs (possibly the one from the alarming sector). If you are not sure, call RC or trigger expert.
Webcams |
Hall Webcams(the first time you open any camera in any browser, accept the security exception) |
Manuals |
Epics on the web
|
Fall 2018 Run
|
|
|
Spring 2018 Run
|
|
|
Remote Shift Schedule and Signup
Bluejeans
- Permanent connection for communication and screen sharing between remote and local shift workers.
- https://bluejeans.com/950041022
- Or, on a clonpc## as user=clasrun, in a terminal run "bluejeans"
- For audio in the counting house, for now we can use one of the rarely-used phones.
WebOPI
- Requires CUE login
- Beamline Overview
- BONuS RTPC Overview
Strip Charts
- Requires CUE login
- Upstream/Midstream Halo Counters
- Downstream Halo Counters
- Beam Currents
- BPM Positions
- Note: All BPM axes now go from -2.0 to +1.0 mm
- empty
Monitoring Histograms
- Changing the reference run/logentry requires updating the URL by the Run Coordinator
- Reference, current, and 5 most recent runs for runs WITHOUT the FT.
- For runs WITH the FT (August 13 - August 18 7 a.m), please use the following link: FT ON
For Shift Expert
- Bluejeans is running on clonpc14 - DO NOT CLOSE THE SESSION
- The bluejeans session should always share the screen, it is the lower monitor of clonpc14. The alarm handler must be displayed on that monitor to allow remote shifter to keep eye on it.
- Audio for the bluejeans session goes through the phone, 269-5247. This is the phone attached on the second row of tables (former RC station), behind the shift expert station.
- DO NOT HANG UP or MUTE the phone, if you have to mute then do not forget to unmute.
- About every 6 hours the bluejeans will disconnect the phone if this happens, use the microphone tab in the bluejeans window and choose "Switch to phone audio" then select "Call me" and give the phone number, 757 269 5247.
|
|
![[3]](https://logbooks.jlab.org/files/2020/08/3819334/clonpc15_20200802_082100.gif){kind=link}
![[4]](https://logbooks.jlab.org/files/2020/08/3819288/clonpc15_20200802_065630.gif){kind=link}
![[5]](https://logbooks.jlab.org/files/2020/08/3819288/clonpc15_20200802_065634.gif){kind=link}
![[6]](https://logbooks.jlab.org/files/2020/08/3819288/clonpc15_20200802_065636.gif){kind=link}
![[7]](https://logbooks.jlab.org/files/2020/08/3819288/clonpc15_20200802_065643.gif){kind=link}
![[21]](https://logbooks.jlab.org/files/2020/02/3790604/clonpc16_20200223_223424.gif){kind=link}