Difference between revisions of "Run Group D"

From clas12-run
Jump to navigation Jump to search
Line 76: Line 76:
 
== Responsibilities ==
 
== Responsibilities ==
  
=== Getting Started Standards  ===
+
* The RC duties are detailed in the [[Media:COO-RGDE.pdf|COO]].
* Ensure the safe and efficient operation of the experiment; see the detailed duties in the [[Media:COO-RGDE.pdf|COO]].
+
* Feel free to contact upcoming shifts about their training or site access status.
* Check the shift schedule and training status of your shift crew (before YOU start!);
+
* The training requirements for the shift are:
:‡ Training Status: [https://misportal.jlab.org/training/people/srl Skill Requirements List]; Fill in the user's name and search.
 
::◊ Training requirements for shift should also be in the [[Media:COO-RGDE.pdf|COO]], and they are:
 
 
:::- ES&H Orientation (SAF 100);<br>
 
:::- ES&H Orientation (SAF 100);<br>
 
:::- Radiation Worker Training (SAF 801);<br>
 
:::- Radiation Worker Training (SAF 801);<br>
 
:::- Oxygen Deficiency Hazard Training (SAF 103);<br>
 
:::- Oxygen Deficiency Hazard Training (SAF 103);<br>
 
:::- Hall B Safety Awareness Walk-Through (SAF111)
 
:::- Hall B Safety Awareness Walk-Through (SAF111)
* Remote shifts are listed [[Upcoming-remote-shifts|here]]
+
* Remote shifts are listed [[Upcoming-remote-shifts|here]].
* Backup shifters are listed [https://jeffersonlab-my.sharepoint.com/:x:/r/personal/baltzell_jlab_org/Documents/Backup%20Shifters.xlsx?d=wf18dc5909cf84424840d73cc9bd8ce8e&csf=1&web=1&e=QwY1fs here]
+
* Backup shifters are listed [https://jeffersonlab-my.sharepoint.com/:x:/r/personal/baltzell_jlab_org/Documents/Backup%20Shifters.xlsx?d=wf18dc5909cf84424840d73cc9bd8ce8e&csf=1&web=1&e=QwY1fs here].
  
 
=== Daily ===
 
=== Daily ===
* Attend the MCC meetings (7:45 AM and 8 AM Daily) ''in-person when running except on the weekend, which are virtual on ZOOM but @ 9 AM''.
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&Dagger; See the ZOOM connection info under {{#switchtablink:Important_Links|Important Links}} tab.
 
  
* Update the run plan in the {{#switchtablink:Short_Term_Schedule|Short Term Schedule}} tab in coordination with the AC and the run group, and communicate it to the shift crew.  
+
* Attend the in-person PD/RC and MCC/OPS meetings (7:45 AM and 8 AM Daily). The PD will communicate when and how meetings happen on the weekend.
 +
** See the ZOOM connection info under {{#switchtablink:Important_Links|Important Links}} tab.
 +
 
 +
* Update the run plan in the {{#switchtablink:Short_Term_Schedule|Short Term Schedule}} tab in coordination with the run group coordinator, and communicate it to the shift crew.
 +
 
 +
* Log the RC summary in the HBLOG.
 +
 
 +
* Send a daily invitation for the RC meeting to hallb, clas_members, and rgd_ana mailing list.
 +
 
 +
* Run the RC meetings at 1 PM in CH Room 200C and on ZOOM, except on Wednesdays.
 +
* On Wednesdays, the RC meeting is on ZOOM ONLY @ 15:30 due to the conflict with the 13:30 MCC meeting and the unavailability of the CH room afterwards.
  
* Run the RC (/OA) meetings at 1 PM in CH Room 200C and on ZOOM, except on Wednesdays, which is in ZOOM ONLY @ 15:30 due to the conflict with the 13:30 MCC meeting and the unavailability of the CH room afterward;
 
 
:&Dagger; To connect to Zoom in CH Room 200C, follow these steps:
 
:&Dagger; To connect to Zoom in CH Room 200C, follow these steps:
 
::&loz; Turn on the projector with the white remote control, then log into your CUE account. If not listed yet, click "Not listed?" in the bottom left to add your account.  
 
::&loz; Turn on the projector with the white remote control, then log into your CUE account. If not listed yet, click "Not listed?" in the bottom left to add your account.  
Line 100: Line 105:
 
::&loz; Since the room doesn't have audio, don't connect with the computer audio, but choose "Call Me," then enter the number in the small Cisco answering device near the microphone in the middle of the conference table. Once it rings, answer the call, and you will be prompted to join the meeting.  
 
::&loz; Since the room doesn't have audio, don't connect with the computer audio, but choose "Call Me," then enter the number in the small Cisco answering device near the microphone in the middle of the conference table. Once it rings, answer the call, and you will be prompted to join the meeting.  
 
::&loz; The room has a camera, so you may turn it on, if you wish, in the ZOOM browser interface.  
 
::&loz; The room has a camera, so you may turn it on, if you wish, in the ZOOM browser interface.  
 
* Check on shift crews at the start of each shift, especially if they are new.
 
 
* Send the RC summary to the HBLOG, hallb, clas_members, and rgd_ana mailing list.<br>
 
  
 
=== Weekly ===
 
=== Weekly ===
 
* Attend in-person the MCC scheduling meeting on Wednesdays at 1:30 PM and report the run status via the submitted [https://ace.jlab.org/apps/pd/run-coordinator-reports Run Coordinator Weekly Summary];
 
* Attend in-person the MCC scheduling meeting on Wednesdays at 1:30 PM and report the run status via the submitted [https://ace.jlab.org/apps/pd/run-coordinator-reports Run Coordinator Weekly Summary];
 +
 
:&Dagger; The report summarizes the data-taking status concerning the available beam used (ABU), any beam losses (BANU), CW on target (ABU+ABNU), etc., from 7 AM past Wed. through 7 AM the Wednesday the report is due, in addition to any standing and critical issues, plans for next week, and general comments/requests, if any.
 
:&Dagger; The report summarizes the data-taking status concerning the available beam used (ABU), any beam losses (BANU), CW on target (ABU+ABNU), etc., from 7 AM past Wed. through 7 AM the Wednesday the report is due, in addition to any standing and critical issues, plans for next week, and general comments/requests, if any.
 
:&Dagger; Our beam time accounting info can be retrieved either from:
 
:&Dagger; Our beam time accounting info can be retrieved either from:

Revision as of 08:20, 28 October 2023

[edit]
Role Phone Number
Hall B Run Coordinator (757) 575-7540 (cell)
Hall B Physics Division Liaison (757) 876-1789 (cell)
MCC 7048
Crew Chief 7045
Crew Chief (757) 876-3367 (cell)
Program Deputy (757) 876-7997 (cell)
RadCon (757) 876-1743 (cell)
Gate House Guard 5822
Location Phone Number
Hall B Floor 5165
Hall B Space Frame 5170 and 5171
Hall B Forward Carriage 5371
Hall B Gas Shed 7115
Hall B Counting House 5244 (Shift Expert)
Hall B Counting House (757) 329-4846 (Shift Expert cell)
Hall B Counting House 5245 or 5126 (Shift Worker)
Hall A Counting House 5501
Hall C Counting House 6000
Hall D Counting House 5504
Hall B System Phone Number On-Call Person
Engineering (757) 748-5048 (cell)
(757) 897-9060 (cell)
Engineering On-Call (primary)
Denny Insley (secondary)
Slow Controls (757) 748-6922 (cell) Nathan Baltzell
Beamline (757) 303-3996 (cell) Eugene Pasyuk
DC (757) 218-4372 (cell)
(757) 748-5048 (cell)
Florian Hauenstein (primary)
Engineering On-Call (secondary)
SVT/MVT/MM (757) 541-7539 (cell)
(757) 753-7769 (cell)
Yuri Gotra (primary)
Rafo Paremuzyan (secondary)
ECAL (757) 810-1489 (cell) Cole Smith
FTOF/CTOF/CND (757) 344-7204 (cell) Daniel Carman
ALERT (757) 329-4844 (cell) Raphael Dupré
HTCC/LTCC (757) 344-7174 (cell) Youri Sharabian
FT (757) 344-1848 (cell) Raffaella De Vita
BAND (757) 310-7198 (cell) Florian Hauenstein
RICH (757) 344-3235 (cell)
(757) 748-6922 (cell)
Christopher Dilks (primary)
Nathan Baltzell (secondary)
DAQ (757) 232-6221 (cell) Sergey Boiarinov
HYDRA (317) 550-9226 (cell) Torri Jeske
Authorized Hall B Solenoid/Torus Operators
Hall B Denny Insley, Morgan Cook, Eugene Pasyuk
Magnet Group Probir Ghosha, Renuka Rajput-Ghoshal
Detector Support Group Brian Eng, Pablo Campero, Tyler Lemon
DC Power Onish Kumar, Sarin Philip
  • Note, all non-JLab numbers must be dialed with an area code. When calling from a counting-house landline, dial "9" first.
  • To call JLab phones from outside the lab, all 4-digit numbers must be preceded by 757-269
  • Click Here to edit Phone Numbers. Note, you then also have to edit the current page to force a refresh.

Click to Edit solely this tab. P.S.: You then have to click on the top Edit, next to Read, followed by "Save changes" to force the current page to refresh.

Run Coordinator: Lamiaa El Fassi (757) 575-7540

PDL: Nathan Baltzell (9-) 757 876-1789; Office: x5902

RG/Analysis Coordinator (AC): Lamiaa El Fassi (630) 605-4259

Daily RC/OA Meetings at 13:00 in CH Room 200C, and on ZOOM: URL Link (Meeting ID: 1611984219; Passcode: RGD-RC23), except on Wednesdays, which is on ZOOM ONLY @ 15:30 due to the conflict with the 13:30 MCC Scheduling/Planning meeting.

Important Notes

Do NOT change FSD settings without approval from RC or beamline expert!

  • In case of medical or fire emergency, call 9-911 from a LANDLINE phone in the Counting House and inform the Crew Chief and RC.
  • In case of evacuation of the Counting Room (e.g., due to a fire alarm), grab the shift expert phone and leave the room through the "EXIT" door leading to the parking lot. Do not leave through the hallway! Call the RC once outside so the latter will call MCC to turn the beam OFF to Hall B and inform other Hall B staff members.
  • If you have symptoms or a positive COVID test, do NOT come to your shift - instead, inform the PDL and the RC and JLab medical services immediately to arrange replacements.
  • Remote shift: If needed, worker shifts can be taken from off-site. Inform the PDL and RC. Also, make sure that you follow the instructions under the "Shift Worker" tab, as you need some lead time to set up!
  • Note to Shift Takers:
◊ Read the ESAD, COO, RSAD, and ERG documents found on the "Current RG-D Run Info" as well as the yellow folder in the counting room. Before you start your first shift on the current experiment, you must sign on the list in the yellow folder. Thereby, you confirm that you have read and understand their content.
◊ Read at least the previous 24 hours logbook entries to be aware of recent run details and changes;
◊ Arrive at least 10 minutes early for shift change;
◊ Refresh the runwiki page/tabs; look over it/them every shift:
- Follow the Shift Expert/Worker tab's instructions.
Do not spend more than 15-20 min trying to fix a problem. In case of any issue, contact the On-call Experts or the RC.
Review these important links:
- Recent Changes and Current Issues
- General How-to for beam restoration procedures and Møller runs.

Run Plan, Thursday, Dec. 14th - Friday, Dec. 15th, 2023

  • Continue CuSn data-taking as follows until 7 AM on Dec. 15th when the beam will be taken away for the RG-K setup:
- Target: CuSn
- Beam Blocker: IN
- Beam current: 100 nA +/- 4 nA on 2C21A
- Run duration: 100 M events
- DAQ configuration: PROD67
- Trigger file: rgd_out_v2.3_Q2_1.2.trg

P.S.: If the beam is widely fluctuating above 105 nA, constantly overshooting over 105 nA on 2C21A, please ask MCC to narrow down the fluctuation margin to keep it below 105 nA based on 2C21A reading. They could put the beam current monitor lock on 2C21A in case that may help with the wide fluctuation and/or adjust our laser phase as well as narrow down the slit width, see ELOG entries 4224300 and 4225220.

General Instructions

  • The main lights in the Hall (dome lights) and the Forward Carriage lights are being kept off because of light leaks affecting some of the detectors (especially ECAL sec 2 and 3). If these lights are switched on during an access, they should be switched off when leaving the Hall.
• Refer to these photos: ON: [1], OFF: [2]
• We have a new dome light switch outside of the hall in the labyrinth. The lights can be turned off without entering the Hall. See logbook entry: https://logbooks.jlab.org/entry/4204460
• Note that when the dome lights are switched off, they cannot be turned back on immediately because they require 10-15 min to cool down.
  • Beamline:
• Ignore the Faraday Cup beam current (see Current Issues).
• Do not run over 30 minutes above 90 nA with a 5-pass beam without the beam blocker IN . The beam blocker must be IN for long-running at high currents for 5-pass operations.
• If there is any concern about beam stability, ask MCC if orbit locks are on (they should be).
• Document any beam condition change and send scaler GUIs to HBLOG.
• If HWP is changed, start a new run:
◊ Please check with the beamline expert whether the SLM and FCup charge asymmetry (Q-Asym (%)) readings are set properly, as they will be out of range in case Eugene's script is not run by the experts after the HWP change. The absolute values should be less than 0.1%.
  • For beam tuning and Møller runs
• Make sure the SVT/MVT are set as follows: SVT: LV ON; HV OFF, and MVT: LV ON; HV ON in SAFE Mode.
- When turning MVT OFF, do not press "BMT HV OFF" from HV->Overview->Actions, but instead select BMT HV ON - Safe Voltage. The same applies to FMT select FMT HV ON - Safe Voltage.
- When turning MVT ON: go to HV->Overview->Actions, and select BMT (/FMT) "HV ON - Full Voltage".
- DO NOT start runs in SAFE Mode for Production/Calibration/Various-Studies.
• Turn DC HV off only for beam tuning; if no beam is available or when the beam is stable, keep them on even if you are not taking data.
  • To access the Hall:
• If you need to enter Hall B, alert Denny Insley (757-897-9060, cell/text: 757-810-9576) and the Run Coordinator (757-575-7540).
• Check in with the shift personnel before entering the hall.

DAQ Operation & related General Instructions

  • (New) STANDARD end and start of the run:
- "End run"
- "Abort"
- "Download"
- "Prestart"
- "Go"
  • If any ROC crashes, find out which one (look at the ROC Busies GUI or ROCS tabs in the DAQ runcontrol), and end the ongoing run, which might fail to end properly, and that is OK. You SHOULD do "Cancel" and "Reset" in the DAQ Run Control before issuing a roc_reboot command ON JUST THAT ROC and continuing with the DAQ Restart chain from "Configure". Test with ping ONLY instead of ssh to check whether the ROC is backed up.
When pinging a ROC, e.g., "ping mvt1", stop it with Ctrl+C after a couple of printouts showing different latency times.
  • Restart DAQ from Scratch when changing configuration or trigger files, or when the run did not end correctly, or when ROC(s) is(are) rebooted, or when RICH is Fastly/Fully recovered, as follows:
- "Cancel", "Reset"
- "Needed operations: change configuration or trigger files, or a run did not end correctly, or a ROC crash or RICH is Fastly/Fully recovered";
- "Configure"
- "Download"
- "Prestart"
- "Go"
  • Configuration is PROD66/67 (or PROD66_PIN for fast reconstruction of some calibration/reference runs from the cache disk).
As of 11/25/23, whenever there is an issue with the RICH alarms that can't be resolved within 10 to 15 minutes, please contact the RC (/AC) and the RICH experts to discuss the issue. Meanwhile, we may switch the DAQ configuration to PROD67_noRICH until further notice; see the logbook entry #: 4226898.
  • In case of DAQ Disk space alarm, call the DAQ expert.
  • Trigger files:
- Inbending production: rgd_inb_v1.0.trg
- Outbending production: rgd_out_v2.1_Q2_1.2.trg or rgd_out_v2.3_Q2_1.2.trg with low-Q2 suppression
  • During "Configure", observe the ROC terminals to see whether any ROC has error messages.
  • Some ROCs keep printing error messages while functioning properly; please see their list in this logbook entry # 4215299 to avoid unnecessary reboot since the printout in the ROC terminal won't change. If they get disconnected, the message will appear in the run control itself!
  • In case of DAQ livetime "0" alarm, RESTART DAQ from Scratch;
• Note that you also get a DAQ livetime alarm at the start of a run; that is OK and can be simply acknowledged.
  • Unless a RICH problem crashes the DAQ, do NOT interrupt an otherwise smooth run for a RICH recovery. Instead, do this in between runs.
  • The shifter MUST check and reset monitoring histograms at least once every 30-60 minutes to ensure no significant detector problems. Sometimes, this is the only way to know that a specific ROC needs to be rebooted.
  • Pop-up window: Enter the beam current and the type of run (e.g., Production and HWP state, IN or OUT) in the "Comment" field.
  • After each step of a DAQ RESTART, ensure it is completed in the Run Control message window. Also, during "Configure", look for error messages in the ROC terminals.
• Contact the DAQ expert if you have any questions or doubts about how to proceed.
  • At the start and end of every run, check that the data filled in the Run Condition Data Base (RCDB) is correct, especially the manually filled fields.

Reference Values

FSD Thresholds from RG-D

  • Upstream: 2 kHz
  • Midstream: 2 kHz
  • Downstream: 100 kHz
  • BOM: 50 kHz
  • 5 ms dwell time

Harp Scan References from RG-D

  • 2H01 [3]; x= -1.4 mm; y= 0.0 mm
  • Tagger (2C24A) [4]: x= -2.0 mm; y= +0.6 mm
  • 2C21A [5]: x= -0.4 mm; y= +0.6 mm (less important; values form RG-C)

IA for Q Asymmetry

Responsibilities

  • The RC duties are detailed in the COO.
  • Feel free to contact upcoming shifts about their training or site access status.
  • The training requirements for the shift are:
- ES&H Orientation (SAF 100);
- Radiation Worker Training (SAF 801);
- Oxygen Deficiency Hazard Training (SAF 103);
- Hall B Safety Awareness Walk-Through (SAF111)
  • Remote shifts are listed here.
  • Backup shifters are listed here.

Daily

  • Attend the in-person PD/RC and MCC/OPS meetings (7:45 AM and 8 AM Daily). The PD will communicate when and how meetings happen on the weekend.
  • Update the run plan in the Short Term Schedule tab in coordination with the run group coordinator, and communicate it to the shift crew.
  • Log the RC summary in the HBLOG.
  • Send a daily invitation for the RC meeting to hallb, clas_members, and rgd_ana mailing list.
  • Run the RC meetings at 1 PM in CH Room 200C and on ZOOM, except on Wednesdays.
  • On Wednesdays, the RC meeting is on ZOOM ONLY @ 15:30 due to the conflict with the 13:30 MCC meeting and the unavailability of the CH room afterwards.
‡ To connect to Zoom in CH Room 200C, follow these steps:
◊ Turn on the projector with the white remote control, then log into your CUE account. If not listed yet, click "Not listed?" in the bottom left to add your account.
◊ Once logged in, use the recommended Google Chrome to join the ZOOM meeting on the browser or use the ZOOM application.
◊ Since the room doesn't have audio, don't connect with the computer audio, but choose "Call Me," then enter the number in the small Cisco answering device near the microphone in the middle of the conference table. Once it rings, answer the call, and you will be prompted to join the meeting.
◊ The room has a camera, so you may turn it on, if you wish, in the ZOOM browser interface.

Weekly

‡ The report summarizes the data-taking status concerning the available beam used (ABU), any beam losses (BANU), CW on target (ABU+ABNU), etc., from 7 AM past Wed. through 7 AM the Wednesday the report is due, in addition to any standing and critical issues, plans for next week, and general comments/requests, if any.
‡ Our beam time accounting info can be retrieved either from:
◊ OPS logbook entries with tag 'Beam Accounting': Timesheet Summaries;
Beam Time Manager; Physics Time Accounting.
‡ Refer to the RC Timesheet to calculate the percentage of scheduled experiment hours to date.
‡ The total experiment data collected to date could be retrieved from the up-to-date "Accumulated Charge" depicted on the "Total Charge on Target Plot" in the Monitoring tab.
‡ A detailed plan of the target configuration change will be generated as soon as the run starts; however, an overview of it is summarized in slide 6 of this RG-D Overview.
  • Attend the Hall B meeting on Mondays at 9 AM and present your summary slides that should be sent beforehand to the Hall B leader, Patrick Achenbach, after checking them; please, with the Run Group Coordinator Lamiaa El Fassi.


Tasks

  • Come to your shift in time for the shift change, 10-15 minutes before the start of your shift.
  • Follow the Short Term Schedule.
    • The RC must be notified before any run conditions are changed.
  • Operate the Data Acquisition.
  • Take action and log alarms.
  • Communicate with your co-shift person.
    • The shift worker should discuss an issue with the shift expert before making extra logbook entries.
  • Check for and read any comments to log-book entries made during your shift.
  • Keep your shift summary up to date in HBLOG. Record all significant happenings.
    • Refer to relevant logbook entries in the shift summary.
    • Include a run list for the shift with relevant run information.
  • In case of other questions or uncertainties of what to do, call the RC.


New RICH recovery protocol

  • See https://logbooks.jlab.org/entry/4201728 for important details.
  • If there is issues, put screen shots of scalers and temperatures in the RICH logbook. Do not stop an ongoing run but do the fast recovery between runs.
  • Daily full RICH recovery remains part of the protocol, preferably done during day time and only between runs, do not stop an ongoing run. State in your shift summary that you did the recovery (or why you did not).
  • Avoid unnecessary recoveries, see https://logbooks.jlab.org/entry/4208477. The RICH overview GUI shows how many seconds have passed since the last full recovery. Also, there is a minor alarm when 24 hours have passed since the last full recovery. Acknowledge the alarm and wait for the end of the run before doing the recovery.


Fill out BTA hourly.

  • Accelerator beam time (do not change these values):
- ABU: acceptable beam used
- BANU: beam acceptable but not used (should be reflected in our PCC and UED)
- BNA: beam not acceptable (e.g., tuning, drifting, not focused)
- ACC: Accelerator Configuration Change (e.g., pass change)
- OFF: Accelerator is down
  • Experiment beam time (do change these values):
- ER: experiment ready (should be 60 minutes - PCC -UED)
- PCC: planned configuration change (when the beam is masked while radiators are moved, target change, DAQ (stopping/starting runs, tests, etc.)
- UED: unplanned experiment down (hardware failure, DAQ crashes, our problems, ...)
- Exception: BANU due to time spent on beam tuning and BPM checks after beam down should be counted as ER on our side.
  • Enter fields at the "Shift Information" tab;
  • You must save at 7:00, 15:00, and 23:00, to send info to MCC. To see the save button, click 'edit all' or 'edit cc hours.'
  • Sign, viz log completed 8-hour sheet:
  • Note: we can change the timesheet up to 48 hours after completion.

Tasks

  • Come to your shift in time for the shift change, 10-15 minutes before the start of your shift.
  • Discuss an issue with the shift expert before making extra logbook entries, ask the shift expert to note the ongoing issue in the shift summary as well as the start/end of run Comment field in the DAQ pop-up window.

Hourly

  • Reset and check mon12 histograms
    • Note that some plots have radio buttons at the bottom to choose the sector, in which case you should cycle through and check all sectors.

Once per run

  • Submit EPICS screenshots to the HBRUN logbook with beam ON!
◊ Beamline overview, in VNC or Web Browser
◊ CLAS12 trigger rates and DAQ status
  • Upload mon12 monitoring plots to the HBLOG logbook with sufficient statistics
◊ Afterwards, compare them to the reference and recent runs; see the top-right link at the Monitoring tab.
  • Submit strip charts to the HBRUN logbook (set the 'time span width'to be compatible with the run duration)
◊ Beam current and position strip-charts (CS-Studio → Beamline → Strip Charts)
◊ Also accessible in a web browser: Upstream/Midstream Halo Counters, Downstream Halo Counters, Beam Currents, BPM Positions

Once per shift

  • Fill the shift checklist.
  • Per Gavalian's request, upload clas12online plots to the HBLOG logbook of the three tabs, Vertex, Photons, Vertex2D, with sufficient statistics!

For Remote Shifts

Please inform the PDL that your shifts will be remote. Remote shifts are listed here

You must test your setup, including Zoom and VNC connections, at least 24 hours before your first shift!

◊ The passcode is required and contained in this log entry
◊ Remind the expert in the counting house to restart their ZOOM, if necessary
◊ All necessary applications should be available from the desktop menu (left/right click on the desktop)
◊ EPICS GUIs should be accessed ONLY via the desktop menu or by running clascss-remote.

Webcams

EPICS

Data Quality



Clas12design.png