Difference between revisions of "The HPS Run Wiki"

Latest revision as of 21:34, 4 November 2021

COO, ESAD, RSAD Things to read before shift.

Expert shift - refresher video demo

Remote worker shift - requirements and responsibilities

HPS 2021 Run discription and beam parameter requirements

Procedures

Procedure for establishing physics quality beam
Commissioning Plans:
- Trigger
- ECal
- SVT
- Hodoscope
Tunnel/Hall Access
Power Failure

Manuals

JLab Logbooks

HPS (E12-11-006) 2021 run, August - November 5
Beam energy 3.74 GeV
Important: Document all your work in the logbook!
Bluejeans for Run Meetings on DAY and DAY at 3 pm

RC: Maurik Holtrop, Thursday Nov 4 + Friday: Rafayel Paremuzyan

(757) 575-7540
9 575 7540 from Counting Room
Personal cell: +1 (757) 753-7769

PDL: Valery Kubarovsky

(757) 876-1789
9 876-1789 from Counting Room

Important instructions

* Turn OFF beam before target move!

* No access to the alcove if the chicane magnet beacon is ON.

Chicane magnets must be OFF if the hall is in Controlled access and access to the alcove is needed or when the hall is in Restricted access. If Chicane magnets are OFF, they can be energized only when the hall is in Power or Beam Permit, [1].

* Energizing the chicane: Please read the "Turning Chicane ONN/OFF" document and ask if anything is not clear. When you pushe the ON button make sure current readbacks will behave as it is described in the scction III "Sequenc details"

* TURN OFF the chicane magnet if beam is not available due to a thunderstorm. Loss off power would seriously damage detectors.

* Fill out BTA hourly. Click "Load from EPICS" to fill the left side, then make sure numbers are correct. If needed edit the entry.

NO ABUs or BANU if beam tune is in progress, make it all BNA with ER 100%. At the end of the shift save BTA to the logbook!

* Please close any GUI that is not being used, e.g. close harp scanner after harp scan is done, close chicane, after chicane is ON, etc.

* For orbit lock positions on 2H00(X=1.14,Y=-2.9 and on 2H02(X=-2.5,Y=-1.1). Use the 2.82 mm SVT collimator

Before 9/9 this was: 2H00(X=0.3,Y=-1.) and on 2H02(X=-2.4,Y=-1.). Use the 2.82 mm SVT collimator

Run Plan Nov 1 - 5, 2021

Take a production data with hps2021_v2_3.trg trigger file. The beam current should be 120 nA, the target is 20 um W.
- if the beam is down for less than 40 min for reasons that will not alter the beam trajectory to Hall-B, do not end the run instead pause DAQ: From the main hps GUI, open 'DAQ Status', click 'Pause Triggers' and confirm
- Then retract the SVT to 3 mm, leave the bias voltages ON. After the beam is back and the halo and the detector rates look the same, bring the SVT to 0.5 mm and Resume DAQ: From the main hps GUI, open 'DAQ Status', click 'Resume Triggers' and confirm"
- PLEASE close 'Are you sure want to pause trigger' and 'Are you sure want to Resume triggers' windows when you are done
Anytime MCC turns OFF or ON Hall-C, HPS SVT bias HV must be OFF, SVT retracted to 3 mm position. After the transition is complete, and the beam to Hall-B is established, perform a 2H02 harp scan at 100 nA and compare with previous scans. IF consistent (within a few 10s microns), precede with production running.

Hall Access Items (Wednesday, November 3rd):

Refill SVT chiller, check ECal and FEB chiller water levels, refill if needed - experts

Every Shift:

Follow the run plan as outlined by RC
With any issue contact On-Call Experts or RC
Monitor carefully the SVT bias current (see below for details)
If any concern about beam stability, ask MCC if orbit locks are on (they should be ON).
Keep shift summary up to date in HBLOG. Record all that happens.
- Document any beam condition change and send scaler GUIs to HBLOG
- Fill out BTA hourly. Click "Load from EPICS" to automatically fill the left side then make sure numbers are correct. If needed edit the entry. Note: in the "Sift information" tab choose experiment E12-11-006. . At the end of the shift send BTA to the logbook.
- Fill and submit the shift checklist in the logbook

Every Run:

After "Go": Check SVT Occupancy
Update the Run Spreadsheet
Log screenshots of:
- main scaler_hps display
- ECAL FADC rate gui
- Trigger rate gui
- SVT and HODOSCOPE and ECAL Monitoring pdfs
Please watch Hybrid sync GUI for correct status (currently three red lights) and the L0-3 and L4-6 occupancy plots for any empty plots beyond the expected three.

SVT Bias Currents:

Shifters must pay attention to the L0 bias currents. This is on both the SVT motor GUI and the main bias GUI. The bias currents will slowly creep up over the course of a run. When the bias current is between 15-20 uA for either L0 sensor, do the following:

Stop the run.
Take the beam away, turn off SVT bias voltages, and wait 10 min.
Turn the bias voltages back on.
Check that the L0 bias currents have gone back down to ~4 uA
Bring back the beam and start a new run.

Do not let the bias current exceed 20 uA. It takes about 4-5 hours for the bias currents to creep up to those values. Call SVT expert if unsure.

Harp Scan Procedure:

During harp scans, SVT must be retracted (motors/SVT Positioners). HVs on SVT, ECal and Hodoscope should be OFF.
Halo counter FSDs (Downstream, HPS Up, and HPS Down) are masked, Beam current 50 nA, Orbit locks running.
Perform the HARP scan, log the results.
If the beam looks different, you should call beam-line on-call that will advise on the next steps.
If the beam looks different, perform SVT wire scan, the target needs to be retracted as well for this one.

Important Notes:

When re-establishing beam to Hall B after a >1h long downtime follow "General Guidelines" on the right and the Beamline Procedures and Manual on the shift documentation page. Do not hesitate to call the beamline expert.
Do not run more than 60 minutes above 30 nA without the beam blocker in front of Faraday cup. Put beam blocker in for long running at high currents.
For SVT Operations:
- From the time beam first comes into the hall until it is being delivered to the Faraday cup with orbit locks on for a known good trajectory, the SVT should be completely off. This is especially important during tuning to the tagger dump, when mis-steering of the beam can result in very high hard neutron backgrounds.
- Once stable beam is being delivered to the Faraday cup with orbit locks on for a known good trajectory, and the SVT collimator is in position, the FEBs and hybrids can be powered and operated, but the HV bias for the sensors must remain off.
- In order to turn on the HV, the FSD system must be functioning so a beam excursion that hits the collimator will quickly trip off the beam.
- Instructions on how to run SVT DAQ [2]
  - And while confluence is down, here's a copy of the paper instructions and a snapshot from confluence a month ago.

General Instructions

Establishing Physics Quality Beam for HPS:

Make sure SVT is retracted and detectors, SVT, Hodoscope and ECAL are OFF.
If the beam was off for 4 hours or more or in case of troublesome beam tuning, send a beam to the tagger dump:
- Check that beam type is set to "Photon" (HPS_EPICS->Beam->BTA).
- Position "Blank" on the beam for the Hall-B collimator
- Ask MCC to turn ON the tagger magnet and then deliver <10 nA beam to the tagger dump.
- Follow instructions under "Procedures" on the documentation tab for "Beamline"[3]. To tune the beam on tagger dump and perform harp scans beam current should be <10 nA.
- Check the beam position on the tagger viewer, it should be approximately in the center of the screen, see https://logbooks.jlab.org/entry/3915622
- Check X/Y positions on BPMs and harps and make sure they are close to the last known values.
- Background levels on harp scans are comparable to those of previous scans.
Before sending a beam to the Faraday cup
- If HPS chicane must be ON, make sure that chicane GUI says "Chicane Ready"
- Set Hall-B Collimator to 20 mm.
- Check that SVT protection collimator is ON THE BEAM (2.82 mm). If the beam was off for more than 4 hours or the configuration has changed, start with 8 mm
- HPS target is OUT and Hodoscope and ECAL HVs are OFF
- SVT is OUT (positions -1.8987 mm for top and -2,2025 for bottom on the main HPS scaler GUI) and bias HV is off (all red/green indicators RED in SVT bias GUI)
- If any uncertainty, page the experts
- Pay attention to the following corrector values: 2C22H, 2C23V, 2H00H, 2H00V, Compare them to ones that are in [4]. Make sure their values don't differ more than 100%. If there is such a difference, tell MCC about this.
Send beam to the Faraday cup
- If the beam optics is correct, then you should immediately see the beam spot on the Downstream Viewer and on the 2H00 and 2H02 BPMS.
- If you don't see it immediately, and rates on halo counters are more than 100 Khz, then ask MCC, to change to pulsed (tune mode) beam, and ask to restore corrector values. At this point don't accept any CW beam, until the beam spot is visible on Downstream Viewer.
- If the MCC operator is not able to tune the beam for more than 15 minutes, then notify RC!
- Follow the instructions under "Procedures" on the documentation tab for "Beamline"[5] to establish beam to the electron dump.

Note: in order to have reliable readings on two important for HPS BPMs, 2H00 and 2H02, beam current should be >30 nA.

- Ask MCC to establish 30 nA beam with HPS orbit locks on (We need HallBHPSOrb not HallBOrb configuration), then gradually increase to 100 nA. For orbit lock position values see "Important instructions" section at top of the page . Use the 2.82 mm SVT collimator. (positions changed 10/3/21 by mg).
- Tune beam profile at HPS using measured widths 2H02A harps. The required beam profile at 2H02A harp is: X-width <~0.1 mm, Y-width <~ 0.05 mm. Compare the beam profile measured with 2H02A harps with previous scans ([6],[7]).
- Check and log the beam spot on chromax viewer
- The halo counter rates before HPS must be very low:
  1. Upstream Left and Right, and the tagger counters few 10's of Hz,
  2. Downstream counters should count <Hz/nA
  3. The halo counter rates on HPS, HPS-L/R/T/SC, depends on the collimator and HPS chicane settings, but should not be more than 50 Hz/nA [8]
- Positions on BPMs must be close to what they were before [9]
- When everything is stable, perform SVT wire scan and compare with the nominal [10]
- Once all the above is achieved, turn ECAL and Hodoscope HV ON to monitor and log ECal rates
- Insert the 20 um W target (turn beam off first).
Before accepting the beam, make sure all the previous conditions are met. Always read previous log entries, compare settings of BPMs and correctors with previous settings using scaler_hps GUI.

RC daily meetings at 3 pm in the counting-house 2nd-floor conference room.

BlueJeans Info:

URL: https://bluejeans.com/759437905/8110

Meeting ID: 759437905

Status of the 2021 run

Run Coordinator
08/23 - 09/01	Stepan Stepanyan
09/01 - 09/08	Stepan Stepanyan
09/08 - 09/15	Maurik Holtrop
09/15 - 09/22	Rafayel Paremuzyan
09/22 - 09/29	Rafayel Paremuzyan
09/29 - 10/06	Stepan Stepanyan
10/06 - 10/13	Stepan Stepanyan
10/13 - 10/20	Stepan Stepanyan
10/20 - 10/27	Rafayel Paremuzyan
10/27 - 11/4	Maurik Holtrop

Physics Division Liaison
	Valery Kubarovsky

Beamline
	Eugene Pasyuk

SlowControls
	Nathan Baltzell

ECAL/HODOSCOPE
	Nathan Baltzell

DAQ
	Sergey Boyarinov

SVT
08/23 - 09/02	Tim Nelson
09/02 - 10/04	Cameron Bravo
10/04 - 10/15	Vitaliy Fadeyev
10/15 -	Alic Spellman

Webcams:

Tagger Dump
Faraday Cup
- To reboot it, run beam-viewer-reboot (open a fresh terminal if error message contains "proxy")
Alcove, SVT Chiller
- To reboot it, run reboot-svt-camera

Accelerator:

Slow Controls:

Online & Offline:

System/Person	Pager/Phone Number
Run Coordinator	(757) 575-7540 (cell)
Physics Div. Liaison	(757) 876-1789 (cell)
MCC-OPS	7048
Crew Chief	7045
Crew Chief	9-879-3367 (cell)
Program Deputy	9-876-7997 (cell)
Gate House Guard	5822
DAQ/Online	(757) 232-6221 (cell)
SVT	(757) 329-4844 (cell)
ECAL/Hodo	(757) 310-7198 (cell)
Slow Controls	(757) 748-6922 (cell)
Beamline	(757) 303-3996 (cell)
Engineering	(757) 748-5048
Hall-B Floor	5165
Hall-B Space Frame	5170, 5171
Hall-B Forward Carriage	5371
Hall-B Counting Room	5244(Shift1), 5245(Shift2)
Hall-B Counting Room	5247(Expert), 5502(Target)
Hall-B Counting Room	5246(RC1), 5126(RC2)
Hall-B Gas Shed	7115

Overview

Bluejeans for 24-7 communication with counting house: https://bluejeans.com/950041022
Remote VNC connection instructions: https://logbooks.jlab.org/entry/3880107
- All necessary applications should be avaible from the desktop menu (left/right click on the desktop)
EPICS GUIs should be accessed ONLY via the desktop menu or by running hps_epics_remote.

Tasks

Fill shift checklist?
Upload plots to the Hall B logbook once per run
- EPICS beamline overview screen
- Trigger GUI
- ECAL and HODO scaler screens
Upload monitoring plots, SVT, ECal, Hodo, Recon, to the logbook at the beginning of the run and after 2-3 hours if run lasts that long (see instructions for storing plots below).
Upload downstream viewer and the SVT chiller cameras to the logbook no more than once per shift
No need to store MYA strip charts unless it is to indicate a problem
Monitor beam quality
- EPICS overview screen
  - In VNC or Web Browser
- Strip charts
  - In VNC or Web Browser: Upstream/Midstream Halo Counters, Downstream Halo Counters, Beam Currents, BPM Positions
Monitor data quality
- Note, for image quality, these windows should be maximized when saving the plots to PDF.
- On any clon machine as user=hpsrun, these commands will run the different monitoring suites:
  - ECAL: startEcalMonitoring - Reference Plots for Comparison, Updated September 11
  - HODO: startHodoMonitoring
  - SVT: startSVTCheckout
  - Recon: startReconMonitoring
- Collect at least 10 min of data before saving the plots
- Before saving plots, please maximze the monitoring app window to the screen in order to get a good quality plots
- Please save monitoring plots into following sub-direcories
  - ECal Monitoring: /home/hpsrun/ECalMonitoring/
  - SVT Monitoring: /home/hpsrun/svt_monitoring/
  - Hodoscope Monitoring: /home/hpsrun/HodoMonitoring/
  - Recon Monitoring: /home/hpsrun/svt_monitoring/
- In order for SVT occupancies to be useful, they must be acquired without beam trips.
- See Monitoring Gui for description of how GUI works
Communicate with the shift expert about issues with any of the above

Shift Worker Backup Table

To add your name, edit this wiki page or send email to Maurik or Nathan.

Worker Shift Backup
Date	Day	Swing	Owl
Tue, Oct 12	Annalisa D'Angelo	Gabriele Simi	Andrea Celentano
Wed, Oct 13	Glasgow	Luca Marsicano	Annalisa D'Angelo
Thu, Oct 14	Glasgow	Andrea Celentano	Annalisa D'Angelo
Fri, Oct 15	Glasgow	Mariangela Bondì	Annalisa D'Angelo
Sat, Oct 16	Glasgow	Mariangela Bondì	Annalisa D'Angelo

Backup shifter contact info
Name	Email	Cell phone
Annalisa D'Angelo	annalisa.dangelo@roma2.infn.it	+393473634490/(call through WhatsApp if not reachable)
Alessandra Filippe	filippi@to.infn.it	+39 328 9411758
Andrea Celentano	andrea.celentano@ge.infn.it	‭+39 340 9831283‬
Domenico D'Urso	ddurso@uniss.it	+39 349 3630 242
Gabriele Simi	gabriele.simi@pd.infn.it	+39 320 748 3385
Hakob Voskania	voskania@jlab.org	+37495909998
Luca Marsicano	luca.marsicano@ge.infn.it	+39 333 9893725‬
Mariangela Bondì	mariangela.bondi@ge.infn.it	+39 ‭327 1895733‬
Marzio de Napoli	marzio.denapoli@ct.infn.it	+39-3498834468
Maurik Holtrop	holtrop@jlab.org	+1-603-868-8088
Natasa Dashyan	natasha@jlab.org	+37477538360
Nathan Baltzell	baltzell@jlab.org	+1 757 748 6922
Nunzio Randazzo	nunzio.randazzo@ct.infn.it	+39-3472591331
Raffaella De Vita	Raffaella.Devita@ge.infn.it	+39 340 8110156
Raphaël Dupré	raphael.dupre@ijclab.in2p3.fr	+33 670 977 761
Rouven Essig	rouven.essig@stonybrook.edu
Silvia Niccolai	silvia@jlab.org	+33 624 81 67 78
Valeria Sipala	vsipala@uniss.it	+39 339 339 57 04

@@ Line 94: / Line 94: @@
 = Short Term Schedule =
 <center><font size=+5>
-''' HPS (E12-11-006) 2021 run, August - October''''<br>
+''' HPS (E12-11-006) 2021 run, August - November 5 '''<br>
 ''' Beam energy 3.74 GeV '''<br>
 ''' Important: Document all your work in the logbook! '''<br>
@@ Line 100: / Line 100: @@
 </font></center>
-== RC: Rafayel Paremuzyan  ==
+== RC: Maurik Holtrop, Thursday Nov 4 + Friday: Rafayel Paremuzyan  ==
 * (757) 575-7540
 * 9 575 7540 from Counting Room
@@ Line 114: / Line 114: @@
 === <font color=red> * Energizing the chicane: Please read the "Turning Chicane ONN/OFF" document and ask if anything is not clear. When you pushe the ''' ON''' button  make sure current readbacks will behave as it is described in the scction III "'''Sequenc details'''" </font> ===
 === <font color=red> * TURN OFF the chicane magnet if beam is not available due to a thunderstorm. Loss off power would seriously damage detectors.</font> ===
 === * [https://bta.acc.jlab.org Fill out BTA] hourly. Click "Load from EPICS" to fill the left side, then make sure numbers are correct. If needed edit the entry. ===
 NO ABUs or BANU if beam tune is in progress, make it all BNA with ER 100%. At the end of the shift save BTA to the logbook!
 === * Please close any GUI that is not being used, e.g. close harp scanner after harp scan is done, close chicane, after chicane is ON, etc.  ===
 <!-- === * <b>For orbit lock positions on <font color=red>2H00(X=0.3,Y=-1.) and on 2H02(X=-2.7,Y=-0.9). Use the 2.82 mm SVT collimator </font></b> === -->
+=== * <b>For orbit lock positions on <font color=red>2H00(X=1.14,Y=-2.9 and on 2H02(X=-2.5,Y=-1.1). Use the 2.82 mm SVT collimator </font></b> ===
+<!--
 === * <b>Orbit lock positions for <font color=red> one pass beam</font> are <font color=red>2H00(X=1.14,Y=-2.9) and 2H02(X=-1.8,Y=-1.2). Use the 2.82 mm SVT collimator </font></b> ===
+=== * <b> <font color=red> For 1 pass beam with 70 nA beam current, we should not use the beam blocker. This instruction supersedes the beam blocker requirement in the section "Important Notes"  </font></b> ===
+-->
 Before 9/9 this was: 2H00(X=0.3,Y=-1.) and on 2H02(X=-2.4,Y=-1.). Use the 2.82 mm SVT collimator
@@ Line 128: / Line 130: @@
 | valign=top width=50% |
-== Run Plan October 19 - 22, 2021 ==
+== Run Plan Nov 1 - 5, 2021 ==
+<!--
+* Restore physics quality beam for the production running at 2-pass, start from the tagger dump (see instructions on the right)
-* Restore physics quality beam for the production running at 1-pass, start from the tagger dump (see instructions on the right)
 * After desired beam profile and positions are
-<!--
 * Special runs: ''' After October 15 morning run ends'''
 ** Low Luminosity run: trigger '''file hps2021_v2_3.trg''', beam current ''' 60 nA''', target '''20 um''', '''4 hours'''.
 ** Random Trigger Run. Trigger file '''hps2021_v2_2_30kHz_random.trg''' Target '''20 um''', Beam current '''120 nA''', Duration about '''1 hour with a good beam''' (Good means beam off should be less than 20% of the
 ** Moeller run: Trigger file '''hps2021_v2_2_moller_LowLumi.trg''', Beam current '''50 nA''', Target '''8 um''', 200 M events.
 * '''Take production data''':  120 nA beam on the 20 um W target, PROD77 configuration, production trigger '''hps2021_v2_3.trg'''. Each run should last about 4 hours.
-* '''Setup production data taking''':  Start 50 nA beam on the 8 um W target, PROD77 configuration, production trigger '''hps_1.9_v2_3.trg'''. Turn ON detectors, bring SVT in to 0.5 mm. Start the run and check occupancies. Increase the beam current (up to 100 nA) until you reach occupancies of ~1.5% for L0t and L0b. End run and start a new one. -->
+* '''Setup production data taking''':  Start 50 nA beam on the 8 um W target, PROD77 configuration, production trigger '''hps_1.9_v2_3.trg'''. Turn ON detectors, bring SVT in to 0.5 mm. Start the run and check occupancies. Increase the beam current (up to 100 nA) until you reach occupancies of ~1.5% for L0t and L0b. End run and start a new one.
-* '''Take production data''': Beam current 70 nA, target 8 um W. Each run should last about 4 hours. Trigger hps_1.9_v2_5.trg.
+* We will try to have the beam on FCup. with 8 mm collimator in place, <font color=red> No Halo FSD should be masked at this step. </font>
+** If the beam passes through successfully, then we should ask the beam to be off, and put the 2.88 mm collimator, and do 2H02, SVT top and bottom scans to make sure beam position and profile are good.
+* Start a Moeller run 'hps_1.9_Moller_v2_6.trg' for 4 hours, if statistics will be less than 300M, take another run with Moeller trigger.
+* '''Take production data''': Beam current 70 nA, target 8 um W. Each run should last about 4 hours. Trigger hps_1.9_v2_6.trg. -->
+<!-- * Take one hour Empty target run, with the same current, and the same trigger file. -->
+<!-- * Stright through run:
+** Target out, 50 nA beam, 2H02 harp horizontal wire should come closer to the beam, trigger file hps2021_FEE_straight_v2_2.trg, take 50 M events
+** Target out, 50 nA beam, SVT collimator wire should come closer to the beam, trigger file hps2021_FEE_straight_v2_2.trg, take 50 M events-->
+* Take a production data with '''hps2021_v2_3.trg''' trigger file. The beam current should be '''120 nA''', the target is '''20 um W'''.
 ** if the beam is down for less than 40 min for reasons that will not alter the beam trajectory to Hall-B, do not end the run instead pause DAQ: <font color=red> From the main hps GUI, open 'DAQ Status', click 'Pause Triggers' and confirm </font>
 ** Then retract the SVT to 3 mm, leave the bias voltages ON. After the beam is back and the halo and the detector rates look the same, bring the SVT to 0.5 mm and Resume DAQ: <font color=red> From the main hps GUI, open 'DAQ Status', click 'Resume Triggers' and confirm" </font>
@@ Line 150: / Line 160: @@
-===Hall Access Items (Wednesday, October 20):===
+===Hall Access Items (Wednesday, November 3rd):===
 * Refill SVT chiller, check ECal and FEB chiller water levels, refill if needed - experts
@@ Line 498: / Line 508: @@
 |style="border-bottom:3px solid gray;" | '''Gate House Guard''' ||style="border-bottom:3px solid gray;"| '''5822'''
 |-
-| DAQ/Online || (757) 869-2188 (cell)
+| DAQ/Online || (757) 232-6221 <!--(757) 869-2188-->  (cell)
 |-
 |SVT ||  (757) 329-4844 (cell)