Difference between revisions of "PrimEx 2022"

From epsciwiki
Jump to navigation Jump to search
 
(20 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
 +
[https://epicsweb.jlab.org/wave/?myaDeployment=ops&myaLimit=100000&title=&fullscreen=false&layoutMode=3&viewerMode=1&pv=RESET%3Ai%3AGasPanelBarPress1&pv=HD%3Acoda%3Acdc_hv_ai%3Aswitch&pv=HD%3Acoda%3AEventRate&pv=CDC%3Ahv%3AA%3A1%3Av0set&RESET%3Ai%3AGasPanelBarPress1label=RESET%3Ai%3AGasPanelBarPress1&RESET%3Ai%3AGasPanelBarPress1color=%23a6cee3&RESET%3Ai%3AGasPanelBarPress1yAxisLabel=&RESET%3Ai%3AGasPanelBarPress1yAxisMin=&RESET%3Ai%3AGasPanelBarPress1yAxisMax=&RESET%3Ai%3AGasPanelBarPress1scaler=&RESET%3Ai%3AGasPanelBarPress1yAxisLog=&windowMinutes=2400&start=2022-09-01+00%3A00%3A00&end=2022-10-28+10%3A00%3A00&HD%3Acoda%3Acdc_hv_ai%3Aswitchlabel=HD%3Acoda%3Acdc_hv_ai%3Aswitch&HD%3Acoda%3Acdc_hv_ai%3Aswitchcolor=%231f78b4&HD%3Acoda%3Acdc_hv_ai%3AswitchyAxisLabel=&HD%3Acoda%3Acdc_hv_ai%3AswitchyAxisMin=&HD%3Acoda%3Acdc_hv_ai%3AswitchyAxisMax=&HD%3Acoda%3Acdc_hv_ai%3AswitchyAxisLog=&HD%3Acoda%3Acdc_hv_ai%3Aswitchscaler=&CDC%3Ahv%3AA%3A1%3Av0setyAxisLabel=&CDC%3Ahv%3AA%3A1%3Av0setyAxisMin=&CDC%3Ahv%3AA%3A1%3Av0setyAxisMax=&CDC%3Ahv%3AA%3A1%3Av0setyAxisLog&CDC%3Ahv%3AA%3A1%3Av0setscaler=&IBCAD00CRCUR6yAxisLabel=&IBCAD00CRCUR6yAxisMin=&IBCAD00CRCUR6yAxisMax=&IBCAD00CRCUR6yAxisLog=&IBCAD00CRCUR6scaler=&HD%3Acoda%3AEventRateyAxisLabel=&HD%3Acoda%3AEventRateyAxisMin=&HD%3Acoda%3AEventRateyAxisMax=&HD%3Acoda%3AEventRateyAxisLog&HD%3Acoda%3AEventRatescaler=&HLD%3ATGT%3AstatusyAxisLabel=&HLD%3ATGT%3AstatusyAxisMin=&HLD%3ATGT%3AstatusyAxisMax=&HLD%3ATGT%3AstatusyAxisLog=&HLD%3ATGT%3Astatusscaler=&HD%3Acoda%3AEventRatelabel=HD%3Acoda%3AEventRate&HD%3Acoda%3AEventRatecolor=%23eab0f7&CDC%3Ahv%3AA%3A1%3Av0setlabel=CDC%3Ahv%3AA%3A1%3Av0set&CDC%3Ahv%3AA%3A1%3Av0setcolor=%23010101  Overview in EPICS - change the dates to suit]
 +
 +
[https://epicsweb.jlab.org/wave/?myaDeployment=ops&myaLimit=100000&title=&fullscreen=false&layoutMode=3&viewerMode=1&pv=RESET%3Ai%3AGasPanelBarPress1&pv=HD%3Acoda%3Acdc_hv_ai%3Aswitch&pv=HD%3Acoda%3AEventRate&pv=CDC%3Ahv%3AA%3A1%3Av0set&pv=GAS%3Ai%3A%3ACDC_Temps-CDC_D5_Temp&RESET%3Ai%3AGasPanelBarPress1label=RESET%3Ai%3AGasPanelBarPress1&RESET%3Ai%3AGasPanelBarPress1color=%23a6cee3&RESET%3Ai%3AGasPanelBarPress1yAxisLabel=&RESET%3Ai%3AGasPanelBarPress1yAxisMin=&RESET%3Ai%3AGasPanelBarPress1yAxisMax=&RESET%3Ai%3AGasPanelBarPress1scaler=&RESET%3Ai%3AGasPanelBarPress1yAxisLog=&windowMinutes=2400&start=2022-09-01+00%3A00%3A00&end=2022-10-28+10%3A00%3A00&HD%3Acoda%3Acdc_hv_ai%3Aswitchlabel=HD%3Acoda%3Acdc_hv_ai%3Aswitch&HD%3Acoda%3Acdc_hv_ai%3Aswitchcolor=%231f78b4&HD%3Acoda%3Acdc_hv_ai%3AswitchyAxisLabel=&HD%3Acoda%3Acdc_hv_ai%3AswitchyAxisMin=&HD%3Acoda%3Acdc_hv_ai%3AswitchyAxisMax=&HD%3Acoda%3Acdc_hv_ai%3AswitchyAxisLog=&HD%3Acoda%3Acdc_hv_ai%3Aswitchscaler=&CDC%3Ahv%3AA%3A1%3Av0setyAxisLabel=&CDC%3Ahv%3AA%3A1%3Av0setyAxisMin=&CDC%3Ahv%3AA%3A1%3Av0setyAxisMax=&CDC%3Ahv%3AA%3A1%3Av0setyAxisLog&CDC%3Ahv%3AA%3A1%3Av0setscaler=&IBCAD00CRCUR6yAxisLabel=&IBCAD00CRCUR6yAxisMin=&IBCAD00CRCUR6yAxisMax=&IBCAD00CRCUR6yAxisLog=&IBCAD00CRCUR6scaler=&HD%3Acoda%3AEventRateyAxisLabel=&HD%3Acoda%3AEventRateyAxisMin=&HD%3Acoda%3AEventRateyAxisMax=&HD%3Acoda%3AEventRateyAxisLog&HD%3Acoda%3AEventRatescaler=&HLD%3ATGT%3AstatusyAxisLabel=&HLD%3ATGT%3AstatusyAxisMin=&HLD%3ATGT%3AstatusyAxisMax=&HLD%3ATGT%3AstatusyAxisLog=&HLD%3ATGT%3Astatusscaler=&HD%3Acoda%3AEventRatelabel=HD%3Acoda%3AEventRate&HD%3Acoda%3AEventRatecolor=%23eab0f7&CDC%3Ahv%3AA%3A1%3Av0setlabel=CDC%3Ahv%3AA%3A1%3Av0set&CDC%3Ahv%3AA%3A1%3Av0setcolor=%23010101&GAS%3Ai%3A%3ACDC_Temps-CDC_D5_Templabel=GAS%3Ai%3A%3ACDC_Temps-CDC_D5_Temp&GAS%3Ai%3A%3ACDC_Temps-CDC_D5_Tempcolor=%23fb9a99&GAS%3Ai%3A%3ACDC_Temps-CDC_D5_TempyAxisLabel=&GAS%3Ai%3A%3ACDC_Temps-CDC_D5_TempyAxisMin=&GAS%3Ai%3A%3ACDC_Temps-CDC_D5_TempyAxisMax=&GAS%3Ai%3A%3ACDC_Temps-CDC_D5_TempyAxisLog=&GAS%3Ai%3A%3ACDC_Temps-CDC_D5_Tempscaler=  Same, with temperature]
 +
 
==1 Sept==   
 
==1 Sept==   
  
Line 9: Line 14:
 
(the arguments are minus capital i and then the lookback time in seconds)
 
(the arguments are minus capital i and then the lookback time in seconds)
  
 +
The ideal gcf in the email didn't change.  Did something fail with the init or is this still a bug in the email?  The email bug was fixed ages ago.  Probably Thomas over-wrote the file by accident when working on auto-off.
  
 
10:13 switched the AI on.  The next run will be 110505.  They are using a SOLID Be TARGET so Thomas made the auto-shut-off for empty target configurable and switched it off.
 
10:13 switched the AI on.  The next run will be 110505.  They are using a SOLID Be TARGET so Thomas made the auto-shut-off for empty target configurable and switched it off.
Line 18: Line 24:
 
11:25 - Shift log: All CDC and FDC HVs were turned off, and the solenoid current is being ramped down to 0A.  
 
11:25 - Shift log: All CDC and FDC HVs were turned off, and the solenoid current is being ramped down to 0A.  
 
13:12 - Shift log: Beni called to inform us that he turned off the A.I. for the CDC since we turned the voltages off.
 
13:12 - Shift log: Beni called to inform us that he turned off the A.I. for the CDC since we turned the voltages off.
−
 
  
 
==7 Sept==
 
==7 Sept==
Line 29: Line 34:
  
 
15:07 ish The plan for the rest of the run period is to use the liquid He target.  Thomas enabled the auto-off if the target is not in state 3 (full & ready).  This will set the HV to 2125V and switch off the AI's recommending (voltage-setting) when the target is in any other state.  We will need to ask the shifters to switch the AI back on after empty target runs and not before the target is full again.
 
15:07 ish The plan for the rest of the run period is to use the liquid He target.  Thomas enabled the auto-off if the target is not in state 3 (full & ready).  This will set the HV to 2125V and switch off the AI's recommending (voltage-setting) when the target is in any other state.  We will need to ask the shifters to switch the AI back on after empty target runs and not before the target is full again.
 +
 +
==9 Sept==
 +
 +
14:30 Naomi asked the shift crew to switch the AI off because the HV and pressure were moving in opposite directions, with no change in temperature.  It was switched off after run 110686.
 +
 +
==20 Sept==
 +
RoboCDC still off.
 +
 +
Thomas re-initialized. 
 +
 +
Initialization with model: /group/halld/AIEC/utilities/CDC_control_ai/primex/d_cdc_model_needs/saved_gpflow_model/saved_model.pb
 +
 +
InputDict: {'pressure': 100.061, 'temp': 298.84999999999997, 'current': 2.51501625, 'ideal_gcf': 0.141, 'confidence_threshold': 0.03, 'currentA1setV': '2125'}
 +
 +
OutputDict: {'raw_gcf': 0.14064093033868516, 'gcf': 0.14227172117567272, 'stdv': 0.006650096527178576, 'uq_pressure': 100.1532772, 'uq_hvbi': 2.4899867999999996, 'uq_temp': 298.922688}
 +
 +
==10 Oct==
 +
RoboCDC still off.
 +
 +
System now configured to not "auto-off" instead if the UQ is used (rawf_gcf != gcf) it sets back to the baseline voltage (nominally 2125)
 +
 +
==12 Oct==
 +
 +
12:13 System should now be back on.  Currently doing ET runs thus we shouldn't see any effect.  Next run we'll see an effect will be the next non ET run post run 111026.
 +
 +
The system crashed in such a way that it changed the HV during the ET runs, eg 2128V for run 111035. [https://logbooks.jlab.org/entry/4057006 HDRUN ] [https://halldweb.jlab.org/rcdb/runs/info/111035 RCDB ]
 +
 +
The next full target production run was 111041.
 +
 +
== 14 Oct==
 +
 +
09:37 RoboCDC switched off. Model file was missing.  Last run before this was 111057.
 +
 +
18:28 Switched on again. First run after this is 111062.
 +
 +
== 18 Nov ==
 +
 +
Naomi spotted an entry in the morning meeting run page about RoboCDC. Apparently the RC wasn't aware that the auto-2125-for-ET had been implemented (he is now) but remembered to switch the AI off for the empty target run. The AI stayed off until Shifter Thomas noticed this during a full target run.  This is the logbook entry https://logbooks.jlab.org/entry/4085126  Idk yet if there were any or many full target runs at 2125V but anyway this is of course way better than any other unintended HV.
 +
 +
== 9 Dec ==
 +
 +
In an unfortunate sequence of events, RoboCDC tried to set the HV on the CDC while one of the HV modules was not talking to anything else.  This caused a very long alarm, as the setpoint (received by epics) did not match the setpoint readback (the module did not receive the new setpoint). Naomi expanded the instructions in the CDC shift notes to cover this scenario.
 +
https://halldweb.jlab.org/hdops/wiki/index.php/Central_Drift_Chamber_Shift#Extended_HV_alarm_due_to_communications_drop_while_setting_HV
 +
 +
== 15 Dec ==
 +
 +
Naomi asked the RC to switch off the AI as the full target production runs have finished.

Latest revision as of 14:59, 15 December 2022

Overview in EPICS - change the dates to suit

Same, with temperature

1 Sept

Ready to start. This is how to do the initialization (on one of the gluons):

source $HOSS_VENV/bin/activate.csh

python3 /group/halld/AIEC/utilities/CDC_control_ai/primex/AIEC_CDC.py -c /group/halld/AIEC/utilities/CDC_control_ai/primex/AIEC_CDC.cfg -I 15

(the arguments are minus capital i and then the lookback time in seconds)

The ideal gcf in the email didn't change. Did something fail with the init or is this still a bug in the email? The email bug was fixed ages ago. Probably Thomas over-wrote the file by accident when working on auto-off.

10:13 switched the AI on. The next run will be 110505. They are using a SOLID Be TARGET so Thomas made the auto-shut-off for empty target configurable and switched it off.

5 Sept

110583 was the last run before the HV was switched off.

11:25 - Shift log: All CDC and FDC HVs were turned off, and the solenoid current is being ramped down to 0A. 13:12 - Shift log: Beni called to inform us that he turned off the A.I. for the CDC since we turned the voltages off.

7 Sept

CDC & AI switched back on as the target filled and solenoid ramped [1]

110641 was the first run with the solenoid on and target full.

8 Sept

15:07 ish The plan for the rest of the run period is to use the liquid He target. Thomas enabled the auto-off if the target is not in state 3 (full & ready). This will set the HV to 2125V and switch off the AI's recommending (voltage-setting) when the target is in any other state. We will need to ask the shifters to switch the AI back on after empty target runs and not before the target is full again.

9 Sept

14:30 Naomi asked the shift crew to switch the AI off because the HV and pressure were moving in opposite directions, with no change in temperature. It was switched off after run 110686.

20 Sept

RoboCDC still off.

Thomas re-initialized.

Initialization with model: /group/halld/AIEC/utilities/CDC_control_ai/primex/d_cdc_model_needs/saved_gpflow_model/saved_model.pb

InputDict: {'pressure': 100.061, 'temp': 298.84999999999997, 'current': 2.51501625, 'ideal_gcf': 0.141, 'confidence_threshold': 0.03, 'currentA1setV': '2125'}

OutputDict: {'raw_gcf': 0.14064093033868516, 'gcf': 0.14227172117567272, 'stdv': 0.006650096527178576, 'uq_pressure': 100.1532772, 'uq_hvbi': 2.4899867999999996, 'uq_temp': 298.922688}

10 Oct

RoboCDC still off.

System now configured to not "auto-off" instead if the UQ is used (rawf_gcf != gcf) it sets back to the baseline voltage (nominally 2125)

12 Oct

12:13 System should now be back on. Currently doing ET runs thus we shouldn't see any effect. Next run we'll see an effect will be the next non ET run post run 111026.

The system crashed in such a way that it changed the HV during the ET runs, eg 2128V for run 111035. HDRUN RCDB

The next full target production run was 111041.

14 Oct

09:37 RoboCDC switched off. Model file was missing. Last run before this was 111057.

18:28 Switched on again. First run after this is 111062.

18 Nov

Naomi spotted an entry in the morning meeting run page about RoboCDC. Apparently the RC wasn't aware that the auto-2125-for-ET had been implemented (he is now) but remembered to switch the AI off for the empty target run. The AI stayed off until Shifter Thomas noticed this during a full target run. This is the logbook entry https://logbooks.jlab.org/entry/4085126 Idk yet if there were any or many full target runs at 2125V but anyway this is of course way better than any other unintended HV.

9 Dec

In an unfortunate sequence of events, RoboCDC tried to set the HV on the CDC while one of the HV modules was not talking to anything else. This caused a very long alarm, as the setpoint (received by epics) did not match the setpoint readback (the module did not receive the new setpoint). Naomi expanded the instructions in the CDC shift notes to cover this scenario. https://halldweb.jlab.org/hdops/wiki/index.php/Central_Drift_Chamber_Shift#Extended_HV_alarm_due_to_communications_drop_while_setting_HV

15 Dec

Naomi asked the RC to switch off the AI as the full target production runs have finished.

Retrieved from "https://wiki.jlab.org/epsciwiki/index.php?title=PrimEx_2022&oldid=2753"