Cosmics test plans

Objectives:

• Test drive the AIEC software
• Demonstrate that the CDC gain is stable

Schedule:

• 2022-02-25 Fri. - Test fully automated CDC HV control
  • Notify Beni and Mark Stevens of intent to run test on Fri. throughout the day with low voltages (~20V-30V)
  • Modify control script to subtract 2100V from suggested setting before applying
  • Create cronjob under hdops account to run script every 5min.
  • Monitor system occasionally throughout the day so that it runs for 4-6hrs
• 2022-03-01 Tue. - Start system
  • Notify halld@jlab.org that the test is starting and provide contact info.
  • Verify control script no longer subtracts 2100V from suggested setting
  • Confirm CDC gas is flowing and bubbler is correct
  • Confirm CDC alcohol level is good
  • Confirm BCAL chiller is on
  • Turn on all CDC HV channels
  • Turn on all BCAL HV,LV channels
  • Start DAQ system and confirm that cosmic trigger rate is correct (~100Hz?)
    • TODO: Get correct DAQ configuration details from Sergey
  • Set cronjob under hdops account to run script every 2min. and watch for a few cycles
  • Check CDC occupancy in RootSpy
    • TODO: Confirm that the occupancy histogram will work with cosmic events
    • TODO: Add additional monitoring plots to RootSpy
  • Increase the frequency of writing EPICS into the datastream.
  • If all seems to be functioning, reset the crontab to run the script once per hour
  • Make HDLOG logbook entry
  • Monitor progress regularly for 2 weeks. Start new runs if needed.
• 2022-03-14 Mon. - End test
  • Put all CDC HV setpoints back to 2125V
  • Change the EPICS frequency back to normal.

Strategy:

• Designate 2130V as the nominal HV for cosmics, to raise more hits above the noise, but keep 20V headroom for the AI.
• Tell the AI that the HVB current is constant at 9uA so that it can operate within the range of its trained data
• Use 1V steps and 1h runs
  • Starting new run every hour for 2 weeks may not be feasible. Devise scheme to break up long runs?

Needs:

• AIEC software to read epics and wait for a stable period before calculating the gain and setting the HV
  • What determines "stable"?
  • Who?

• Method for scaling the HVB current to what it would be at the nominal HV before running the AI - will not be needed for cosmics but will be needed for CPP.
  • Is this something that could be tested during cosmics run? (yes=implement, no=defer)

• Script to project dE/dx for 4-10 GeV tracks, fit it and calculate the gain factor Not needed for cosmics.
  • Is this for needed for cosmics? (monitoring?)
• CDC_occupancy script needs to be adapted for cosmics if we intend to use rootspy.
• CDC_amp plugin adapted to make histograms for masked wires. Done.

• Code to set whole quadrants to same value(cdc_tool.py)

Questions:

• Do we need to force a B field in the reconstruction sw (wait for Simon to answer)
  • Simon reported progress on straight track reconstruction (email to David and Naomi Feb. 24th)
• Full DAQ or standalone DAQ?
  • Ask Sergey for configuration with only CDC, BCAL and writing only to one file at a time.
  • For run 80239 the 19GB files contained 1/3 M triggers each. This file size is ok.
• When are we doing this?
  • March 1-14, 2022
• Can we write something (epics data) to the evio file whenever the HV is changed? Or increase the epics inclusion rate?

---

---

Locations

• Live Plot of last 24 hours

• Plot of Full 2 weeks

• Software:
  • The code is kept in the repository here: https://github.com/JeffersonLab/AIEC
    • (specifically the utlities and utilities/CDC_control_ai directories)
  • The code itself is checked out and run from the CUE directory /group/halld/AIEC/utilities
  • The top-level script /group/halld/AIEC/utilities/CDC_control_ai/AIEC_CDC.py is run via cronjob every 1 minute. The cronjob is run from the hdops account on gluon47.
  • The cronjob uses the script /gluex/etc/crontabs/tcsh_AIEC_env to setup the environment before running the AIEC_CDC.py python script.
    • The environemnet is setup to: 1.) unset the PYTHONPATH envar to ensure packages come from the virtual environment. 2.) activate the python virtual environment pointed to by the $HOSS_VENV envar. This has the needed package versions installed.
  • The config file used is: /group/halld/AIEC/utilities/CDC_control_ai/AIEC_CDC.cfg
  • The model location is specified in the above config file and is currently: /gluonraid2/AIEC/models/d_cdc_modelneeds/model_pickle4_scilearn0242.pkl .
  • The map of the CDC HV boards designated for the right half of the CDC is built into the cdc_tool.py script. The AIEC_CDC.py sets the HV using the cdc_tool.py script, specifying the boards in the config. file mentioned above. The configuration specifies these as "BRB,BRT" which are the Beam Right Bottom and Beam Right Top quadrants.

Cosmic Run Notes:

---

2022-03-01

Tuesday March 1st 6:39pm : CDC turned on and run started DAQ configuration issues automated system running (image)

2022-03-02

Wed: Temperature found to be less controlled, temps on edge or outside model training. Temperature fixed to 299.25 "ideal_gcf": 0.15292712143476878

Naomi not happy with the controls. rec_scale changed to 15 min (15) under the assumption there was a build up of David finds and fixes bug in cdc_tool.py which caused the setting to take more than the original rec_scale of 5 min. Once fixed rec_scale reset to 5 new: "ideal_gcf": 0.15240693525146817

2022-03-03

Thursday: Naomi not happy with the recommended HV given the pressure.

control system reinitialized. "ideal_gcf": 0.15202469799843948

Naomi notices the default pressure is wrong and corrects it

"ideal_gcf": 0.1523301284531037

11:45am Thomas finds bug in initialization not respecting the fixed values. Bug fixed. "ideal_gcf": 0.1482670653025542

12:21pm Torri starts a new run post bug fix (Run 90716)

1pm-ish beam current returning N/A forcing system to report bad epics. This crashes the model and program. Thomas routes around this N/A as beam current isn't being used and protects the model from this. It is also logged in the DB now

2:05pm EPICS has died (more N/A) across the board. Program no longer crashes but, as expected, the CDC is no longer under control of the AI

epics down between 13:13:13 -> 14:17:14 the pressure didn't change during this period so no harm done to the run. discussed a fail safe mechanism

2022-03-04

Friday pretty big pressure change. HV following as expected.

8:43am Thomas adds the fail safe with no interruption to running. System configuration now stands at: {"model_name": "/gluonraid2/AIEC/models/d_cdc_modelneeds/model_pickle4_scilearn0242.pkl", "poll_time": 1, "rec_scale": 5, "control_window": [2110.0, 2150.0], "epics_trailing_window": 15, "ideal_gcf": 0.1482670653025542, "default_hvbi": -9.0, "require_human": 0, "control_mask": "BRB,BRT", "debug": 0, "default_temp": -299.25, "default_pressure": -100.022,"baseline_V":2130,"fail_safe_timeout":90}