Cosmics test plans

Objectives:

• Test drive the AIEC software
• Demonstrate that the CDC gain is stable

Schedule:

• 2022-02-25 Fri. - Test fully automated CDC HV control
  • Notify Beni and Mark Stevens of intent to run test on Fri. throughout the day with low voltages (~20V-30V)
  • Modify control script to subtract 2100V from suggested setting before applying
  • Create cronjob under hdops account to run script every 5min.
  • Monitor system occasionally throughout the day so that it runs for 4-6hrs
• 2022-03-01 Tue. - Start system
  • Notify halld@jlab.org that the test is starting and provide contact info.
  • Verify control script no longer subtracts 2100V from suggested setting
  • Confirm CDC gas is flowing and bubbler is correct
  • Confirm CDC alcohol level is good
  • Confirm BCAL chiller is on
  • Turn on all CDC HV channels
  • Turn on all BCAL HV,LV channels
  • Start DAQ system and confirm that cosmic trigger rate is correct (~100Hz?)
    • TODO: Get correct DAQ configuration details from Sergey
  • Set cronjob under hdops account to run script every 2min. and watch for a few cycles
  • Check CDC occupancy in RootSpy
    • TODO: Confirm that the occupancy histogram will work with cosmic events
    • TODO: Add additional monitoring plots to RootSpy
  • Increase the frequency of writing EPICS into the datastream.
  • If all seems to be functioning, reset the crontab to run the script once per hour
  • Make HDLOG logbook entry
  • Monitor progress regularly for 2 weeks. Start new runs if needed.
• 2022-03-14 Mon. - End test
  • Put all CDC HV setpoints back to 2125V
  • Change the EPICS frequency back to normal.

Strategy:

• Designate 2130V as the nominal HV for cosmics, to raise more hits above the noise, but keep 20V headroom for the AI.
• Tell the AI that the HVB current is constant at 9uA so that it can operate within the range of its trained data
• Use 1V steps and 1h runs
  • Starting new run every hour for 2 weeks may not be feasible. Devise scheme to break up long runs?

Needs:

• AIEC software to read epics and wait for a stable period before calculating the gain and setting the HV
  • What determines "stable"?
  • Who?

• Method for scaling the HVB current to what it would be at the nominal HV before running the AI - will not be needed for cosmics but will be needed for CPP.
  • Is this something that could be tested during cosmics run? (yes=implement, no=defer)

• Script to project dE/dx for 4-10 GeV tracks, fit it and calculate the gain factor Not needed for cosmics.
  • Is this for needed for cosmics? (monitoring?)
• CDC_occupancy script needs to be adapted for cosmics if we intend to use rootspy.
• CDC_amp plugin adapted to make histograms for masked wires. Done.

• Code to set whole quadrants to same value(cdc_tool.py)

Questions:

• Do we need to force a B field in the reconstruction sw (wait for Simon to answer)
  • Simon reported progress on straight track reconstruction (email to David and Naomi Feb. 24th)
• Full DAQ or standalone DAQ?
  • Ask Sergey for configuration with only CDC, BCAL and writing only to one file at a time.
  • For run 80239 the 19GB files contained 1/3 M triggers each. This file size is ok.
• When are we doing this?
  • March 1-14, 2022
• Can we write something (epics data) to the evio file whenever the HV is changed? Or increase the epics inclusion rate?

---

---

Locations

• Live Plot of last 24 hours

• Plot of Full 2 weeks

• Software:
  • The code is kept in the repository here: https://github.com/JeffersonLab/AIEC
    • (specifically the utlities and utilities/CDC_control_ai directories)
  • The code itself is checked out and run from the CUE directory /group/halld/AIEC/utilities
  • The top-level script /group/halld/AIEC/utilities/CDC_control_ai/AIEC_CDC.py is run via cronjob every 1 minute. The cronjob is run from the hdops account on gluon47.
  • The cronjob uses the script /gluex/etc/crontabs/tcsh_AIEC_env to setup the environment before running the AIEC_CDC.py python script.
    • The environemnet is setup to: 1.) unset the PYTHONPATH envar to ensure packages come from the virtual environment. 2.) activate the python virtual environment pointed to by the $HOSS_VENV envar. This has the needed package versions installed.
  • The config file used is: /group/halld/AIEC/utilities/CDC_control_ai/AIEC_CDC.cfg
  • The model location is specified in the above config file and is currently: /gluonraid2/AIEC/models/d_cdc_modelneeds/model_pickle4_scilearn0242.pkl .
  • The map of the CDC HV boards designated for the right half of the CDC is built into the cdc_tool.py script. The AIEC_CDC.py sets the HV using the cdc_tool.py script, specifying the boards in the config. file mentioned above. The configuration specifies these as "BRB,BRT" which are the Beam Right Bottom and Beam Right Top quadrants. The assignments can be seen in the graphic below.

Cosmic Run Notes:

---

2022-03-01

Tuesday March 1st 6:39pm : CDC turned on and run started DAQ configuration issues automated system running (image)

2022-03-02

Wed: Temperature found to be less controlled, temps on edge or outside model training. Temperature fixed to 299.25 "ideal_gcf": 0.15292712143476878

Naomi not happy with the controls. rec_scale changed to 15 min (15) under the assumption there was a build up of David finds and fixes bug in cdc_tool.py which caused the setting to take more than the original rec_scale of 5 min. Once fixed rec_scale reset to 5 new: "ideal_gcf": 0.15240693525146817

2022-03-03

Thursday: Naomi not happy with the recommended HV given the pressure.

control system reinitialized. "ideal_gcf": 0.15202469799843948

Naomi notices the default pressure is wrong and corrects it

"ideal_gcf": 0.1523301284531037

11:45am Thomas finds bug in initialization not respecting the fixed values. Bug fixed. "ideal_gcf": 0.1482670653025542

12:21pm Torri starts a new run post bug fix (Run 90716)

1pm-ish beam current returning N/A forcing system to report bad epics. This crashes the model and program. Thomas routes around this N/A as beam current isn't being used and protects the model from this. It is also logged in the DB now

2:05pm EPICS has died (more N/A) across the board. Program no longer crashes but, as expected, the CDC is no longer under control of the AI

epics down between 13:13:13 -> 14:17:14 the pressure didn't change during this period so no harm done to the run. discussed a fail safe mechanism

2022-03-04

Friday pretty big pressure change. HV following as expected.

8:43am Thomas adds the fail safe with no interruption to running. System configuration now stands at: {"model_name": "/gluonraid2/AIEC/models/d_cdc_modelneeds/model_pickle4_scilearn0242.pkl", "poll_time": 1, "rec_scale": 5, "control_window": [2110.0, 2150.0], "epics_trailing_window": 15, "ideal_gcf": 0.1482670653025542, "default_hvbi": -9.0, "require_human": 0, "control_mask": "BRB,BRT", "debug": 0, "default_temp": -299.25, "default_pressure": -100.022,"baseline_V":2130,"fail_safe_timeout":90}

2022-03-11

10:47 am Torri started new run, 90718.