CIS OnCall and Troubleshooting

No good solution is too small to go undocumented!!!

OnCall Responsibilities

Updated on 5/10/22

CIS on-call share responsibilities for supporting operation of CEBAF:

1. An on-call period is one week and resets on Wed at 1:30 pm
2. Attend the following meetings:
   1. Daily planning meeting (MCC 8:00 am) Join Zoom Meeting (ID: 160 130 4841, Passcode: 350117)
   2. Weekly scheduling meeting (MCC 1:30 pm Wed) Join Zoom Meeting (ID: 160 130 4841, Passcode: 350117)
   3. B-Team meeting (MCC 1:30 pm Tue) Join Zoom Meeting (ID: 161 118 7954, Passcode: 165766)
3. Check logbook daily and confirm MCC is doing the daily QE measurements
4. Make useful ELOG/POLOG entries and a weekly summary entry (e.g., plots of lifetime)

Power Outage Recovery document

[[[Sharepoint Power Document]https://jeffersonlab.sharepoint.com/:w:/r/sites/CIS/_layouts/15/Doc.aspx?sourcedoc=%7B142C8D43-DF54-44FE-BD38-2C4A2308D1AB%7D&file=CEBAF%20Injector%20Power%20Outage%20Recovery.docx&action=default&mobileredirect=true]]

Greasing the R30 cable

Updated on 1/14/19

A typical maintenance activity (every ~6 months) or whenever the HV cable is reinserted to the ceramic is to regrease the R30 cable/ceramic interface.

1. Lock-out the gun HVPS so high voltage cannot be generated on the cable.
2. Inspect the connection of cable to ceramic flange. It should be metal to metal.
3. Remove the cable flange screws holding it to the ceramic flange, and pull the cable upward, may feel stuck but requires good tug of 10-20 lbs of force to do so.
4. Use lint free cloth to wipe grease from the cable rubber plug, metal interface. Remove spring on banana plug spring, do the same and re-attach. That completes the cable cleaning.
5. Wrap some lint free cloth around forceps being careful there is no bare metal of forceps showing, and wet with toluene. Twist into ceramic and swirl moving upward to remove grease, repeat until fully clean. Do not leave toluene in ceramic as it will dissolve the new grease. This completes cleaning the ceramic.
6. Apply grease to cable rubber plug (not the metal at tip of cable or the spring) and smooth with gloved hand over the surface of the rubber plug. To gauge amount of grease start with about a 3" line from tube, and repeat as needed, typically a total of about 3 lines is sufficient. This completes the regreasing.
7. Insert the plug straight into the ceramic. For non-spring loaded cables set the initial gap between the cable flange and ceramic flange to about 5mm, then insert the screws and tighten down uniformly until the two flanges are metal to metal, ensuring good contact between the cable rubber plug and the ceramic surface. For spring loaded cables, initial gap should be approximately 9mm.
8. Clean up, remove lock-out of HVPS, turn gun HV back on to verify OK.

Gun High Voltage Interlock Faults

* AC Line Volt Fault does not clear: you may want to recycle PSS system or reboot iocin3, see [1] for more details.

250V Gun High Voltage FSD Comparator Faults

Use comparator circuit in ISB monitoring HVPS output to set window (range about +/- 500V) about gun high voltage. Faults can be real (gun HV is out of range) or false (comparator is drifting).

To check if the gun HV is actually drifting the following archived read backs are useful to view ...
* Precision read back of Ross probe
* EPICS signal ISDIN014RDBK01 found from screen "FSD NODE IN01_4"
* HVPS lock set point
* Horizontal orbit at IPM1I02

To determine if the gun HV window comparator is drifting or set incorrectly ...
* Window drift during SAD can be found at startup
* Window drift sensitive to temperature variation e.g. seasonal temperature swing
* Window set too tightly by Ops

To set gun HV window comparator window ...
* While clicking FSD clear button vary POT until resets to find a good region
* High Limit : Increase POT while clicking until fault occurs, then decrease until fault clears and record new high value
* Low Limit : Decrease POT while clicking until fault occurs, then increase until fault clears and record new low value
* Set POT at center of new low and high values; maybe repeat to get consistent values
* Open screen "FSD NODE IN01_4"
* Adjust comparator setpoints - typically about 500 V above and below voltage readback on signal ISDIN014RDBK01
* Compare with temperature on HVPS, epics channel ITMGUNHVPS, to see if cause can be correlated with temperature.

Diagnosing Beam Intensity and Position Noise

Use FastSEE BPM Mode to diagnose intensity and position from injector BPM's with 4 microsecond sampling

To start the fastSEE EPICS screen ...
* To start : Accel Menubar => EDM => BPM => New FastSEE Control
* MAC or Channel Access is required
* Choose a BPM mode and BPM by selecting corresponding ioc e.g. iocse11

Notes for BPMSEE mode ...
* Non-invasive
* BPM spikes update correctly
* Less samples limit to lower frequency
* Make sure FastSee Light is ON

Notes for FastSEE mode ...
* Invasive
* BPM spikes update erratically
* More samples for higher frequency ~60kHz

QE Measurement Script

It is useful for Operations or CIS staff to perform a daily measurement of photocathode QE using an automated script.

To configure and test QE Measurement Script ...
* Determine laser attenuator values that generate same bunch charge from each laser e.g. in the 5-15 uA range
* Load and save these values into the Injector Expert Script
* Run QE measurement script selecting all lasers
* Laser power and PCup current are measured for each laser in sequence
* QE = (124 * I[uA]) / (P[mW]*Wavelength[nm)

QE Scan

Media:CEBAF QE cathode scan procedure.docx

Laser Related Issues

To change the laser pulsewidth ...
* To start : PGun Main (John's laser control screen), then click Fiber Laser Super User, another screen appears.
* From Fiber Laser Super User, click "Frequency Division Controls".  Yes, this name no longer makes sense. It is now the screen to adjust the laser dc bias current. 
* From "Frequency Division Controls", adjust the dc bias current by following the instructions outlined by John Hansknecht, see link Media: laser seed drivr settings.pdf

Resetting LLRF quadrant detection : use with care ! 
* To start : JTABS => Tools Screen (right side of menubar) => Laser Quadrant Init Pulse (lower left related display)
* Pick some diagnostic (separator viewer is best if setting phase of 250MHz rep rate e.g. invisible effect at chopper)
* Choose laser A/B/C and cycled button from from 0=>1=>0 (i.e. leave at zero); repeat until phase is correct

Hall D Laser attenuator Home slippage motor error: Problem shows as large negative number on the attenuator redback and CW beam current unresponsive to changes in attenuator

• The problem appeared when the velocity of the stepper motor did not update after the doc reboot. To solve this issue, see this log entry

Hall B/D Laser beam delivery getting 'stuck' upon recovering after FSD trips

• Media:Ops laser instabilities.pdf

Found Hall B current too low in tune mode with attenuator at maximum incident 18August2021

• This happened on a maintenance day - I had to use the laser bypass key

• Before calling laser on call
• 1. Home/reset for attenuator B (pgun main page, pink box for psub controls, home button)
• 2. Full laser power into power meter: how does it compare to daily QE measurements
• 3. See if the tune mode generator is not operating properly: the SCAM shows "duty factor" so you can calculate how much power you should get in tune mode and viewer limited
• 4. Stare at 4-laser controls, compare lasers, use archiver, see if values look nominal
• 5. If you're not getting sensible power from B laser, contact Shukui (the problem this time was that after work on pockles cell, laser output was lower. I increased the amplifier power at Shukui's direction)

Gun Current Monitoring

The gun current is calculated from the HVPS analog output that reports a voltage proportional to current output.  We typically calibrated this HVPS output against beam well centered in PCUP (assumes no loss).  The equation EPICS uses is a bit odd but Gun_Current = (HVPS - Offset)/Gain.  The Offset is typically 0.15-0.25 V depending upon power supply and the Gain is typically 0.001 V/uA, i.e. ~1V/mA.  Here are some examples with 150 kV Glassman supply from Feb 2016 and Jan 2011.

Laser Amp Reset

Troubleshooting guide:

Lasers have tripped off – need to reset the laser amps

1. Laser RF must be on before trying to fix other laser issues

To turn lasers back on: (Joe prefers doing one laser at a time if multiple are tripped)

1. Emission off (if it isn't already)
2. reset button for the laser – this should take about a minute or so
   1. Laser amp power and laser amp temp should gray out during reset
   2. These will become active once successful reset
3. Emission on
4. Enter value for emission (something different than it was before – 2.0 entered with 2.0 won't change, go to 2.1 then back to 2.0)
5. All happy :)

x-y stage homing

Home switch is on pgun ops main, right bottom in pink square, injector hardware, psub, reset for x-y stage and laser attenuators there.

Spin Dance

Overview

A spin dance most often refers to mapping out the longitudinal component of the beam polarization at an end station as a function of the Wien angle at the injector. The purpose is to determine the Wien angle that maximizes the beam polarization at the experiment. The horizontal Wien filter is typically used b/c it compensates for the net horizontal spin precession incurred by the beam circulating the accelerator. Of course, more generally, any of the injector spin rotators (either Wien and solenoid) can be used e.g. to map out any of the 3 components (x, y or z) of beam polarization at the hall.

Which angles to try?

The measurement of polarization vs. Wien angle will be a sinusoidal function, b/c the polarization is measuring a projection of the beam polarization along the longitudinal direction. It is inefficient to find the Wien angle that gives maximum polarization by measuring near the maximum, because the polarization does not change much. However, it is very efficient to measure where the polarization is changing most rapidly, e.g. further away from the maximum. Ideally, one would measure +/- 90 deg from expected maximum value, but that most often not possible because it is out of range of the Wien filter capability (+/- 100 deg) and because the Wien filter is an optical device it can also make large angle changes laborious. Typically, +/- 45 deg work very well.

Process (takes ~half a shift, give or take)

1. Remind the PD, crew chief and halls of the process, we do this infrequently and with new people
2. Beam goes away from Halls for each Wien angle change for about 15-20 min
3. Before first change useful to do a Burt Save and measure transmission to fc#2 for reference
4. Change Wien angle (see procedure below)
5. Restore beam to halls; priority is for hall measuring polarization, i.e. other halls get what they get (so semi-invasive)
6. Hall measures polarization, typically takes 30-60 min
7. Iterate two or three more time w/ Wien angle changes until a best value is identified (by the hall)
8. Go to the best value, hall will measure final point, but sometimes will ask for one more small change
9. Once final Wien angle is agreed take another snapshot of injector transmission for the logbook
10. For large changes to Wien angle, the injector group will often follow-on to optimize transmission, before/after restoring halls

Changing the Wien filter

• The Wien filter is changed using JMenu=>All Injector Apps=>Injector Steering script
• Choose the appropriate panel, e.g. "Horizontal Wien"
• Send tune beam to fc#1
• Use "Save Setup" to RELS to save values before each change
• Then enter a New value (if inexperienced use small steps ~30 deg), press Set Wien
• Script turns beam off, changes Wien filter
• Turn beam back on to FC1, observe REL orbit
• Take Wien filter off loop, make small changes (~0.1A) to minimize REL orbit; don't worry about it being perfect
• When REL orbit is "good enough", press the Initialize button, when that's done press Steer button
• If REL orbit flattened then great and move on. If REL orbit is bad or beam is lost, then "Restore Setup", re-evaluate, try again

Bypassing the gun 15 deg dipole magnet

In the instance that the gun high voltage power supply needs to be turned on (high voltage conditioning, for example) and the 15 degree dipole magnet cannot be energized, its interlock can be by-passed. UITF example bypassing the gun dipole magnet interlock

It is very important to understand that when this is done, the gun beam line gate valve must be locked out CLOSE. This closed valve is then the only protection to the laser input viewport from potential field emitted electrons

Wien Filter HV System: description, procedure and troubleshooting

Media:CEBAF Wien filters HV system operational guidelines updated April 11 2022.pdf