Essential Shift Instructions

1. Login into the logbook at the beginning of the shift.
2. Log everything you do in the log book:
1. Drag the TEXT icon into the log-book.
2. Make your entry, insert an image via the "Insert Image" menu. To make the images: in directory /home/d0smt/SHIFTS/yyyy/mmm/dd (e.g. /2002/feb/01) enter "import [return]"; then click on the window you want to grab.
3. Archive your entry immediately (select entry with ctrl-left click and archive from the right mouse button menu) to note the proper time.
3. Keep the run going
1. Make sure none of the SMT crates is 100% Front end busy. You can see this in the DAQ Monitor GUI, but normally the DAQ shifter will also tell you. If it is try the following until the 100% front end busy disappears:
1. Ask the DAQ expert for a SCL init.
2. Can you find a HDI responsible? One or many HDIs in a row showing high DVDD current (in the SMT DVDD CURRENTS GUI) can stall the readout. Ask for a run pause and redownload those HDIs from the corresponding SEQ in the download GUI (SMT Calibration GUI).
3. Ask for a run pause and do a "Reinit VME" in the SMT Download GUI
4. Ask for a run pause and do a "download" for that crate (same GUI)
5. If everything of the above fails: Ask for a run pause, power cycle the corresponding VME crate, do a "reinit VME" and continue the run. How to powercycle the VME (=VRB) crate: 2nd floor Movable Counting house, upper module in each rack, big 3 pull switch right above sector "N" sign. The crates are labeled by their geographic sector (e.g. 0x61). Push the black reset button on the VBD after this until the "error" LEDs go off.
2. Check for HV trips. If a HV pod is tripped (red in global HV display - "smtAll"):
1. Go to the detailed Tab for the HV crate GUI ("HVC Channel Monitor Display").
2. If "V_read" shows about 0 V skip the next item.
3. If "V_read" shows the opposite polarity of "V_max" ramp the neighboring HV pod down to 0 Volt. Hitting the "Reset" button (at a rate of 5 Hz) while doing so might speed up the process. If this HV pod shows "Pause" state, hit the "Resume" button. Do this until the reset for the tripped HDI is successful. To ramp a HV pod, unlock it first
4. Hit the "Reset" button (this will only reset unlocked HVs!).
5. Select "Set HV"->"100%" from the global HV GUI (smtAll)
6. Hit the "Ramp" button.
7. When the pod state turns to "Holding" lock all HV pods.
8. Note pod and time of the trip in the log book.
3. Check for high IDVDD current (red) and tripped (red and black blinking) HDIs ("SMT DVDD CURRENTS" GUI top screen):
1. If the IDVDD current is too high (red): redownload this HDI from the corresponding SEQ in the download GUI ("SMT Calibration")
2. If the low voltage is tripped (red and black blinking):
1. Ramp down the HV for all HDIs in that Interface board (click on the corresponding "INT_NW_1A02" button on the download GUI to get the HV GUI for this Interface Board).
2. Switch off the corresponding HV pods.
3. Power off the tripped HDIs: Click on the corresponding HDI and select "power off" from the menu.)
4. Redownload those HDIs: Click on the SEQ_nnmXii button and select "download".
5. Ramp the HV back up
3. Note the trip in the log book.
4. If an HDI tripped and revival attempts fail:
1. Ask the DAQ shifter to pause the run.
2. Left click on the HDI from the download GUI.
3. Select "power off".
4. Right click on the HDI.
5. Select "disable".
6. "Reinit VME" for that crate.
7. Ask the DAQ shifter to restart the run.
8. Make a log book entry noting the IB and HDI and the symptom (trip, high DVDD, no download).
5. If there is no global run going have a dummy (SDAQ) SEQ run going to reduce IDVDD current:
1. Start a Taker (command taker after setup d0online)
2. Modify -> Change Trigger -> commissioning/tracking/zero-bias-smt-seqall-sr-2.0 -> START
3. Stop this run as soon as the global run is going to start again. The taker window in which you started this trigger will now have a "STOP" button to do this
6. If this does not help (DAQ/TFW 100% L1 or L2 busy, master clock problems):
1. Select "pulses on" from the global "SMT Download GUI".
2. When normal datataking continues select the "pulses off" button.
7. Monitor the interface board temperatures (in the IB CRATE TEMPERATURES GUI
4. Monitor data quality
1. SMT examine instructions are in the red folder labeled "SMT Examine Guide".
2. Run examine and compare the plots with the reference plots.
• Log on to d0ol22 (setup d0online; d0ssh d0ol22)
• /online/examines/t02.08.00/smt_exam/bin/Linux2-KCC_4_0/run_online.sh (you get a window called "SMT_histos")
• Type "init" and "start" in this window as requested.
• Look at least at the cluster charge for all barrel detectors in the summary folder. If you want to look at additional information, go up a level in the SMT_histos folder and look at the cluster charges for the detectors corresponding to the plot you want to check in more detail (select the plots you want to see with ctrl-left click). When you find out which ladder/wedge is responsible for a bad cluster charge distribution you can try to find out why by looking at the average pulse and event display for that detector. You can try redownloading it.
• At the end of the run save the display in the e-log (directory ~d0smt/SHIFTS/2002/month/day/shift, command "import pictureName.gif", elog menu "Insert->Image->from a file")
3. Run the occupancy GUI ("SMT Occupancy MONITOR").
• If there is continuous high occupancy (marked red) for some HDIs save the display in the e-log.
• If a single VRB is continuously high you can try to redownload it. If a single HDI is continuously too high, try to redownload it.
• If the overall occupancy is too high (the DAQ expert may complain) call an expert.
4. Run the SMT event display.
5. Change HV if necessary (in the smtAll GUI.
1. Ramp the HV up on request of the shift captain:
1. ON button
2. Set HV -> 100%
3. RAMP button
4. If any pod trips: RESET immediately, then click RAMP again.
5. When all pods are fully ramped an holding (green) LOCK the pods.
2. Ramp the HV down on request of the shift captain:
1. UNLOCK the pods
2. Set HV -> 0
3. RAMP
4. After a value <5V is reached for all pods (takes about 3 minutes) select OFF.
3. HV should be up during a global run while a good store is in the machine.
4. HV must be off:
1. BEFORE the beam is aborted
2. During beam studies
3. If the beam is not stable or losses are too high
4. During scraping
6. Take Calibration Runs whenever possible
1. A new calibration should be taken at least once a week.
2. HV is needed for the calibration. Good opportunities are:
• During an access when no work on SMT is ongoing
• After an access while there is no beam in the machine (ask the shift captain)
• During agreed on quiet time.
3. Follow the instructions for SDAQ calibration run (section "short/long version calibration" in the instructions binder)
7. What to do when the SEQ power trips
1. Symptom: the IB (SMT DVDD CURRENTS) GUI will show that large areas are off (orange) without apparent reason. This procedure will take about 30 minutes.
2. Start the "SEQ LV Power Supply" GUI if needed (seq_gui.py)
3. Verify that one supply is really off and causing the problem.
4. Inform the shift captain.
5. Make a log book entry.
6. Notify the CFT shifter of the trip and that you want to switch off the power of all of their south/north sequencer and DFE supplies. The SMT and CFT power supplies are linked.
7. Turn off the HV for south or north (unlock them from the HV Channel Monitor display, then ramp down in smtAll.
8. Switch all HDIs in that detector half off: "HDI off" for all odd crates for the south half and for all even crates for the north half.
9. Details South/North -> Power Control -> Off -> Yes from the seq_gui.py ("SEQ LV Power Supply")
10. Check whether the IB power supplies are OK (IB LV PS GUI)
1. Click on the grey bar saying e.g. SMT_LV_SW0
2. Verify that all fields are green
3. If e.g. AVDD-8/9 has crowbarred (it"ll show up red):
4. Switch the power Off for that supply
5. Go to the 3rd floor, rack M301, covered AC power switches are at the top of the rack, labelled "IB SMT enable switches". Turn the corresponding switch off, wait 1 minute, turn it on again, flip the cover back on.
6. Switch the power supply back on (in SEQ LV Power Supply-> details).
11. Turn the North / South SEQ power on again after it has been off for at least 3 minutes (Details South/North -> Power Control -> Off -> Yes from the seq_gui.py).
12. Redownload the south/north crates and turn HV back on.
8. Store the channel archive (Friday day shift)
1. Follow the instructions in section "channel archiver".
9. Send a shift summary e-mail to d0smt@fnal.gov and logout from the logbook.

• DAQ Monitor:
• smtAll:
Global High Voltage control
SMT DVDD CURRENTS:
DVDD current per HDI
SMT Download GUI:
Download per crate, or (via "Details") by sequencer, by HDI
SEQ LV Power Supply: /home/d0smt/Monitoring/seq_gui.py
Control sequencer power supplies
HVC Channel Monitor Display:
High voltage control per pod
IB LV PS:
Interface board low voltage control
SMT_Histos: /online/examines/t01.67.00/exam_new/bin/Linux2-KCC_4_0/run_online.sh
Examine; from SMT_histos you can select what plots to look at
SMT Occupancy MONITOR
Occupancy per HDI