Run Checklist
(This page can be found online at:
http://d0server1.fnal.gov/users/strang/web/fpd/documents/run_checklist.html
and the physical paper checklist is also available online at:
http://d0server1.fnal.gov/users/strang/web/fpd/documents/Run_Checklist_print.doc (as a Word document).
For establishing operating positions, use the following physical checklist:
http://d0server1.fnal.gov/users/strang/web/fpd/documents/OpPos_Checklist_print.doc (as a Word document).)
This is a list of all tasks that need to be performed when pots are to
be inserted close to the beam. If additional information is needed on a
task, click on the link to get to the detailed information. Some items on
this list only pertain if pots are being moved. Ignore them if you are not
moving pots.
Review the Emergency
Procedures before starting to move the pots so they
are fresh in your mind.
Also review FPD Issues to understand the current state
of the system.
!!!!!IF ANYTHING STRANGE HAPPENS, TURN OFF CONTROL LINES!!!!!
Afterwards, try looking at Troubleshooting
to resolve.
FPD Station Layout (Desktop locations for different programs)
Please refer to the map above the console for the location
where programs should be started.
Status Verification
- Start filling out the physical checklist. The physical checklist is a
.doc file named Run_Checklist_print.doc in the same directory as this file.
See address above. This online checklist has more details than the paper
checklist with links to detailed instructions. If you are establishing operating
positions, use the OpPos_Checklist_print.doc file from the same directory instead.
- Read and understand the daily
runplan.
- Login to the elog book to record all pertinent information.
Instructions can be found at
FPD elog book quick instructions.
- Read the last 8 days worth of FPD information in the log book and
emergency incident book to see what has been done recently and if there are
any problems. If you can't fine enough entries in the online machine, go to
the browser and go to:
Online logbook to
read these entries. (On the left hand side, unselect INCLUDE_ALL and
select FPD and only FPD entries will show up.)
- Review FPD Issues to understand the current state
of the system.
- Check that the emergency line and both control lines (A and P) are
off. These are located and labeled above the monitor at d0ol49.
- Verify that
the Pot Motion Software is running in the correct desktop. It must be
running before you can proceed. All bars should be blue.
- If any LVDT bars are steady red in the Pot Motion Main Screen there
is a hardware failure of some kind. See the
Hardware Failure
descriptions.
- If the LVDT bars are orange in the Pot Motion Main Screen, the
A/S have been turned off and need to be turned on.
- If any bars are yellow, the system is attempting to initialize these
pots and bring them home. Either turn on the control line and drivers and
let this happen or wait until the software gives up.
- Verify that the
HV software is running in the correct desktop.
- Verify that the
Rate Watcher program is running. This is a chronjob program so it should
automatically restart itself if it isn't running.
- Verify that the
SES Alarm Display program is running.
- If there are any Minor or Major alarms, attempt to clear them. You can
use the guidance offered in the SES program for some ideas of what to do to
clear the alarm.
- Verify that the Rack
Environment Monitor program is running.
- Verify that the ACNET screen to the right of d0ol49 is setup correctly.
Details of the ACNET windows can be found
here.
- Verify that the Halo rates are being updated properly in C72<4>.
- If needed, start the LMB
Control program. This is needed if you are going to use the LMBs.
- If needed, you can spy on
Epics Values using these programs. (Especially Low Voltage Power Supply values).
This is needed if you don't trust the readback from another program.
- If needed, start the LM
expert program. This is needed if ACNET is not working properly in
order to observe D0 values of Luminosity variables (p-halo, a-halo etc).
Also to debug where the problem is if necessary.
Initial Preparation
- Turn on the
HV to the L0 and Veto PMTs using values in runplan. You can use SET
100% to automatically set all of the voltages and then ramp.
- Rates should be updating in the Pot Motion Control Main screen.
- Look over the examine plots for anything unusual. If you find something,
make a copy using the import command and place in the logbook along with
a comment of what looks strange.
- Save a screencapture of the lumberjack plot from the previous store
in the elogbook.
- Go to the FPD Run State page and save
a screencapture in the elogbook. This keeps track of the singles rates, luminosity and halo rates.
If this page isn't accurate, notify Vladimir Sirotenko and make copies of the Beams division page (
Beams Division Status Page), the halo rates from
C72 and the FPDGui instead.
- Verify with the shift captain that the beamspot positions hasn't significantly changed.
- Verify that the cameras
for the pots to be moved as given in the runplan are working.
- Start the halo view software to give it time
to start integrating. Use it during insertion to determine when halo limits have been reached.
- Open expert mode in the FPDGui.
- Verify with Shift Captain in the D0 MCR that we have permission to move
the pots. He will check with the MCR and will notify MCR when we are
ready to move the pots.
- Unless instructed by an uber-expert, do not use the emergency line
for anything. It should regularly only be used at the end of stores when
pots are not responding to software commands.
- Pots are moved in the order: Dipole, A vertical, A Horizontal, P vertical,
P Horizontal
- Make sure
the Camera Monitor is showing the proper motor(s) for the pot
you are going to move. You cannot see down pots.
- Turn on the driver for the individual castles you are going
to move. Go to COMMANDS->DRIVER and select the
castles you want to power. In the Driver Status panel and A/S Status panel,
the powered castle should turn green showing it has power.
- Turn on appropriate 5V control line for the side you are going to
move leaving the other off. Move all pots on one side before going to
the other.
- Issue the move
commands to the pots.
- Verify both through the Pot Control software and the camera that the
pots are moving and that they stop at the right location.
- Look in Pot Control software that the LVDT bar is blue for
all pots except the one(s) you are moving which will be green. If any
others are green, see Emergency Procedures.
- If any LVDT bars are red there is a hardware failure of some kind.
Stop the motion you are doing and see the
Hardware Failure
descriptions.
- Turn on MAPMTs as each group is inserted (dipole, a pots, p pots) using
the instructrions in Data Collection instructions below.
- When all desired pots are inserted, turn off the drivers (and control lines
if appropriate) and move the next set repeating the above commands.
- When finished, make sure both control lines and the emergency line is off.
- Close Expert window in FPDGui.
- Switch ACNET so that it reads out D0 halo, CDF halo and SCALER values
for the pots being read out by the DAQ (D2I, A2U, P2U, P1I upper limits to 400000).
To do this, replace the final l in the variable name by s.
- Make a picture of these variables for the current insertion and place in the
elogbook.
- Turn off
the cameras.
- Send a mail to d0fpd_local@fnal.gov
about the insertion. Include the final pot positions and any information
about the insertion.
- Notify Shift Captain that all pots have been inserted.
Collecting Data (integrated)
- If the rates are below ~350kHz,
turn on the HV to the
MAPMTs. You can use set 100% to do this followed by ramp. Record what time
you turn on each set of tubes.
- Verify that the examine is setup
and running properly. (i.e. you see correlations and stuff above pedestal).
- Record the current D0 runnumber in the elogbook.
- Place insertion information in the elogbook.
- Instruct the captain about the tasks they will be responsible
for.
- Monitor rates (ignore D0 halo spikes if no other rates increase)
- Describe Rate watch alarm
- Should be decaying with time. Gradual oscillations ok. If spikes, check
tubes. If sustained increase, or numerous spikes or anything else strange,
page expert. Check every 30 minutes or so.
- Monitor alarms. If there are any LVPS or HV (major or minor) or any major
scaler alarms.
- Rescrape: Check with MCR if it is minor or major. If it is minor, do nothing
special but watch for trips. If it is major, ramp all voltages to standby, when
scraping is finished ramp voltages back to 100%.
- Remove pots and end of store and put system in standby (at 15 minute EOS
call from MCR)
- Issue a stop command to the examine
- Call on call expert when Luminosiry reaches 45 to come insert pots or anytime
if there is an emergency.
- Record nonexpert shifters name in logbook
- Set MAPMT HV to standby (0%).
- Turn on Control Lines.
- Move pots
to the home position using the POTS HOME->INIT command from the main
FPDGui screen.
- Verify that all pots are at home from the C72<1> screen. All locations
should be at 1. If any are at 0, issue another INIT command until
all are at 1. When they are all at 1, drivers and A/S should turn off
automatically.
- Turn off the Control Lines.
- Ramp HV to
L0 Tubes and Veto Tubes to standby (0%).
- If Drivers are not turned off automatically (all tabs should
be orange if they are off) verify in C72<1> that all pots have home bit
set. If they are all set:
- Turn off the drivers in the Pot Motion Control Program.
Otherwise, follow instructions above to bring all pots home.
- Record the current D0 runnumber in the elogbook.
Emergency / Strange Behavior
EMERGENCIES: IMMEDIATELY TURN OFF 5V CONTROL LINE if they are on (bring
problem pots home when control is restablished at end of store). Click on link below for
procedure to recover from problem.
If you have a Driver Failure, try turning the Control Line on and off to
clear it.
ALARMS THAT AUTOMATICALLY STOP POTS
- Rate exceeds 500kHz and control line on. Turn off control line. Try to determine
why rate got so high. If it is not due to a failure of the pot motion, and the pot can be
retracted to a safe operating position, lower trigger HV (to bring rate slightly below 500kHz),
retract pot and return voltage to 100%. Cannot move pot until rate is below 500kHz.
- LVDT value exceeds software limit. Turn off control line. Try to determine why
pot moved to software limit. It is safest to wait until end of store to try to remove the pot.
- LVDT value exceeds wanted position. Turn off control line. Try to determine why pot
didn't stop. It is safest to wait until end of store to try to remove pot.
- LVDT direction doesn't match wanted direction. Turn off control line. If the problem was
because of an LVDT fluctuation, you can issue the command to move again with no problems. If the
alarm was for any other reason, wait until the end of store to try to remove the pot.
IMPORTANT EMERGENCY PROBLEMS
-
Pot Motion Program freezes or PC crashes.
-
Power PC (VME Crate) crashes.
- Pot already
at final position (position closest to beam) starts to move further in.
- Pot does
not stop at designated final position (position closest to beam).
STRANGE BEHAVIOR OF MOVING POT: ISSUE
INDIVIDUAL POT STOP
COMMAND, ALL STOP COMMAND OR EMERGENCY STOP COMMAND (bring problem pots
home when control is restablished).
Click on link below for procedure to recover from problem.
- Pot starts
moving home by itself (if stop doesn't work, let it go home
and make sure it doesn't come back in).
- Pot part way
in starts moving in by itself (if stop doesn't work, turn off 5V).
- LVDT is not
updating while pot is moving (if stop doesn't work, turn off 5V).
- Pot doesn't
stop at the designated position which is not final position (if stop
doesn't work, turn off 5V.)
STRANGE BEHAVIOR POT NOT OBEYING COMMANDS. Click on link below for
procedure to recover from problem.
- Pot doesn't
move when given move command by software.
- Pot doesn't
move home when given command by software.
- Pot doesn't
move home with emergency line.
OTHER STRANGE BEHAVIOR. Click on link below for procedure to recover from
problem.
- Rates are not
updating.
- HV not responding or tripping.
- ACNET not updating.
- Major SES Alarms.
- Examine not working.
SMOKE ALARM
NOTE: Spurious Smoke Alarms appear at beginning of stores during shot setup.
This appears to be due to Rack Monitor power during shot setups.
- The software logs all smoke alarms. If an alarm persists for a long
enough period of time, an alarm window will open on the current active
desktop and a FIRUS alarm will be triggered. [A Smoke Detected
(FPDRun) is not a real smoke alarm]. Press reset on the alarm
window. Also, use the cameras to look in the areas shown in the alarm
window for any signs of smoke. If smoke is present, contact the Shift
Captain to have the appropriate authorities contacted.
If the alarm does not reset, use the
Alternate Fire
alarm reset resetting the appropriate alarm in the tunnel tab.
If the alarm window reappears, turn off all 5V circuits and restart the
Pot Motion Software. When the software is once again active, try to reset
the alarm. This sometimes works when the reset alone doesn't work.
If the alarm does not reset and there is no smoke, we need an access to
the tunnel to determine the problem. We cannot move the pots until the
alarm is reset.
ANYTIME THERE IS A SMOKE ALARM WHEN A/S ARE ON, AFTER RESETTING ALARM,
BE SURE TO TURN THE A/S BACK ON AS A SMOKE ALARM IS SUPPOSED TO TURN OFF
THE A/S.
INCIDENT REPORT
If anything strange happens that requires an emergency procedure to fix, we
need to create an incident report. This should be sent to the dofpd_all
list with the detailed report of the run as well as recorded in an
incident log book in the SCR.
Gather as much of the following information as possible:
- Position of all pots when problem occured.
- Which pots were moving and how they were moving.
- Status of the computer and the VME crate.
- Time of the problem.
- Methods attempted to fix the problem.
- Results of attempts.
If you have any questions or comments, please mail me
at: strang@fnal.gov
This page last updated: 08-Mar-2005 01:02 PM (GMT
-06:00)