Emergency Recovery

Do not use the Emergency Line unless you know what you are doing. In almost all cases, wait for instruction to use it from an uber-expert and then it will almost always be only at the end of the store, not during a store. If something breaks during an insertion, we will regularly leave it in place until the end of store and try to fix it later.

Pot Motion Program Freezes or PC Crashes

• Turn off all 5V control lines if they are on.
  
  
• Check that d0olctl38 in the MCH1 Porch is not in an error state. If it is, reset it using instructions below.
  
  
• If this doesn't fix the problem, close all pot related programs (Pot Motion, HV and Emergency Stop) programs if the computer is responsive.
  
  
  • If you can't close them the normal way, use a ps command at an xterm prompt to get the processid and use a kill -9 processid command to close them
  
  
• If computer is not responsive or crashed, reboot the computer. Issue a Shutdown -h now command from an xterm connected to d0ol49. After loging back into the machine, press Ctrl-Alt-+(from the keypad) to change the resolution.
  
  
• Restart the Pot Motion Program.
  
  
• When the program is started, if all pots are not at home, an initialization window will open which reads "Pots are not in home, move them home first".
  
  
• To bring the pots home, acknowledge the message and turn on the 5V control lines. You will see the LVDT bars change color (from blue to red and back to blue) as the power is turned on at the castle. When it has returned to stable (blue), go to the Pots Home --> Init button. The amplifiers and drivers will be turned on, the pots will come home at high speed and the amplifiers and drivers will turn off. Then turn off the control lines.
  
  
• If the pots were at the final operating positions when the program froze, acknowledge the init message and do nothing further. Remember that the pots can only be brought home once the program has been restarted after the pots have been inserted. DO NOT try to insert the pots any further.
  
  
• Restart the rest of the programs. Use the instruction from the checklist.
  
  
• Try to determine why program froze.

Power PC Crashes

• Turn off all 5V control lines if they are on.
  
  
• Reboot the Power PC by pressing the reset button. If not successful, turn the power off and on.
  
  
• If you were in the middle of a run, end the run, mark it as bad.
  
  
• Verify that the Pot Motion Control Program is correctly communicating with the RM in the tunnel (i.e. correctly reporting LVDT values and states of pots before the Power PC crashed). If it is not, follow the Pot Motion Program recovery above.
  
  
• Investigate why Power PC crashed.

Pot in final position starts to move in or doesn't stop at final position.

• Turn off 5V control line for that pot.
  
  
• If a window popped up informing you of the pot movement, something external to the program caused the pot to move.
  
  
• If the pot started moving and it was noticed only because the LVDT bar in the Main Screen of the Pot Motion Software turned green then the software issued the command. Look in the software log for possible reasons. (momentary alarm, miscommunication with the RM etc.)
  
  
• If the pot didn't stop moving at the required position, I have no idea what is wrong. It needs to be diagnosed by experts before the pot can be moved.
  
  
• If the reason for the pot motion is determined and we are confident that the pot will not move any further in then turn on the 5V line and bring the pot home.
  
  
• If we are not sure the pot won't move further in we can't risk turning the 5V back on. Retract any pots on the other 5V circuit, turn all 5V off and wait until the store is dumped to try to remove the pot. If the pot cannot be removed, we need to demand an access to manually retract the pot. Another store cannot be inserted until the pot is retracted.

Pot starts moving home by itself.

• Issue self stop, all stop and emergency stop commands, in that order until pot stops.
  
  
• If pot stops, try to determine why it moved home. If it reached the end of a move, then started home instead of stopping that is a feature of the software if it isn't sure of the location of the pot.
  
  
• If pot doesn't stop, let it continue home and make sure that it stays there. If it starts moving in, see the Pot starts moving resolution.
  
  

Partially inserted pot starts moving in by itself

• Issue self stop, all stop and emergency stop commands, in that order until pot stops.
  
  
• If pot doesn't stop, turn off 5V control line.
  
  
• Look at logs to try to determine why pot started moving.
  
  
• If we are confident pot will not move further in, turn on 5V and bring pot home.
  
  
• If we are not sure the pot won't move further in we can't risk turning the 5V back on. Retract any pots on the other 5V circuit, turn all 5V off and wait until the store is dumped to try to remove the pot. If the pot cannot be removed, we need to demand an access to manually retract the pot. Another store cannot be inserted until the pot is retracted.

LVDTs not updating for pot that is moving

• Issue self stop, all stop and emergency stop commands, in that order until pot stops.
  
  
• If pot doesn't stop, turn off 5V control line.
  
  
• Look at logs to try to determine why LVDT is not working.
  
  
• If proper LVDT contact is made, return the pot home to reset the home value then reinsert the pot.
  
  
• If LVDT contact cannot be made, bring the pot home, using the camera to verify that the pot is moving. Wait for expert determination of the problem before the pot is moved again. (The software doesn't actually use the LVDT at all, it is just a check against the step motor for us to look at).
  
  
• If other aspects of the system are not working, check the VME controller.

Pot doesn't stop at designated position which is not final position

• Issue self stop, all stop and emergency stop commands, in that order until pot stops.
  
  
• If pot doesn't stop, turn off 5V control line.
  
  
• If you issue a stop command before pot naturally stops you can only continue a move in the same direction or bring the pot home.
  
  
• I have no idea why the pot wouldn't stop. The problem needs to be diagnosed by experts before the pot can be moved.
  
  
• Do any physics possible in the runplan with the other side of the detector. The pots on the problem side need to remain in place until the store is dumped, then the pot needs to be removed. If it won't come out by software, we need to demand an access to manually remove the pot before another store can be inserted.

Pot not moving when given software command

• If the pot is moving at high speed then suddenly stops, the software will attempt to continue moving at low speed. The motor will keep trying to move the pot, but it will not be seen spinning in the window.
  
  
• Verify that the Emergency Stop button is not activated as it will prevent any pot motion. If it is, turn it off, wait a few minutes, and then try moving again.
  
  
• If the pot still doesn't move, issue a stop command, wait for a little while then try moving the pot again at low speed.
  
  
• If the pots are inserted and they need to be moved home but they are not obeying the home command from the software and you have verified everything else (especially emergency stop) then turn on the emergency line. When the pots start moving, turn off any control lines that are on. When the pots return home, turn off the emergency line. If the pot gets stuck while moving under the emergency line, see the next procedure.
  
  
• If the pot still won't move after this, turn off the 5V and we will need to demand an access after the store to manually retract the pot.
  
  
• If the pot stops moving but the motor keeps turning, the coupler has broken. We need to demand an access to the tunnel to replace the coupler and retract the pot.

Pot not moving when emergency line is on

• Wait until all other pots are home then turn off emergency line.
  
  
• If the pots will obey the software turn on the 5V for that pot and try to move the pot out a small distance at slow speed. If the pot moves, bring it home at high speed with the software.
  
  
• If no pots are moving, check the Emergency Stop button. If it is activated it will prevent the pots moving. Turn it off, wait a few minutes then repeat the attempt to move the pots.
  
  
• If the pots won't obey the software, wait a few moments, then try the emergency line again. If it still doesn't work, we need to leave the pot in place until the store is over and try to reastablish control with the software. If this doesn't work, we need to demand an access to retract the pot.

Rates not updating

• If there is no beam, rates will not update. This is to be expected. If the rates are frozen at a high value causing problem (triggering rate alarms), reboot d0olctl38 to reset the rates to a default value. You can do this by going to the telnet screen connected to d0olctl38 (the one that shows scrolling scalar information) and typing ^X or going out onto the porch and pressing the red reset button on the d0olctl38 module.
  
  
• Check that the scalar software is working.
• Check that the external pulser is working and connected to CAMAC crate.
• Check that the CAMAC crate is on.
• Check that the HV is on.
• Check that the discriminator is working.
• Repower the trigger electronics using the instructions in this binder or from the link at the end of the details section.

HV not working

• Check that the appropriate crate is on.
• Check that the appropriate VME controller is working.
  
  
• If a particular channel is tripping off because it is reaching the current limit, you can make a temporary change to the epics limit value. For instruction on how to do this, follow the epics spy instructions to temporarily increase the limits.

ACNET not working

• Check with Vladimir Sirotenko that the ACNET Gateway is up.

Major Alarms

• Depending on alarm try the appropriate step to fix it.
  
  
• You can use the guidance button in the SES display to get some instructions on what you can try.

Examine not working

• Not really sure.

Hardware failures

This page last updated: 08-Feb-2005 02:40 PM (GMT -06:00) v1.6