|
|
Lightning Triggered Shutdown |
|
Schedule Information
Store 8594 terminated on a quench due to a
lightning strike
Accelerator experts were unable to
re-establish Tevatron ramp at 980 GeV
Experts report a suspected failed Tevatron dipole at D16-2 (turn to turn short in winding is
suspected)
Warm-up of the Tevatron
D sector is in progress
Likely down for at least seven days to
replace the failed dipole magnet and a spool piece
Current projection is that stacking will
resume on Tuesday 29 March 2011 during the day shift
The Tevatron will
be cold again on Wednesday morning 30 March
Shot setup estimated for Thursday evening 31
March
Run magnet off cosmics
through 0700 on Thursday 24 March
Ensure that calorimeter calibrations runs are successfully completed
Then arrange to have the solenoid and toroid ramped down and locked out
Ramp SMT and muon systems to full
Record magnet off cosmic triggered data (only TTK prescales including CAL noise monitoring) with standard configuration of readout crates included in the run
Shift crew should routinely monitor dataflow and examines, filling out checklists regularly, with runs transition at least every eight hours
Requests for other activities or access to the collision hall should be directed to the run coordinator team, but expert activity anticipated during day shifts
Open North EF and cathedrals Thursday
morning
Expect access available by 1100
While detector is open run zero bias or cosmics with all crate that are not required to other purposes by subsystem experts, but ensure that recording is OFF
Verify detector ready for closing by Monday
evening at 1800
Close detector again at 0700 on Tuesday 29
March
Ramp magnets after detector closing (to
ensure that detector is fully closed)
Acquire Calorimeter and SMT calibrations
Record magnet on cosmics
Monitor examines and alarms and address
features asap to ensure that the detector is ready
for collisions when they arrive
The Fermilab program planning schedule is available via this link.
All
requests for survey and/or alignment crew support during the shutdown should
have been submitted through Bill Cooper (Cooper@fnal.gov)
with copies to d0rc@fnal.gov.
Candidate Task List for This Shutdown
Infrastructure
· Complete repairs on Trane
chiller and test?
· Chiller
maintenance (not this shutdown)
· Replacement of
Mycom relief valves during power outage? (not this
shutdown)
· Eliminate
remaining daisy chained extension cords during power outages (not this
shutdown)
· Perform surveys/alignments (not this shutdown)
· UPS
maintenanceCryo and HVAC (not this shutdown)
· Address warm
electrical panel?
· Improve radiation
shielding at low beta quads? (not this shutdown)_
·
Further
stabilization of the low beta quad girders? (not this shutdown)
· Routine maintenance
o
Check transfer line vacuums and pump as appropriate
o Warm up
solenoid lead? (not this shutdown)
o Warm up VLPC
cryostats to 40K (not this shutdown)
o
Warm up west VLPC lid gasket?
o Check Freon
charge on silicon chillers?
· Investigate possible water
leak in east cathedral between SMT power supplies
· Perform safety
system tests (not during this shutdown)
o Test remote
power down buttons?
o Check torques
on solenoid bus connections?
o Test rack
protection systems?
· Crate/Rack maintenance
o
Verify blanks in open slots of crates?
o
Clean heat exchangers as appropriate
o
Check status of fans/blowers
o
Check status of Cal preamp fans
Silicon
Microstrip Tracker
· HDI recovery effort ---in progress
o
Download failures
o
Investigate currently disabled chips
· Continue investigation of
response as a function of Time since Level 1 accept ---in
progress
· Continue preparations for optimization
of operating parameters (disk HV and SVX parameters)
· Add resistance
to SMT AVDD power supply feedback to eliminate possibility of oscillation?
· Adjust SMT IB
voltage limits to reduce possibility of oscillation?
· Verify stability of HV
system and check HV calibrations
· Adjust Sequencer PS voltage
levels
· Complete R2D0 installation
and commissioning ---installation complete 25 March
2011
· Address LVPS readback issues?
· Check silicon
chiller setpoint calibration and adjust as
appropriate?
· Updated calibration
Central Fiber Tracker
· Continue investigation of
updated AFE readout firmware ---offline
verification underway
· Replace AFE with failed TripT ---complete
· Update VRBC firmware (noise
immunity for buffer counts after SDAQ verification)?
· Address light leak in FPS
readout system
· Adjust 5V in AFE power
supply 2B?
· Stabilization
of AFE power supplies? (not this
shutdown)
· Thermal cycle
of cryostatreset heater operating points to 9K
(not this shutdown)
· Complete
Temperature Control fix implementation
(not this shutdown)
· VRB re-assignments
to balance loadsfor high lumi running (not a
preferred activity)?
· Testing AFE spares on the
platform?
· Hunt for source of occasional
AFE power supply trips when platform is occupied
· Hunt for sources of rare
coherent discriminator noise?
· Check on the
power distribution for the AFE fans?
(not this shutdown)
· Investigate
re-biasing of VLPCs? (not this
shutdown)
· Preventative
replacement of aging LVPS in cathedral???
(not this shutdown)
CALorimeter
· Monitor for calorimeter
noise ---in progress
· Top off liquid
argon levels?
· Turn off the
CAL HV during a long shutdown (not this
shutdown)
· Address cal preamp power
supply failures PA02P, PA11P and perhaps
PA05P---in progress
· Check the status of cal
preamp fans
· Check CAL HV supplies calibration
· Another round of CAL pulser tests to check for SCA failures
· Address individual channel
issues
o
Isolate sparking channels
· Investigate ICD channels
that are still not performing up to expectations
o Individual
channel recoveries
o PMT
replacements
o
HV stabilization
o
HV adjustments
o
Additional
ICD high voltage optimization
o
Check
on status of potentially mis-routed ICD pulser cable
· Opto-isolator check for all BLS supplies??? (not
this shutdown)
· Improve
calorimeter feedthrough port monitoring? (not this
shutdown)
Central
MUON
· Remove two PDT
wires and inspect (not
this shutdown)
· Hunt for
potential PDT gas leaks (not this shutdown)
· Clean HV service cards as
appropriate
· Prepare dry air compressors
for routine summer service---in progress
· Investigate sources of data
flow interruptions---in progress
· Add water to A
Layer PDT gas? (not this shutdown)
· Complete the
process of cleaning up 1553 communications paths?
· Test control boards in
attempt to improve hot link reliability?
· Verify thin FEBs in all
locations in upper cathedral
· Recover failed A Layer PDT
channels ---complete
· Relocate PDT
FEBs to optimize location of FEBs with ATMEL chips?
· Recover disabled A layer scintillator channels if any develop
· HV supply calibrations?
· In-situ tests of SRC spares---in progress
Forward
MUON MDT
· Monitor
radiation aging via cosmic ray response
· HV dark
current measurements
· Test dead and noisy channels
and adjust list of masked channels
· VME LVPS
maintenance (including fans and replacement of fuses)
· Measure ADB PS
output voltages
· Verify calibration of all HV
PS---complete 22 March 2011
· Inspect MDT
gas system
o Test
calibrations and levels of gas losses
· Reprogram CPLD
in FE electronics
Forward
MUON pixels
· LVPS
maintenance (including fans and fuse replacements)
· Verify
calibration of all HV PS
· Individual
channel recoveries
o Replacement of
failed PMTs and bases
· LED calibration of all
counters
o
Verify stability of gains and timing
· Radioactive
source check on calibration of selected channels
· Install spare
synch cables
· Investigate/address First
Crossing Errors
· Test spare SRCs in collision
hall
· Survey pixel
octant locations
Level
1 Central Track Trigger
· Recover failed mixer board
· Investigate events with
large numbers of discriminators fired in cosmics ---in progress
· Investigate satellites in
zero bias discriminator fired distributions---in
progress
· Retire FPS
triggering to reduce probability of downtime during running?
Level
1 CALorimeter
· Hunt for individual cell
noise sources
· Investigate and recover
disabled trigger towers (if any)
· Run VME I/O stress test?
Level
1 Muon
· Perform high rate tests
· Replace L1 Muon trigger
board (to address BCERROR)
· Check for bad cable
connections, parity errors and re-terminate cables as necessary---in progress
· Verify spare status
· Perhaps
replace onboard batteries?
Level
2
· Fix waterleak
in M121 ---complete 25 March 2011
· Test spare SLICs (in online
system) ---in progress
· Investigate source of global
mismatches
· Investigate source of
trigger rate dips
· Enhance tools to identify
source of FEBs
Level
2 Silicon Track Trigger
· Investigate file descriptor
issue in IOCs
· Complete investigation of
the performance problems when beam is near 0?
· Verify error handling and
busy assertions?
Trigger Framework
· Investigate SMT busy
· Update VME scaler firmware
· Test spare TCC?
· Trigger rate tests (deadtime investigations)?
DAQ/Online
· Update kernels of all online
nodes, farm nodes and DAQ cluster---in progress
· Upgrades/maintenance as
appropriate
· Test new SBCs (in CFT
crates on Tuesday) --- status of spares mounted on
extenders remains uncertain
· Test DAQ
functionality with network link disconnected (~2 hour test)?
Collision Hall Access Plan
Current plan is to attempt to
maintain the collision hall interlocks at least until decision is made
regarding whether to reconfigure detector
Rad safety authorized collision
hall activities include
Routine maintenance
Inspections
Troubleshooting
Installation and removal of
equipment in accordance with procedures
Prohibited collision hall activities
include
Eating, drinking and smoking
Cutting, drilling or
grinding on radioactive materials
Work in areas posted as
High Radiation and/or Contamination Areas
Dont forget to frisk yourself
and survey anything that comes from the collision hall as you exit.
Shift Coverage (and responsibilities) During the (Unscheduled) Shutdown
Anticipate that detector shifters may be allowed to
go on-call for a portion of this shutdown (from 1800 on 21 March 2011 through
at least 1600 on Monday 28 March 2011).
Reduced shift coverage is as follows:
· On-call support for each
subsystem should be available as usual.
· Captains in control room
during the day (with additional responsibility to attempt to keep DAQ running
when possible), and DAQ shifter in the control room around the clock on evening.
o
During periods of reduced control room presence, captains and DAQ
shifters are expected to also monitor SMT IDVDD and heartbeat guis to ensure that the SMT is appropriately protected.
· If detector shifters are
on-call, they must report to the control room at the beginning of their shift
(as scheduled---tracking farmer may report in at 0800), and verify the status
of the subsystem (including alarms), perform checklists, and leave a contact
number.
o
While cosmics are running tracking shifter is
also asked to post CTT examine plots and verify that special examine is running
at CFT console
§ If
examine is not running please restart it in the following way: 1) click on
'upper arrow' , 2) type 'init', 3) click on 'Enter', 4) type 'start', 4) click
on 'Enter'. All this from top rightmost field of desktop windows manager on
d0ol102:0:0. When Comics end stop this
examine using Ctrl_c.
o
CALMUO shifter
should monitor for calorimeter noise, post plots as appropriate, and check for
alarms, paging experts for problems they cannot solve.
If the transition to supervised access is made, the
control room shifter is asked to verify that the issued supervised access keys
are accounted for at the beginning and end of each of their shifts.
Additional Reference Information
2011
Com Ed Scheduled
shutdown web page
Fermilab Environment, Safety and Health Section
PPD Implementation of Integrated
Safety Management Policy, Hazard Analysis Procedure and Form
Last updated: 26 March 2011 G.
Ginther numeric page 1-630-218-3217