|
|
2009 Shutdown |
|
Shutdown Schedule Information
DETECTOR is CLOSED.
COLLISION HALL is SECURED.
The
start of HEP is now expected later next week on or before Saturday 19
September.
Store 7183 delivered on 15 September
The latest AD Run Coordinator schedule is available here.
Propose to install NETAPP upgrade on September 10.
The
current expectation is that the 2009 shutdown may be the last several month
scheduled shutdown of Tevatron Run II.
Assuming that the program runs through the end of Fiscal Year 2011,
there may be an order four week additional scheduled downtime for maintenance
late in 2010.
According
to the Program Planning Office, the 2009 shutdown is required for necessary
maintenance, civil construction in the Main Injector area, and other work. The Accelerator Division will make use of the
available shutdown time to re-align Tevatron components and warm six houses to
improve vacuum quality. Power distribution
maintenance will also be performed.
As
of November 2008, the Fermilab Program Planning Office announced that the 2009
accelerator shutdown was scheduled for the ten weeks from Monday, 15 June,
through Sunday, 23 August. As of April
2009, the drivers of the shutdown duration were expected to be the MiNu civil
construction and the booster improvements, leading to a ten week shutdown
duration (with alignment activities during the eleventh week). As of the May 2009 meeting, the plan to
warm-up six Tevatron houses drives the shutdown duration to 12 weeks, with
booster and Main Injector commissioning during the 12th week of the shutdown.
The start date for the shutdown is fixed
as 14 June 2009. An overview of the
DZero shutdown schedule is available here.
Half hour site wide power outages
scheduled for Monday 22 June and Monday 27 July at 0700. The (multi-hour)
power outage at DZero to perform maintenance on the local
power distribution was completed on Saturday 20 June between 0700 and
1530.
The current Accelerator Division
Shutdown Schedule is available here.
All
requests for survey and/or alignment crew support during the shutdown should
have been submitted before 10 April.
Please contact Bill Cooper (Cooper@fnal.gov)
regarding such requests (with copies to d0rc@fnal.gov).
Shutdown
Schedule Overview xls pdf
Accelerator
Division Schedules
Candidate Task List
Infrastructure (including Mechanical
Ops group activities)
· Perform
surveys/alignments---Monday June 15
· Address LN2
Dewar 39 vacuum issue---scheduled to begin July 13—cooldown scheduled for 19
August
· UPS
maintenance—Cryo and HVAC---completed Tuesday June 16
· Install wooden
steps to center platform on west side--complete
· Improve
radiation shielding at low beta quads?—AD installed shielding
· Improve
stability of low beta quads
· SNEG bake-out?
· Routine
maintenance
o
Check transfer line vacuums and pump as appropriate
o
Warm up solenoid lead
o
Warm up VLPC cryostats to 50K—natural consequence of
power distribution maintenance
o
Warm up west VLPC lid gasket
o
Replace ODH heads in collision hall
o
Check Freon charge on silicon chillers
· Address water
leak in MCH1—tightened on June 20
· Investigate
water leak in east cathedral between SMT power supplies—not observed on opening
· Perform safety
system tests
o
Test remote power down buttons--complete
o
Check toroid bus connections—in progress
o
Check torques on solenoid bus connections--complete
o
Test rack protection systems?
· Refurbish all
collision hall blowers that have not been refurbished in past three
years—complete
· Crate/Rack
maintenance
o
Verify blanks in open slots of crates?
o
Clean heat exchangers as appropriate
o
Check status of fans?
o
Check status of
· Volk’s water
level work again
· Replace
luminosity monitor scintillators--complete
Silicon
Microstrip Tracker-- v1---many
activities require gap access
· Complete R2D0
installation and commissioning
· Check silicon
chiller setpoint calibration and adjust as appropriate
· Investigate
possibility of reducing chiller operating setpoint?
· Investigate
possibility of increasing humidity?
· Replace
suspect Interface Boards—in progress
· HDI recovery
effort---underway
o
HDIs that failed as gap was closed last time
o
Recover B1-6-9
o
Data in wrong order
o
Chips with 00 data
o
Chips with 00 ID
o
Download failures
o
Investigate currently disabled chips
· H disk
efforts—in progress
o
H3-2-12 and H3-1-14 IB issue
o
Investigate HV problems
o
Investigate Interface Board problems
· Complete
watchdog installation and commissioning
· Weiner Power
supply diode upgrade—completed June 19
· Address LVPS
readback issues
Central Fiber Tracker
· Thermal cycle
of cryostat—reset heater operating points to 9K----complete
· Address cold
spots in cryostat? (test and run both
heaters simultaneously for 8A7, 2A5, and 9A5)---complete
· Test automated
shutdown of AFE power on AFE temperature excursion—complete
· Investigate
possibility of external crash button for AFE power
· Evaluate and
deploy readout improvement (to save 200 nsec per L1 accept)—CFT and CPS
implemented
· Complete
Temperature Control fix implementation?—only installed on boards removed for
other reasons
· VRB
re-assignments to balance loads?—will not be implemented this shutdown
· Testing AFE
spares on the platform—complete
· Replace 12
tript in CFT with occasional readout issues—(three FPS boards too!)--complete
· Hang
curtains?---investigated and see no evidence that curtains will be
beneficial—complete, no need for curtains
· Hunt for
sources of rare coherent discriminator noise
· Address AFE
LVPS readback issues—complete
· Check on the
power distribution for the AFE fans?
· Investigate
re-biasing of VLPCs?
· Top off liquid
argon levels on SEC and CC---complete
· Address cal
preamp power supply failures---complete
· Replace bricks
in pulser power supplies---complete
· Investigate
root cause of HV related fall 2008 purple haze---complete
· Replace CAL HV
supplies that were out of spec---complete
· Address
individual channel issues---in progress
o
Isolate sparking channels
· Investigate
ICD channels that are still not performing up to expectations
o
Individual channel recoveries—fixed 64 channels
o
PMT replacements---30 PMTs replaced
o
Improve NW pulser connections---complete
o
ICD preamp connector update?---not addressed
o
HV stabilization---25 pods replaced
o
HV adjustments—48 channels adjusted
· Verify output
voltages and voltage trip limits for all HV supplies—complete
· Improve
shielding of cable paths against muon noise
Central MUON
· Remove two PDT
wires and inspect—scheduled for Tuesday June 23 and complete
· Address Crate
0x35 and 0x3a readout issues (if not previously addressed)
· Complete the
process of cleaning up 1553 communications paths?
· Test control
boards in attempt to improve hot link reliability
· Verify thin
FEBs in all locations in upper cathedral---complete
· Recover failed
A Layer PDT—complete
o
PDT 025 L1 Errors and FEB
o
PDT 036 disabled
· Install
additional (second) dry air purge for
PDT HV
o
A Layer PDTs
· Clean PDT HV
service cards---in progress
o
PDT 222, 232, 212 HV distribution---scaffolding on
the west side or over the edge?
o
Install additional dry air purge
o
Including PDTS 115, 135, 116, and 136
· Install FEB
bypass firmware in any bottom PDTs that are pulled out---complete
· Relocate PDT
FEBs to optimize location of FEBs with ATMEL chips
· Recover
disabled A layer scintillator channels--complete
· Repair two
APhi Scintillation Counters
o
At least one phototube that needs to be replaced in
A Layer
Forward MUON
· Check all
Front-Ends and refresh MDT firmware—should start this week
· Verify
calibrations
o
Repair a broken wire for pixel LED
calibration--complete
Level 1 Central Track Trigger
· DFEF
investigations and revival?
· Implement
remote power cycle capability for the Mixer?
· Eliminate
master slave configuration of power supplies (if not previously addressed)
· Blower
replacements
· Hunt for
individual cell noise sources during prep for power outages?
· Investigate and
recover disabled trigger towers (if any)
· Insure that
sailboat monitoring is fully implemented
· Weiner Power
supply diode upgrade—complete
· Exercise/swap
Weiner power supply---complete
· Run VME I/O
stress test
Level 1 Muon
· Verify spare
status
· Address power
supply readback issues
· Check cabling
to make system more robust--complete
· Perhaps
replace onboard batteries?
· Install cable
protection for racks in Platform East entrance--complete
Level 2
· Investigate
source of global mismatches—has not been occurring recently
· Investigate
source of trigger rate dips
· Investigate
cause of timeouts on run starts
· Enhance tools
to identify source of FEBs
· CIC power
supply repair--complete
Level 2 Silicon Track Trigger
· Complete
investigation of the performance problems when beam is near 0
· Address Crate
0x70 reliability issue (if not previously addressed)
· Evaluate
status of cable protections
· Verify error
handling and busy assertions?
· Install cable
protection?
Trigger
Framework—TFW unavailable due to maintenance on Monday 27 July
· TFW
maintenance and stress tests
o
Replace rear fan in M101
o
Install airflow switch in M101
o
Check cooling status in M101, M122, M123, M124
o
How about similar checks in M100?
· Investigate
M122 top scalar crate hangs—in progress
· TCC
maintenance/upgrades?
· Trigger rate
tests?
DAQ/Online
· Upgrades/Maintenance
as appropriate
o
NETAPPS hardware/firmware/software upgrade—new
NETAPP PO placed
o
Improve cluster robustness?
o
New SATABeasts for data disks—
· Test “new”
SBCs---underway
· Additional or
larger monitor for ACNET console--installed
· Test DAQ
functionality with network link disconnected (~2 hour test)—scheduled for 8
Sept
Shift Scheduling During the Shutdown
Expect to operate with reduced shift crew coverage
from 16 June through 4 September, with only one shifter scheduled in the
control room at a time (in addition to the Mechanical Ops shifter). This control room shifter will be
responsible for monitoring detector status and responding to alarms (contacting
experts as needed), powering down the SMT HDIs if the DVDD current draws get
too large, powering down the AFEs if the AFE board temperatures get to high,
addressing requests for DAQ support, including attempting to maintain the DAQ
data flow with as many crates as available in the default global run. General directions for shifters in the
control room during the shutdown can be found here.
In addition to the on-call experts, the following
shift coverage is anticipated during the majority of the shutdown:
0000-0800 DAQ shifter and Ops shifter
0800-1600 Captain and Ops shifter
1600-2400 DAQ shifter and Ops shifter
Supervised access keys are to be issued by only
Captains, Ops shifters, or the Run Coordination team.
If a supervised access key is required after 1600,
or before 0800, the Ops shifter will issue the key.
The control room shifter is asked to verify that the
issued keys are understood at the beginning and end of each of their shifts.
Access Plan
Supervised access keys are to be issued by only
Captains, Ops shifters, or the Run Coordination team. Please make sure to return your key before
leaving the area.
The
DZero collision hall is expected to be in supervised access mode most of the
time between Monday 15 June through Friday 4 September.
Don’t forget to frisk yourself and survey anything
that comes from the collision hall as you exit.
Radiation
survey in the gap yielded maximum readings of 7mR/hour. Only those qualified
for gap access are allowed in the gap.
Pocket dosimeters must be worn while working in the gap.
Additional Reference Information
Fermilab
Environment, Safety and Health Section
PPD Implementation of Integrated
Safety Management Policy, Hazard Analysis Procedure and Form
Last updated: 15 September 2009 G.
Ginther numeric page 1-630-218-3217