DØ Run II SMT Shift Info
last modified by Andrei Nomerotski August 26 2001
Online version: http://d0server1.fnal.gov/projects/Silicon/www/ShiftInfo.html
SMT shifter is responsible for SAFE and stable operation of SMT. There are several important aspects of safe operation :
- Low Voltages (LV) applied to
- High Voltage (HV) to bias detectors
- Radiation environment at the Tevatron
Stable operation includes also
functioning Data Aquisition (DAQ) both for data taking to Level3
(PrimaryDAQ=PDAQ) and for calibrations
Normally, the SMT display in the D0
Control Room will have LV and HV summary GUIs open as shown below
(if not, see the monitoring section how to open them). Generally, any RED spots in those GUI will require your action.
LV GUI shows 8 SMT Interface
crates (SE0, SE1, SW0, SW1, NE0, NE1, NW0, NW1) with 18 Interface Boards
in each crate (ex.SE0A02) and
associated Power Supply (ex.SMT_LV_SE0). Each IB has 8 channels shown as boxes in the GUI. Color coding :
= LV is ON, no error status
Orange = LV is OFF
blinking Red/Black = LV is TRIPPED, error status
Grey = IB channel not connected
pushing PS (ex.SMT_LV_SE0) or IB (ex.SE0A02) buttons will open detailed GUIs for these objects.
HV GUI : (here for South only)
= Ramped up, OK status
Orange = switched ON, not ramped up
Yellow = ramping
Red = tripped
Blue = switched OFF
Though this GUI has HV controls in the top line we recommend to use more detailed GUI to operate HV. It can be activated by pushing HV PS buttons (ex. smts1) and is explained in the monitoring section.
radiation environment is monitored by SMT Loss Monitor and is available via
ACNET on page E35.
If you don't know how to access this information - ask Shift Captain.
Power Up and Down Procedures
To power up SMT from scratch you will need to switch on 3 sets of crates :
1. VRB crates in Moving Counting House 2nd floor (MCH2)
2. Sequencer crates at the Platform
3. Interface crates in the Cathedral
the last two are inside the Collision Hall and are operated remotely via GUIs.
login as d0smt on the online cluster
where the SMT supplies are
SMT_LVSEQ_S/1 - for sequencer crate
SMT_LVSEQ_S/2 - for sequencer crate PC20-1
SMT_LVSEQ_S/3 - for sequencer crate PC19-0
SMT_LVSEQ_S/4 - for sequencer crate PC19-1 ç used by the CFT
SMT_LVSEQ_N/1 - for sequencer crate PC
SMT_LVSEQ_N/2 - for sequencer crate PC
SMT_LVSEQ_N/3 - for sequencer crate PC
SMT_LVSEQ_N/4 - for sequencer crate PC ç used by the CFT
to run summary GUI for PS :
pushing the PS buttons opens individual PS GUI.
to run separate PS GUI :
that brings up the GUI for the crate SE-1 (use SMT_LV_SE0 for SE-0 etc.)
Buttons On/Off switch the supplies on and
off. Note: do not leave those displays running for a long time -
most likely they have a memory leak and after some time (~1 hour) tend to use all resourses of the computer.
To quit : Go to 'File' and 'Quit'
to power SMT off
to run the GUI accessing the Rack Monitor Interfaces :
The monitoring of HDI voltages and currents:
to run global display for IB channels and LVPS status :
> ./IBidvdd.py & to see all IB channels and their DVDD current consumption, see below
> ./IBtemps.py & to see all IB channels and the HDI temperatures
to run IB currents display separately use :
>./SMTIB_wide.py SE0A07.inp (SE0A07 does it for the IB crate SE-0A slot 7, use other names as needed)
The following window should pop up:
Left click on the Status box shows the status.
to monitor temperatures in the IB crates :
each IB crate has 6 temperature
sensors. The temperature is shown in deg C. If a temperature is too high,
it turns red. If 4 temperatures in a crate are higher 35 deg C, this crate is automatically turned off.
a useful tool to diagnoze 1553
(MIL1553 is a Slow Control Protocol for Sequencer and Interface crates)
problems is Mscan. For a given
channel of 1553 controller it outputs a map of responses of all possible addresses of all possible Remote Terminals (RTs).
Normally a good response without errors is green. To run :
>Mscan.py d0olctl40 4 ! this particular channel is used for IB crate SW-0
this is the current SMT 1553 configuration where you can find the 1553 mapping of all seq and IB crates .
SMT uses 4 PPC mv2300/mv2304 in two crates in M300 MCH3
quadrant CPU channel devices cable type
d0olctl57 0 10 SEQ
new black, #1 ç this used to be d0olctl40
1 10 SEQ PC19-0B new black, #2
2 10 SEQ PC20-1B new black, #8
3 3 South SEQC, 4 West RM new black, #9
4 18 IB SW-0 new black, #17 (G)
5 18 IB SW-1 new black, #18 (H)
d0olctl56 0 10 SEQ
new black, #5 ç this used to be d0olctl37
1 10 SEQ PC20-0B new black, #6
2 10 SEQ PC20-1A new black, #7
3 4 East RM new black, # 19 (A)
4 18 IB SE-0 new black, #20 (C)
5 18 IB SE-1 new black, #21 (D)
d0olctl50 0 10 SEQ
new black, #10
1 10 SEQ PC03-0B new black, #11
2 10 SEQ PC03-1A new black, #12
3 --empty --
4 18 IB NW-0 new black, #22 (I)
5 18 IB NW-1 new black, #23 (J)
d0olctl39 0 10 SEQ
new black, #14
1 10 SEQ PC04-0A new black, #15
2 10 SEQ PC04-0B new black, #16
3 3 North SEQC new black, #13
4 18 IB NE-0 new black #24 (E)
5 18 IB NE-1 new black, #25 (F)
HV monitoring :
SMT uses 8 6U VME crates of HV power
supplies. Each crate holds up to 6 modules
(motherboards) and each module has 8 power supplies (pods). There are 4 crates for the South
(smts1, smts2, smts3, smts4) and 4 crates for teh North (smtn5, smtn6, smtn7, smtn8).
this is the mapping of PPCs for the HV crates :
smts2 - d0olctl48
smts3 - d0olctl49
smts4 - d0olctl25
smtn5 - d0olctl23
smtn6 - d0olctl52
smtn7 - d0olctl51
smtn8 - d0olctl53
> cd /online/config/smt/hv
if you want to deal with a single HV
crate, use for ex.
> hv_crate.py smts1.hvc
change accordingly if you want to use
another crate. All incarnations of the HV GUIs
have similar functionality :
left click on the control box gives a menu with possible actions. Bias voltage can be input by left clicking the V_set window.
check HV necessary for a certain configuration of HDI's :
- download your favorite configuration
- in /home/d0smt/monitoring
will check the HV pods necessary for your configuration and turn them ON
if it finds them OFF. It will not touch them if they have astatus different from OFF.
b) it will list for each IB the list of HV pods needed, their status and potential anomalies.
c) it will create a file on-config.hvc in /home/d0smt/hv corresponding to the OFF pods
found in a). If you type
>hv_crate.py on_config.hvc &
then globally ramp up those pods.
d) it creates also 2 configuration files
- curr_config.hvc : with pods close to their current trip point
- volt_config.hvc : with pods not set at full bias
Summary HV page :
use smtn.hvd for the North, smts.hvd
for the South. Clicking on boxes with crate names bring up detailed GUIs
for those crates.
You need to connect to the variables
you want to plot by entering their EPICS name and clicking on the 'connect'
in the "Controls" window. To give you an idea of what these variables look like, you can look in the files smt*.cfg in
the directory /projects/archive/current. These are all the SMT EPICS variables which are archived by the Archiver. In general,
for the HV, the variable name starts with SMT_HV_* like e.g. SMT_HVC_122P/CURR to look at the bias current for pod 122P,
and SMT_HVC_122P/VOLT for its voltage. For variables related to LV and temperature for the HDIs, since they are monitored
by the Interface Boards, their names start with SMT_IB_* like e.g. SMT_IB_SE0A02-C/TEMP for the temperature of the HDI
connected to IB in SE crate 0 half A slot 2 channel C. Other available variables for that channel are /V_AVDD, /V_AVDD2,
/V_DVDD, /I_AVDD, /I_AVDD2, /I_DVDD for the voltages and currents of the 3 power supplies necessary to bias the HDI.
In StripTool you can change displayed parameters or choose various display options by right clicking and going to Controls. Also you can prepare and save (or load) .cst files with your favorite configurations of parameters.
tool to look at EPICS archives :
<achive_file> is a name like dir.*
After that you should see a window. Click on the field under "Name
choose the name (can use a wildcard or a regular expression to see available variables) . Or simply press 'Enter' and
browse the complete list of variables in your archive.
to bring up the download GUI:
This pops up the main menu shown below:
to quit : push 'quit' button.
If you are sure that everything is correctly enabled (at present normally it is the case) you can click on the `download' button of the crate
that you want to read out in the main window, for ex. `VRBCR_69'.
The download button should turn green after ~15 s (the first time `download’, `reinit VME’ or `details’ is used for a given crate it can take ~1min). If not, the download failed.
Go to trouble shooting.
If you are not sure that everything is correctly enabled push `details' button of your favorite crate. That will give you the following window:
grey: condition unknown
; dark grey : condiion unknown, HV on
yellow : disabled ; orange : disabled, HV on
green : download OK ; sea green : download OK, HV on
sea green (for VRB) : download OK, simulated data selected
red : ERROR status
each box in this window is an
electronics component (going from left to right : VRB - Sequencer - Interface
Board - HDI,
also there are VRBC, SEQController and VBD on the top). VRBCR box performs actions on all VRBs in the crate.
Generally, left click on a box will open a menu for hardware actions (like for Sequencer SEQ_200A06 above),
right click on a box will open a menu for DataBase modifications. After the menu is open: left click selects an option,
right click removes the menu.
Many useful GUIs can be activated from the GUI above :
To power up your favorite HDIs :
Be aware of different types of Downloads and PowerOffs:
`cal inject' button will pop up a window with the following graph :
Use the scroll bar in the lower part
of the window to check the complete data output.
If the result does not look like this, try to take a couple of more cal inject events (hit the `cal' button or the `cal inj' button again). If that doesn't help something is wrong. Consult the database if this is a known feature. You may also try to troubleshoot or call expert.
Make sure that the
download is done (see Download section). After that start Taker on an online
(keep in mind that d0ol05 and d0ol14 belong to SMT so it's better to use those) :
other useful tools to have are coormon and daq_monitor :
the Taker window. Select the menu item `modify/change trigger'.
Select a trigger configuration file e.g. zero-bias-smt-0x61-sr-1.0 from the commissioning/tracking sub-directory
/online/data/coor/configurations/commissioning/tracking. "Zero bias" configurations use the
Bunch_Crossing clock to fire i.e. they know about the Abort Gaps and should provide good
start button. Normally 0x61 should then ship data to L3.
If there are other problems look at the Trouble Shooting Section.
if you want to change the readout
configuration without turning on or off HDIs :
- enable VRB and VRB channels you want to read out
- disable VRB and VRB channels you do not want to read out
- do a 'reinit VRB'. The VRBs you disabled might appear in red. You need 'J3 disable' them for them to turn yellow.
- do a SEQC 'use SCL', 'reset'
VRBCR 'VRB reset'
Remember : for this to work, the VRB channels you have enabled must be connected to HDIs which have been previously
powered up and downloaded.
Checking L3 information
Go to http://d0l3dev.fnal.gov/l3_auto_start/farm_config/config.asp
, choose "Default" as configuration and click on
"Show log files". First, look at the ETG log file. Click on "cmd.exe" for d0l3etg.fnal.gov. You'll get something like this :
2001/08/26 16:42:36:380 NetETG=> Got event #: 11690...Triggers: 0 0 0 2
2001/08/26 16:42:36:411 SBR=> Began Event # 11690 in 246 usec
2001/08/26 16:42:38:692 SBR=> Ended Event # 11690 for all streams after 31ms.
2001/08/26 16:42:38:692 All waited-on crates read for some future event. Event 11694 incomplete!
2001/08/26 16:42:40:911 Stream 0 was missing crates: x33 x34
2001/08/26 16:42:40:911 NetETG=> Got event #: 11700...Triggers: 0 0 0 2
2001/08/26 16:42:40:911 SBR=> Began Event # 11700 in 138 usec
2001/08/26 16:42:51:911 SBR=> Ended Event # 11700 for all streams after 0ms.
2001/08/26 16:42:51:911 NetETG=> Got event #: 11710...Triggers: 0 0 0 2
2001/08/26 16:42:51:911 SBR=> Began Event # 11710 in 137 usec
2001/08/26 16:43:02:849 SBR=> Ended Event # 11710 for all streams after 0ms.
look for missing SMT crates. If a crate is missing that normally indicates some problem which should be
debugged and fixed.
To look at the MCH2 VRC log file, click on "cmd.exe" for d0l3vrc2a.fnal.gov. You'll get something like this :
2001/08/26 22:30:24:988 Read in 228 usec...Event 10542 - Crate 0x69 - Size 9152
2001/08/26 22:30:25:008 Read in 261 usec...Event 10542 - Crate 0x6a - Size 9784
2001/08/26 22:30:25:018 Read in 351 usec...Event 10542 - Crate 0x68 - Size 15048
2001/08/26 22:30:25:018 Read in 341 usec...Event 10542 - Crate 0x66 - Size 14808
2001/08/26 22:30:25:018 Read in 325 usec...Event 10542 - Crate 0x64 - Size 13936
2001/08/26 22:30:25:038 Read in 250 usec...Event 10542 - Crate 0x63 - Size 10544
2001/08/26 22:30:25:038 Read in 345 usec...Event 10542 - Crate 0x60 - Size 14904
2001/08/26 22:30:25:038 Read in 365 usec...Event 10542 - Crate 0x62 - Size 15840
2001/08/26 22:30:25:038 Read in 387 usec...Event 10542 - Crate 0x51 - Size 16964
2001/08/26 22:30:25:038 Read in 500 usec...Event 10542 - Crate 0x6b - Size 22328
2001/08/26 22:30:25:038 Read in 1368 usec...Event 10542 - Crate 0x67 - Size 16008
2001/08/26 22:30:25:068 Read in 362 usec...Event 10542 - Crate 0x65 - Size 15704
2001/08/26 22:30:26:030 Read in 631 usec...Event 10543 - Crate 0x61 - Size 28336
You should see here all crates you are trying to read out. The size is given in bytes. For a full readout crate
the size should not vary and should be between 100 and 120 Kbytes. For sparse readout (as above) the size should vary on an event by event
basis and should be less than 50 Kbytes.
new instructions how to run SDAQ are here
during SDAQ running it is useful sometime to look at the FE CPU (=PowerPC). The mapping is here and also below:
crate name IOC port
0x61 smt0_1 d0olsmt08 30
0x62 smt1_0 d0olsmt13 21
0x63 smt1_1 d0olsmt07 7
0x64 smt2_0 d0olsmt00 28
0x65 smt2_1 d0olsmt06 27
0x66 smt3_0 d0olsmt09 26
0x67 smt3_1 d0olsmt05 6
0x68 smt4_0 d0olsmt02 10
0x69 smt4_1 d0olsmt03 25
0x6A smt5_0 d0olsmt14 31
0x6B smt5_1 d0olsmt04 24
In order to look at an individual crate during an SDAQ run do the following:
1. telnet t-d0-mch2
2. login as ioc, password 0……0
3. connect <port>
below is obsolete.
Make sure that the download (Step 1.) is done.
Go to the Taker window. Select the menu item `modify/change trigger'.
Select the file calib -smt-0x63-1.0 from the tracking sub-directory.
Hit the start button:
Check the progress in the Calib_Manager_GUI window:
One calibration cycle should take about 10 minutes. If the run is not finished in this time:
Start the SDAQ Monitoring
setup onl_smtcalib devel
The following window pops up:
Choose the correct crate, `VRBCR-63' , the VRB that you want to monitor (`VRB-6301'), an HDI in that channel (`SVXE-6903_C'), or a chip of that HDI (`SVXE-6903_C_00). You can monitor complete Crates, VRBs, HDIs, or single chips. Note that some choices are disabled. If the calibration run is already over, click on `continuous mode' to get to `single mode'. While the calibration is ongoing, choose a different update time. The resulting histogram will be updated every n minutes. After you made your selections, click on the `Start!' button.
In the Histoscope selection window you get the following choices:
Select what you want to look at: `Pedestal@/VRBCR-63.....':
Run SMT Event Display
please look at
Gavin's page how to run Event Display
you need to be on d0ol05 (d0ol14 does not have enough memory).
there is a configuration file you may need
to change to have things working : cfg.dat where you
describe the VRB hardware configuration which is being read out. To run just type :
The display shows 6 barrels and 6 associated F-disks
in 6 windows. Before you start the display position
the windows as you like and never touch them again (otherwise they tend to display the same event really
many times!). P-side hits are shown in red with the height proportional to the hit amplitude. Pedestals are
not subtracted. N-side hits are shown in blue.
please look at Alex' page how to run
you need to be on d0ol05 (d0ol14 does not have enough memory).
there are 2 configuration files you may
need to change to have things working : cfg.dat where you
describe the VRB hardware configuration which is being read out and smt_examine.rcp where you
describe how you want examine to run (see details on the above link). Use :
>run_online.sh if you don't want any cluster information
in histograms. Otherwise use :
there is also a summary GUI based on SMTExamine to output occupancies and average amplitudes in all SMT detectors. To run the version which shows VRB crates :
>examine_gui.py crates &
for the version which shows SMT geometry
>examine_gui.py detectors &
SMT uses 12 VRB crates in MCH2, named after their Geographic Sectors GS:0x60 through 0x6B. The Geographic Sector is marked above the crate and at the side. It contains from left to right: the front end processor (Power PC, PPC), VBD, VBD repeater, VRBs, VRBC, see the photo below:
1. The download button turns red.
2. The download button turns green, but the cal inject fails.
3. You get loads of error messages and nothing happens
Make sure that the front end processors are switched on:
The crate that is
downloaded must be switched on. Check CPU lights on the VRBs and power light on
the VBD and VRBC. Probably reset the front end.
No events coming to L3
Issue a `SCL init' from Coormon
Check the Front End Busy status on the DAQ Monitor display (start_daq_monitor_display, choose L1, Geo Sector). Check whether a GS associated to an SMT crate sends too many FEBusy. If not, bug someone else. In the unlikely case it is SMT:
Redownload and make sure that cal inject shows the correct pattern. Corrupted data (missing 0xc0c0 end marker) can cause the problem.
Was the trigger configured correctly? Free and reload the trigger configuration from the taker window.
Sometimes reinitializing the L1 Trigger Frameword (L1TFW) helps (quite rare). Before doing that stop all your and other's runs!
If there are no events coming to the VBD (ETG shows incomplete events for SMT crate, the VRBC is getting no triggers: the 2 blue lights on the VRBC front panel are not lid), initialize the VRBC. Click on the `details' button of the main SMT GUI window for the right crate and left click on the VRBC (top row). Click on the `init' field.
If the VRBC is getting triggers (both blue lights are flashing on the VRBC front panel), but the event number on the VBD front panel is not increasing:
Power cycle the VRB crate
The VBD shows that it receives events, the event number is increasing: This is almost certainly a L3 problem. Look into trouble shooting there (reset from WWW, reset button, reb....)
SDAQ calibration starts, but never ends
This is usually due to a problem on the front
end. To get the error message log on to the correct frontend (telnet
d0olsmt07, login: ioc,
Abort the ongoing run and try again. You will get some information for the new run from the frontend.
keep records of ANYTHING you do with SMT in the Control room Logbook
to start :
go to 'Detector Shift', 'SMT', 'SMT log' to
open a window. Drag 'Text' icon into the window to write a message.
It is a good idea also to login so the entries will be stamped with your name.
To capture a screen (e.g. a window with a time plot by
StripTool) and insert it into the logbook :
on Xtns/Screen shot
click on Grab
click on the window which you want to save, after a few seconds the captured image will pop up in a window.
right click on the image, go to 'File', 'Save as'
go to /home/d0smt/shifts/2001/<month>/<day>
type any descriptive filename.jpg, save the file
go to logbook, click 'Insert image', insert image from a file
Browse for the file, click /home/*, open the saved file