DØ Run II SMT Shift Info
last modified by Andrei Nomerotski August 26 2001
Contents:
Goals
Power Up and Down
Procedures
Download
Monitoring
Take Data to L3
Take a SDAQ Calibration Run
Start the SDAQ Monitoring
Run SMT Examine
Trouble Shooting
Logbook
Online version: http://d0server1.fnal.gov/projects/Silicon/www/ShiftInfo.html
other links:
SMT shifter is responsible for SAFE and stable operation of SMT. There are several important aspects of safe operation :
- Low Voltages (LV) applied to
electronics
- High Voltage (HV) to bias detectors
- Radiation environment at the Tevatron
Stable operation includes also
functioning Data Aquisition (DAQ) both for data taking to Level3
(PrimaryDAQ=PDAQ) and for calibrations
(SecondaryDAQ=SDAQ).
Normally, the SMT display in the D0
Control Room will have LV and HV summary GUIs open as shown below
(if not, see the monitoring section how to open them). Generally, any RED spots in those GUI will require your
action.
LV GUI shows 8 SMT Interface
crates (SE0, SE1, SW0, SW1, NE0, NE1, NW0, NW1) with 18 Interface Boards
in each crate (ex.SE0A02) and
associated Power Supply (ex.SMT_LV_SE0). Each IB has 8 channels shown as boxes
in the GUI. Color coding :
Green
= LV is ON, no error status
Orange
= LV is OFF
blinking Red/Black = LV is TRIPPED, error status
Grey
= IB channel not connected
pushing PS (ex.SMT_LV_SE0) or IB (ex.SE0A02) buttons will open detailed GUIs for these objects.

HV GUI : (here for South only)
Green
= Ramped up, OK status
Orange = switched ON, not ramped up
Yellow = ramping
Red = tripped
Blue = switched OFF
Though this GUI has HV controls in the top line we recommend to use more detailed GUI to operate HV. It can be activated by pushing HV PS buttons (ex. smts1) and is explained in the monitoring section.

SMT
radiation environment is monitored by SMT Loss Monitor and is available via
ACNET on page E35.
If
you don't know how to access this information - ask Shift Captain.
Power Up and Down Procedures
To power up SMT from scratch you will need to switch on 3 sets of crates :
1. VRB crates in Moving Counting House 2nd floor (MCH2)
2. Sequencer crates at the Platform
3. Interface crates in the Cathedral
the last two are inside the Collision Hall and are operated remotely via GUIs.
login as d0smt on the online cluster
>setup
d0online
>cd
/home/d0smt/monitoring
>seq_gui.py &

where the SMT supplies are
SMT_LVSEQ_S/1 - for sequencer crate
PC20-0
SMT_LVSEQ_S/2 - for sequencer crate PC20-1
SMT_LVSEQ_S/3 - for sequencer crate PC19-0
SMT_LVSEQ_S/4 - for sequencer crate PC19-1
ç used by the CFT
SMT_LVSEQ_N/1 - for sequencer crate PC
SMT_LVSEQ_N/2 - for sequencer crate PC
SMT_LVSEQ_N/3 - for sequencer crate PC
SMT_LVSEQ_N/4 - for sequencer crate
PC ç used by
the CFT
to run summary GUI for PS :
>setup d0online
>cd /home/d0smt/monitoring
>./IBLVsmall.py

pushing the PS buttons opens individual PS GUI.
to run separate PS GUI :
>cd
/online/config/smt/SMTLV
>setup d0online
>./ibps.py SMT_LV_SE1
that brings up the GUI for the crate SE-1 (use SMT_LV_SE0 for SE-0 etc.)

Buttons On/Off switch the supplies on and
off. Note: do not leave those displays running for a long time -
most likely they have a memory leak and after some time (~1 hour) tend to use
all resourses of the computer.
To quit : Go to 'File' and 'Quit'
to power SMT off
to run the GUI accessing the Rack Monitor Interfaces :
>setup d0online
>cd /online/config/ctl
>./pfm.rmi &
The monitoring of HDI voltages and currents:
to run global display for IB channels and LVPS status :
>setup d0online
>cd /home/d0smt/monitoring
>source setup
> ./IBidvdd.py & to see all IB
channels and their DVDD current consumption, see below

> ./IBtemps.py & to see all IB channels and the HDI temperatures
to run IB currents display separately use :
>setup d0online
>cd /online/config/smt/SMTIB
>./SMTIB_wide.py SE0A07.inp
(SE0A07 does it for the IB crate SE-0A slot 7, use
other names as needed)
The following window should pop up:

Left click on the Status box shows the status.
to monitor temperatures in the IB crates :
>cd /home/d0smt/monitoring
>setup d0online
>./IBdegC.py

each IB crate has 6 temperature
sensors. The temperature is shown in deg C. If a temperature is too high,
it turns red. If 4 temperatures in a crate are higher 35 deg C, this crate is
automatically turned off.
a useful tool to diagnoze 1553
(MIL1553 is a Slow Control Protocol for Sequencer and Interface crates)
problems is Mscan. For a given
channel of 1553 controller it outputs a map of responses of all possible
addresses of all possible Remote Terminals (RTs).
Normally a good response without errors is green. To run :
>setup onl_apps
>Mscan.py d0olctl40 4 ! this particular
channel is used for IB crate SW-0
this is the current SMT 1553 configuration where you can find the 1553 mapping of all seq and IB crates .
SMT uses 4 PPC mv2300/mv2304 in two crates in M300 MCH3
quadrant CPU channel devices cable type
SW
d0olctl57 0 10 SEQ
PC19-0A
new black, #1 ç this used to be d0olctl40
1 10 SEQ
PC19-0B
new black, #2
2 10 SEQ
PC20-1B
new black, #8
3 3 South SEQC, 4 West
RM new black, #9
4 18 IB
SW-0
new black, #17 (G)
5 18 IB
SW-1
new black, #18 (H)
SE
d0olctl56 0 10 SEQ
PC20-0A
new black, #5 ç this used to be d0olctl37
1 10 SEQ
PC20-0B
new black, #6
2 10 SEQ
PC20-1A
new black, #7
3 4 East
RM
new black, # 19 (A)
4 18 IB
SE-0
new black, #20 (C)
5 18 IB
SE-1
new black, #21 (D)
NW
d0olctl50 0 10 SEQ
PC03-0A
new black, #10
1 10 SEQ
PC03-0B
new black, #11
2 10 SEQ
PC03-1A
new black, #12
3
--empty --
4 18 IB
NW-0
new black, #22 (I)
5 18 IB
NW-1
new black, #23 (J)
NE
d0olctl39 0 10 SEQ
PC03-1B
new black, #14
1 10 SEQ
PC04-0A
new black, #15
2 10 SEQ
PC04-0B
new black, #16
3 3 North SEQC
new black, #13
4 18 IB
NE-0
new black #24 (E)
5 18 IB
NE-1
new black, #25 (F)
HV monitoring :
SMT uses 8 6U VME crates of HV power
supplies. Each crate holds up to 6 modules
(motherboards) and each module has 8 power supplies (pods). There are 4
crates for the South
(smts1, smts2, smts3, smts4) and 4 crates for teh North (smtn5, smtn6, smtn7,
smtn8).
this is the mapping of PPCs for the HV crates :
smts1 -
d0olctl08
smts2 - d0olctl48
smts3 - d0olctl49
smts4 - d0olctl25
smtn5 - d0olctl23
smtn6 - d0olctl52
smtn7 - d0olctl51
smtn8 - d0olctl53
>
setup d0online
> cd /online/config/smt/hv
if you want to deal with a single HV
crate, use for ex.
> hv_crate.py smts1.hvc
change accordingly if you want to use
another crate. All incarnations of the HV GUIs
have similar functionality :

left click on the control box gives a menu with possible actions. Bias voltage can be input by left clicking the V_set window.
to
check HV necessary for a certain configuration of HDI's :
-
download your favorite configuration
-
in /home/d0smt/monitoring
>source setup
>./fullTestHV.py
a) this
will check the HV pods necessary for your configuration and turn them ON
if
it finds them OFF. It will not touch them if they have astatus different from
OFF.
b)
it will list for each IB the list of HV pods needed, their status and potential
anomalies.
c)
it will create a file on-config.hvc in /home/d0smt/hv corresponding to the OFF
pods
found
in a). If you type
>hv_crate.py on_config.hvc &
you can
then globally ramp up those pods.
d)
it creates also 2 configuration files
- curr_config.hvc : with pods close to their current
trip point
- volt_config.hvc : with pods not set at full bias
Summary HV page :
>set
d0online
>cd /online/config/smt/hv
>./hv_det_mod.py smtAll.hvd
use smtn.hvd for the North, smts.hvd
for the South. Clicking on boxes with crate names bring up detailed GUIs
for those crates.
Time plots
>setup d0online
>StripTool &
You need to connect to the variables
you want to plot by entering their EPICS name and clicking on the 'connect'
button
in the "Controls" window. To give you an idea of what these variables
look like, you can look in the files smt*.cfg in
the directory /projects/archive/current. These are all the SMT EPICS variables
which are archived by the Archiver. In general,
for the HV, the variable name starts with SMT_HV_* like e.g. SMT_HVC_122P/CURR
to look at the bias current for pod 122P,
and SMT_HVC_122P/VOLT for its voltage. For variables related to LV and
temperature for the HDIs, since they are monitored
by the Interface Boards, their names start with SMT_IB_* like e.g.
SMT_IB_SE0A02-C/TEMP for the temperature of the HDI
connected to IB in SE crate 0 half A slot 2 channel C. Other available
variables for that channel are /V_AVDD, /V_AVDD2,
/V_DVDD, /I_AVDD, /I_AVDD2, /I_DVDD for the voltages and currents of the 3
power supplies necessary to bias the HDI.

In StripTool you can change displayed parameters or choose various display options by right clicking and going to Controls. Also you can prepare and save (or load) .cst files with your favorite configurations of parameters.
Another
tool to look at EPICS archives :
>cd /projects/archive/smt/current
>setup chan_archiver
>Xarr.py <archive_file>
<achive_file> is a name like dir.*
After that you should see a window. Click on the field under "Name
Pattern" and
choose the name (can use a wildcard or a regular expression
to see available variables) . Or simply press 'Enter' and
browse the complete list of variables in your archive.
to bring up the download GUI:
>cd onl_smtnew/util
>source smt-setup
>./smt_gui.py
This pops up the main menu shown below:

to quit : push 'quit' button.
If you
are sure that everything is correctly enabled (at present normally it is the
case) you can click on the `download' button of the crate
that
you want to read out in the main window, for ex. `VRBCR_69'.
The
download button should turn green after ~15 s (the first time `download’,
`reinit VME’ or `details’ is used for a given crate it can take ~1min). If not,
the download failed.
Go
to trouble shooting.
If you are not sure that everything
is correctly enabled push `details' button of your favorite crate. That
will give you the following window:

Color Codes:
grey: condition unknown
; dark grey : condiion unknown, HV on
yellow : disabled ; orange : disabled, HV on
green : download OK ; sea green : download OK, HV on
sea green (for VRB) : download OK, simulated data selected
red : ERROR status
each box in this window is an
electronics component (going from left to right : VRB - Sequencer - Interface
Board - HDI,
also there are VRBC, SEQController and VBD on the top). VRBCR box performs
actions on all VRBs in the crate.
Generally, left click on a box will open a menu for hardware actions (like for
Sequencer SEQ_200A06 above),
right click on a box will open a menu for DataBase modifications. After the
menu is open: left click selects an option,
right click removes the menu.
Many useful GUIs can be activated from the GUI above :
To power up your favorite HDIs :
Be aware of different types of
Downloads and PowerOffs:
`cal inject' button will pop up a window with the following graph :

Use the scroll bar in the lower part
of the window to check the complete data output.
If the result does not look like this, try to take a couple of more cal
inject events (hit the `cal' button or the `cal inj' button again). If
that doesn't help something is wrong. Consult the database if this is a known
feature. You may also try to troubleshoot or call expert.
Make sure that the
download is done (see Download section). After that start Taker on an online
computer
(keep in mind that d0ol05 and d0ol14 belong to SMT so it's better to use those)
:
>setup d0online
>taker
other useful tools to have are coormon and daq_monitor :
>coormon
>start_daq monitor
Go to
the Taker window. Select the menu item `modify/change trigger'.
Select
a trigger configuration file e.g. zero-bias-smt-0x61-sr-1.0 from the commissioning/tracking
sub-directory
/online/data/coor/configurations/commissioning/tracking.
"Zero bias" configurations use the
Bunch_Crossing
clock to fire i.e. they know about the Abort Gaps and should provide good
SVX
pedestals.
Hit the
start button. Normally 0x61 should then ship data to L3.
If
there are other problems look at the Trouble Shooting Section.
if you want to change the readout
configuration without turning on or off HDIs :
- enable VRB and VRB channels you want to read out
- disable VRB and VRB channels you do not want to read out
- do a 'reinit VRB'. The VRBs you disabled might appear in red. You need 'J3
disable' them for them to turn yellow.
- do a SEQC 'use SCL', 'reset'
VRBC 'init'
VRBCR 'VRB reset'
Remember : for this to work, the VRB channels you have enabled must be
connected to HDIs which have been previously
powered up and downloaded.
Checking L3 information
Go to http://d0l3dev.fnal.gov/l3_auto_start/farm_config/config.asp
, choose "Default" as configuration and click on
"Show log files". First, look at the ETG log file. Click on
"cmd.exe" for d0l3etg.fnal.gov. You'll get something like this :
2001/08/26 16:42:36:380 NetETG=> Got event #: 11690...Triggers: 0 0 0 22001/08/26 16:42:36:411 SBR=> Began Event # 11690 in 246 usec2001/08/26 16:42:38:692 SBR=> Ended Event # 11690 for all streams after 31ms.2001/08/26 16:42:38:692 All waited-on crates read for some future event. Event 11694 incomplete!2001/08/26 16:42:40:911 Stream 0 was missing crates: x33 x342001/08/26 16:42:40:911 2001/08/26 16:42:40:911 NetETG=> Got event #: 11700...Triggers: 0 0 0 22001/08/26 16:42:40:911 SBR=> Began Event # 11700 in 138 usec2001/08/26 16:42:51:911 SBR=> Ended Event # 11700 for all streams after 0ms.2001/08/26 16:42:51:911 2001/08/26 16:42:51:911 NetETG=> Got event #: 11710...Triggers: 0 0 0 22001/08/26 16:42:51:911 SBR=> Began Event # 11710 in 137 usec2001/08/26 16:43:02:849 SBR=> Ended Event # 11710 for all streams after 0ms.
look for missing SMT crates. If a crate is missing that normally indicates some problem which should be
debugged and fixed.
To look at the MCH2 VRC log file, click on "cmd.exe" for d0l3vrc2a.fnal.gov. You'll get something like this :2001/08/26 22:30:24:988 Read in 228 usec...Event 10542 - Crate 0x69 - Size 91522001/08/26 22:30:25:008 Read in 261 usec...Event 10542 - Crate 0x6a - Size 97842001/08/26 22:30:25:018 Read in 351 usec...Event 10542 - Crate 0x68 - Size 150482001/08/26 22:30:25:018 Read in 341 usec...Event 10542 - Crate 0x66 - Size 148082001/08/26 22:30:25:018 Read in 325 usec...Event 10542 - Crate 0x64 - Size 139362001/08/26 22:30:25:038 Read in 250 usec...Event 10542 - Crate 0x63 - Size 105442001/08/26 22:30:25:038 Read in 345 usec...Event 10542 - Crate 0x60 - Size 149042001/08/26 22:30:25:038 Read in 365 usec...Event 10542 - Crate 0x62 - Size 158402001/08/26 22:30:25:038 Read in 387 usec...Event 10542 - Crate 0x51 - Size 169642001/08/26 22:30:25:038 Read in 500 usec...Event 10542 - Crate 0x6b - Size 223282001/08/26 22:30:25:038 Read in 1368 usec...Event 10542 - Crate 0x67 - Size 160082001/08/26 22:30:25:068 Read in 362 usec...Event 10542 - Crate 0x65 - Size 157042001/08/26 22:30:26:030 Read in 631 usec...Event 10543 - Crate 0x61 - Size 28336
|
|
You should see here all crates you are trying to read out. The size is given in bytes. For a full readout crate
the size should not vary and should be between 100 and 120 Kbytes. For sparse readout (as above) the size should vary on an event by event
basis and should be less than 50 Kbytes.
new instructions how to run SDAQ are here
during SDAQ running it is useful sometime to look at the FE CPU (=PowerPC). The mapping is here and also below:
IOC mapping
crate name IOC port
0x60
smt0_0
d0olsmt11 8
0x61
smt0_1
d0olsmt08 30
0x62
smt1_0
d0olsmt13 21
0x63
smt1_1
d0olsmt07 7
0x64
smt2_0 d0olsmt00
28
0x65
smt2_1
d0olsmt06 27
0x66
smt3_0
d0olsmt09 26
0x67
smt3_1
d0olsmt05 6
0x68
smt4_0
d0olsmt02 10
0x69
smt4_1
d0olsmt03 25
0x6A smt5_0
d0olsmt14 31
0x6B smt5_1
d0olsmt04 24
In order to look at an individual crate during an SDAQ run do the following:
1. telnet t-d0-mch2
2. login as ioc, password 0……0
3. connect <port>
the information
below is obsolete.
Make
sure that the download (Step 1.) is done.
Go
to the Taker window. Select the menu item `modify/change trigger'.
Select
the file calib -smt-0x63-1.0
from the tracking sub-directory.
Hit
the start button:
Check the progress in the Calib_Manager_GUI window:
One calibration cycle should take about 10 minutes. If the run is not finished in this time:
Start the SDAQ Monitoring
setup onl_smtcalib devel
ssdaq
account: smt_calib@d0onprd1
password:
The following window pops up:

Choose the correct crate, `VRBCR-63' , the VRB that you want to monitor (`VRB-6301'), an HDI in that channel (`SVXE-6903_C'), or a chip of that HDI (`SVXE-6903_C_00). You can monitor complete Crates, VRBs, HDIs, or single chips. Note that some choices are disabled. If the calibration run is already over, click on `continuous mode' to get to `single mode'. While the calibration is ongoing, choose a different update time. The resulting histogram will be updated every n minutes. After you made your selections, click on the `Start!' button.
In the Histoscope selection window you get the following choices:

Select what you want to look at: `Pedestal@/VRBCR-63.....':

Run SMT Event Display
please look at
Gavin's page how to run Event Display
you need to be on d0ol05 (d0ol14 does not have enough memory).
>setup d0online
>cd /online/examines/t01.51.00/smt_event_display/bin/Linux2-KCC_3_4
there is a configuration file you may need
to change to have things working : cfg.dat where you
describe the VRB hardware configuration
which is being read out. To run just type :
>rund0ve.sh
The display shows 6 barrels and 6 associated F-disks
in 6 windows. Before you start the display position
the windows as you like and never touch them
again (otherwise they tend to display the same event really
many times!). P-side hits are shown in red with
the height proportional to the hit amplitude. Pedestals are
not subtracted. N-side hits are shown in blue.

please look at Alex' page how to run
Examine
you need to be on d0ol05 (d0ol14 does not have enough memory).
>setup d0online
>cd /online/examines/t01.xx.00/smt_examine/bin/Linux2_KCC_3_4
there are 2 configuration files you may
need to change to have things working : cfg.dat where you
describe the VRB hardware
configuration which is being read out and smt_examine.rcp where you
describe how you want examine to run
(see details on the above link). Use :
>run_online.sh if you don't want any cluster information
in histograms. Otherwise use :
>run_clusters_online.sh
there is also a summary GUI based on SMTExamine to output occupancies and average amplitudes in all SMT detectors. To run the version which shows VRB crates :
>examine_gui.py crates &
for the version which shows SMT geometry
use :
>examine_gui.py detectors &
SMT uses 12 VRB crates in MCH2, named after their Geographic Sectors GS:0x60 through 0x6B. The Geographic Sector is marked above the crate and at the side. It contains from left to right: the front end processor (Power PC, PPC), VBD, VBD repeater, VRBs, VRBC, see the photo below:

Download failed
1. The download button turns red.
2. The download button turns green, but the cal inject fails.
3. You get loads of error messages and nothing happens
Make sure that the
front end processors are switched on:
The crate that is
downloaded must be switched on. Check CPU lights on the VRBs and power light on
the VBD and VRBC. Probably reset the front end.
No events coming to L3
Issue a `SCL
init' from Coormon
Check the Front
End Busy status on the DAQ Monitor display (start_daq_monitor_display, choose
L1, Geo Sector). Check whether a GS associated to an SMT crate sends too many
FEBusy. If not, bug someone else. In the unlikely case it is SMT:
Redownload and
make sure that cal inject shows the correct pattern. Corrupted data (missing
0xc0c0 end marker) can cause the problem.
Was the trigger
configured correctly? Free and reload the trigger configuration from the taker
window.
Sometimes
reinitializing the L1 Trigger Frameword (L1TFW) helps (quite rare).
Before doing that stop all your and other's runs!
If there are no
events coming to the VBD (ETG shows incomplete events for SMT crate, the
VRBC is getting no triggers: the 2 blue lights on the VRBC front panel are not
lid), initialize the VRBC. Click on the `details' button of the main SMT GUI
window for the right crate and left click on the VRBC (top row). Click on
the `init' field.
If the VRBC is
getting triggers (both blue lights are flashing on the VRBC front panel), but
the event number on the VBD front panel is not increasing:
Redownload
Power cycle the
VRB crate
The VBD shows that it receives events, the event number is increasing: This is almost certainly a L3 problem. Look into trouble shooting there (reset from WWW, reset button, reb....)
SDAQ calibration starts, but never ends
This is usually due to a problem on the front
end. To get the error message log on to the correct frontend (telnet
d0olsmt07, login: ioc,
password:
).
Abort the ongoing run and try again. You
will get some information for the new run from the frontend.
keep records of ANYTHING you do with SMT in the
Control room Logbook
to start :
>setup d0online
>start_daq logbook
go to 'Detector Shift', 'SMT', 'SMT log' to
open a window. Drag 'Text' icon into the window to write a message.
It is a good idea also to login so the entries
will be stamped with your name.
To capture a screen (e.g. a window with a time plot by
StripTool) and insert it into the logbook :
>gimp &
click
on Xtns/Screen shot
click on Grab
click on the window which you want to save, after a few seconds the captured
image will pop up in a window.
right click on the image, go to 'File', 'Save as'
go to /home/d0smt/shifts/2001/<month>/<day>
type any descriptive filename.jpg, save the file
go to logbook, click 'Insert image', insert image from a file
Browse for the file, click /home/*, open the saved file