INTENT:

This is intended as a RFC (Request For Comment) on how we are going to make the VBD (Dave Cutts and Ray Zeller) and VRB (Mark Bowden) and the SARC (Mark Baert) all work together.

PRESENT SCHEME:

Initially the VRB addressing was set up in A24 and A32 mode (no A16 mode at present to save space in the FPGA’s and keep things lean and mean) very similar to the old FADC cards with each card taking up 2k of space with a repetitive memory pattern including a byte count and FIFO for the data and various control registers. (I think Mark Bowden at one point had this as a 32 bit longword count so that is not a problem for him. He just needs to know what this shoult be. I am not sure of the impact of this to the SARC and the rest of the crate electronics.) Each card would take up 2K of space starting where the previous card left off. This is set by a combination of DIP switches and by pins on the backplane that add to the base address according to the slot it’s plugged into. There’s physical room for 16 VRB cards in the crate but probably only 12 will be used at most for various other reasons beyond the scope of interest here.

Looking at the VBD pretty much only from the software side which processes a list of 32 bit addresses for the word count and FIFO’s Mark Baert and I naively assumed that the VBD was 32 bit capable through out and that the 2K assigned to each VRB was unnecessarily restrictive and could be opened up to say 64K bytes per board. Mark Baert and I mistakenly told Mark Bowden this and the dialogue begins with M.Bowden asking if he can go ahead and implement these changes.

WE ENTER THE DIALOGUE:

This first letter is from Mark Bowden who is asking me about how he should set up his addressing. The problem is that basically I don’t know what to tell him because I don’t really understand how the VBD does work now much less how it will work in the future.

At 09:09 AM 5/9/97 -0500, you (Mark Bowden signified by ">") wrote:
>Reply to: RE>VRB addresses
>
>
>Bob,
>
>I will leave the VRB addresses where they are for now. I was just asking
>whether the 2 KByte address limit still applied.
>
>-Mark
>
>
Hello Mark,
Leaving the address where they are for now is a good idea.
I did not realize how limited the VBD really is and probably
had you change your addressing in a basically (wrong) headed
way...
Basically I'm trying to find out how the vBD WILL work...
I just found out that "drastic" changes are in the works for Run II.
I guess for NWA it does not matter as Mark Baert and Mike Utes
say we will not be using a VBD and we will be using a
68k processor of some kind there. So I guess there are two
requirements: immediate and long term. So for the short
term I guess all that is required is that it works any way we
can make it work for beam studies.

For the real Run 2 I guess they (collaborators at D0, Dave Cutts,
Ray Zeller, Dean Schaumburger and others I know not of...) plan to make
changes to the VBD that include taking away "features" Central Tracking
people had grown used to but were going to be abandoned for run II
to recover microcode resources and to gain speed. The way the VBD
really works is not at all like it's (special Central Tracking Mode)
software registers appear. Dean Schamberger was most informative.
Parenthetical comments are my own unless a name appears next to them.
What follows is my current understanding from a conversation yesterday with
Dean S.:

1. All present VBD's in D0 will get new upgraded firmware for Run II.

(so that all VBD's are the same.) (M. Johnson)

2. The VBD never had and (never will) any A32 bit addressing capability.

It has no drivers for the high address byte. (The high byte

of the 32 bit address we have been loading to it's software

registers are placebos and ignored.) (Mike Matulik says that he

thinks he has seen the VBD drive the bus with 0148 hex in the

high 16 bit address on the VME Metro and that Mike Utes might know.)

3. Many Central Tracking "special Features" will be stripped out to

reduce and simplify the microcode in the VBD and to increase it's

speed. So basically the VBD as Central Tracking personal are used

to looking at it will have to change our evil ways:

a. The VBD support for byte count is expensive in time, and microcode

and will change to a native mode 32 bit word count.

(This was a special FADC patch to make it work with our

custom Zero suppression chip.)

b. The word count will always be obtained from short I/O

space: A16 AND the correct VME short I/O Address Modifier.

(This explains why the word count and the data FIFO must

be in the same 64K block. The way they did the patch was

to leave the high 8 bits float at the data address while it

did the A16 address. This kludge will be removed!)

All (true 32 bit word and not byte) word counts must be

available in A0 to A16 space and respond ONLY with

the CORRECT VME Address Modifier (I guess upper bits are

xxxxxxxx xxxxxxxx: whatever VME spech is...)

c. Data (including FIFO's) will be read from A24 space with the

A24 VME address modifier. (I guess the upper high byte is xxxxxxxx:

whatever VME spech is...

Can the data FIFO's really be 64K apart??)

(Obvious?) Implications of the above for the VRB (as I understand it.)
If all of the above is true..
Then what that means for the VRB is that it would move (back) to a
split addressing scheme similar to what it originally had except that
now the split would be A16/A24 instead of the original A24/A32 split.
If what Dean says is true then A32 support could be dropped???
Another complication Mike Matulik brought up is that Mike Baert's
controller card must now be brought down to A16 short I/O space so that
the VBD and controller card can hand shake?? (Isn't that done through
special hardware backplane lines??) Since the controller
card at present is presently ~32K of address space that doesn't leave a
lot of space for other things unless it is changed to a dual address
scheme as well...
The final infringement on this precious A16 short I/O space is
that whatever processor card will go in there will use up a chunk
of space as well. All of Central Tracking crates will have 68K's
or an as yet undecided processor sitting in them.

In my opinion YOU SHOULD NOT MAKE ANY (MORE) CHANGES ON D0's
BEHALF UNTIL D0 CAN GET IT'S POOR SELF TOGETHER AND figure out how all our
cards are going to really work and where and how they can be addressed
unless they are very minor fixes to what you currently have.

I sincerely apologize for my confusion over how the VBD works now
and how it will work in the future. I hope we will learn more
as a result of this mail message to the principles as I know them!

Sincerely,
R. Angstadt

-----------------------------------------------------------------------

Next is my reply to a letter from Dean S. signified by ">" with my comments terminated by "(r.a.)":

At 04:57 PM 5/10/97 -0400, you (Dean) wrote:
(Dean)
>Hi Bob,
>After talking with you, I remembered that we had discussed with Zeller
>the fact that someday one might want his board to exist in A32 space, but
>thought we decided to save money and not do that. I checked a board,
>and it is clear that it can understand A32 addresses, but it is less
>clear if it can DMA using A32 addressing. I will check with Ray Zeller,
>but it might be that the "block transfers" can be done in A32 mode. But
>the word count is still only supported in A16 mode.
>
>Another Feature of the VBD that we did not discuss, and Marvin knows, but
>maybe you are unfamiliar with concerns block transfer lengths. Because
>VME specs assume there are multiple bus users, with essentially equal
>priority, and that sometimes hardware can hang, they "suggest" that any
>bus master arbitrate for ownership of the bus atleast once every 20 micro-sec.

i wasn't aware of this VME rule. (r.a.)

(Dean)
>Also, a single block transfer is not allowed to be longer than 255 transfers.
>The VBD ignores both of these "rules". That is, once it has been granted bus
>control, it only releases the bus after it has finished the list of data
>to transfer (or detected an error). This can be a very long time, it the
>total data approaches 1/4 Mb (size of memory buffer in VBD). Also, the
>length of a single block transfer is only limited by the number of bits
>in the word count (2^16).

i think i've seen the VBD repeat the same fifo address every ~255 bytes on
the VME Metro and do the next block with the VRB in CT mode. Although both
Marks and I have seen things move in this block format I do not know if it
actually did release the bus for re-arbitration... I just assumed that
re-arbitration took place because the address was put on the bus but
i did but did not look in enought detail to say either way. Maybe one of
the Mark (Baert and Bowden) remember and/or have a printout of this?
The statement that the VBD does not re-arbitrate while processing it's
list rings true from FADC days but I don't know if it was ever patched
(fixed/changed) (r.a.)

>(Dean)
>That reminds me of another "feature" of the VBDs that is less well known.
>Although the VBd has 256Kb of memory, it uses 4 (or maybe 8) long-words of
>that memory for internal bookkeeping. That means that there are really only
>256Kb - 16 (or maybe 32) bytes. The less known feature is that if you send
>is a set of word-counts that exceeds 65532 (or maybe 4 less) words, it
>attempts to do the transfer anyway, trashes its internal pointers and
>(typically but not always) continues that last DMA transfer for ever. Since
>it has bus ownership, the only was for the transfer to end, is if your VRB
>stops acknowledging its requests. This was never a problem in run 1, because
>we knew exactly how large the maximum data from a crate could be, and
>limited it to less than 256Kb. So this only happened when the wordcount
>(in your case byte count) was corrupted. Although I am not sure about the
>final crate assignments, because of all the silicon channels, I suspect that
>"rule" is not satisfied for run 2.

This is the best explanation of the length limits I've heard yet. thank you! (r.a.)

>
>Dean
>

do you have Zeller's email address?? i guess i've lost it. my last email to him
bounced. (could one of you forward my last message concerning this to him?)
thanks again.
R. Angstadt

-----------------------------------------------------------------------

Next is all from Dave Cutts with no comments by me (or anyone):

Date: Mon, 12 May 1997 22:12:34 -0400 (EDT)
From: Dave Cutts <cutts@hep.brown.edu >
(rest of header deleted r.a.)
Hi Dean and Bob,

i am faxing the entire collection of messages to Ray and spoken
with him just now (at 10pm). i had expected him to come into Brown
and had left them for him. (i am ay BNL all week, groan, on a Review).

regretfully this Review is meetings from 8am to 6pm solid so i'm on
net only at night.

sounds like there's useful progress being made! Bob - keep asking the questions!

regards
dave

-----------------------------------------------------------------------
-----------------------------------------------------------------------

IMPLICATIONS AS I (r. angstadt) SEE THEM AS OF 5/15/97 ~16:28/P

For Run II CT tries to design it’s addressing of its boards towards a split A16/A24 addressing scheme (with A32 support just dropped to keep it’s FPGA’s lean and mean?) as Dean Describes with the true 32 bit word count packed list in VME A16 space (with A16 short I/O space address modifiers such as hex 2d) and the data (FIFO’s) up in A24 space with an A24 VME address modifiers. I don't see any problem with word counts, data, FIFO's, and control registers all mixed in A24 space as were the FADC's. This was not a problem in Run 1 (that I know of) for CT and I don't see how it can be a problem in Run II as long as it is a known and fixed repetitive scheme.

1. What impact does this have on the SARC? (Does the VBD get the 32 bit word count from the VRB or the SARC? So if from the VRB then the SARC does not have to reside in short I/O space and can do hand shaking across special back plane lines????? Mark Baert (et. al.) needs to answer this.
2. Where to put the A16 address space of the VRB? As Dean points out every processor will be different and CT carries the unfortunate burden of having a processor in every crate. Previously the D0 high voltage lived comfortably with various Motorla 68K boards in VME short I/O space with the first board at hex ff8000 , the second at hex ff8100..up to six boards at ff8500. Each board took 256 bytes. This ff8xxx region might be utilized in some manner:
1. If all the VBD uses is the word count then my proposal to Mark Bowden is to just echo the 32 bit word count down here at say ff8600 for the first board, ff8604 for the second, ff8606 for the third ect. (Make the base address movable with dip switches and the word count increment with the packplane address pins.) If the VBD needs some information that the SARC has maybe Mark Baert can echo just that information down in this space as well.
2. Put everything down here in a 256 byte wrap (if it will fit) and not bother with A24 space except for diagnostics?
3. It would be nice to know for sure if the VBD can or cannot drive A31 to A24 lines (counting from 0) or the high byte of an A32 address. (It would really be nice if maybe Ray Zeller. could comment on this.) If this is true then all the Marks could drop A32 from their FPGA code in D0 mode to make their stuff run as lean and mean as the VBD.
4. The last question and perhaps most important question is can the VBD process a list of FIFO' or data using all of the 24 bit address space. I.e., Can the list of FIFO's be 65535 bytes apart? (Well Ray, can it?) (updated May. 16, 1997 at 08:42)

-----------------------------------------------------------------------

-----------------------------------------------------------------------

Next is all from Dave Cutts with no comments by me. (i apologize for slight format changes due to difficulties in controlling how html handles space and tabs but every effort has been made to preserve basic format and content. r.a.) :

Return-Path: <cutts@hep.brown.edu>
Date: Thu, 22 May 1997 14:02:24 -0400 (EDT)
From: Dave Cutts <cutts@hep.brown.edu >
Subject: re: VBD usage
To: angstadt@FNAL.GOV
Cc: utes@FNAL.GOV, dave <cutts@brown.edu >, baert@FNAL.GOV,
Marvin <mjohnson@FNAL.GOV >, Jan <hoftun@hep.brown.edu >,
Gordon <gwatts@FNAL.GOV >, Dean <dean@cusb.physics.sunysb.edu >
Reply-to: Dave Cutts <cutts@hep.brown.edu >
Content-id: <32316.864324144.5816274.14317@ >
Content-disposition: inline
X-Importance: normal
X-Sensitivity: normal

hi Bob,

i have talked about the VBD with Dean and also checked with Ray.

Dean tells me that at the D0 ECB meeting recently he discussed
matters with Marvin, concerning what guidelines should be applied
in making the convergence between the VBD and the VRB functionality.
specifically, for Run 1, Ray added functionality to the VBD that
was not in the D0 spec, in order to work with the FADCs.
the readout worked, but the added functionality is costly in terms
of the total readout time (it results in additional setup time to
each DMA operation). at the ECB meeting Friday, Marvin affirmed
his earlier agreement that for Run 2 these added work-arounds would
not be required, but the VBD would be operated in accordance to
its design - so as to maximize the realized bandwidth.

in short, the VBD will then operate,
in reading WCs : A16D16
in DMA : A24D32

an expanded description:

1. 1. reading word counts: there is a list of 16-bit numbers (previously loaded into the VBD) which specify 16 bit addresses; each number corresponds to: a word count (that is the number of longwords [32-bit] to be read by the subsequent DMA). this word count is read via address modifiers that specify 16-bit address decoding only, and its location responds with 2 bytes of data. (in short, a A16D16 cycle)
   
2. 2. reading the data: there is another list (previously loaded into the VBD) each entry of which provide 2 16-bit numbers, of which according to the original VBD spec provide 24-bits of information, which is the address of the data to be read. then for the DMA, the VBD supplies an address modifier defining 24-bit addressing, and gives 3 bytes of address information; and reads with each of its data strobes 4 bytes of information (in short, A24D32)

......................................................

Further notes:

1. ) the VBD was actually built so that it supports DMAs with an address modifier for 32 bits; the upper 8 bits are available and can be programmed. however, one cannot cross a 16 Meg boundary as the counter is still only 24 bits.
2. ) the VBD as it stands strictly complies with all the VME rules in the VME specs. the issue for example about releasing the bus periodically and arbitrating are 'suggested' in the VME spec but they are NOT rules as it is clear that for some applications (such as realtime devices needing optimum performance) such guidelines are not appropriate. further, it is incumbent that all other modules in the system must work with devices that are built strictly in accordance with the VME rules, and not reqire or expect that 'suggestions' be followed.

--------------------------------------------------------------------

Bob,

Ray's ISDN lines are down so he is not directly reachable via
email. i'm happy to act as a conduit, tho i may need to talk with Ray
or Dean to respond with some knowledge.

i hope the above is helpful.

this information (and lots more) is also available in your copy
of the Users Manual.

cheers
dave

---------------------------------------------------------------------
---------------------------------------------------------------------

Thanks a lot Dave! This gives us a lot better guidlines.
We'll try to work towards the above rules!