Would you like to make this site your homepage? It's fast and easy...
Yes, Please make this my home page!
================================================================================
Note 26.0 MicroVAX II multicomputing No replies
FURILO::JACKSON 700 lines 19-AUG-1985 11:36
--------------------------------------------------------------------------------
+---------------+ +-----------------+
| d i g i t a l | | uNOTE # 026 |
+---------------+ +-----------------+
+----------------------------------------------------+-----------------+
| Title: The MicroVAX II CPU Multicomputing | Date: 28-JUN-85 |
| Capability | |
+----------------------------------------------------+-----------------+
| Originator: Mike Collins | Page 1 of 12 |
+----------------------------------------------------+-----------------+
The MicroVAX II CPU may be configured as an arbiter CPU or as one of
three auxiliary CPU's. Therefore it is possible to configure a Q-bus
system with multiple MicroVAX II CPUs. This 'multicomputing' capability
is the topic of this MicroNote.
Multicomputing Introduction
The multicomputing capability is a hardware design feature of the
MicroVAX II. It allows a maximum of four MicroVAX II CPUs to reside on
the same Q-bus. Figure 1 below is a block diagram which shows how 2 of
a possible 4 CPUs may be configured on the bus. One of the processors
will be the arbiter and the others will be auxiliaries.
NOTE
There is no shared memory between processors, therefore
this system should not be considered as symmetrical
multiprocessing.
Each processor may use expansion memory modules provided that there are
sufficient Q/CD slots in the backplane. See the configuration section
for more detail on the configuration rules.
+---------------------+ +---------------------+
| +---------+ | | +---------+ |
| | | | | | | |
+---------+ +---------+ +---------+ +---------+
| Arbiter | | | |Auxiliary| | |
|Processor| | Memory | |Processor| | Memory |
| | | | | | | |
| | | | | #1 | | |
| | | | | | | |
+---------+ +---------+ +---------+ +---------+
| |
| |
-----------------------------------------------------------
Q-bus
-----------------------------------------------------------
Page 2
Figure 1 - Multicomputing Configuration
CAUTION
Digital's 32-bit operating systems DO NOT support the
multicomputing feature. Users who wish to customize
their systems to take advantage of the multicomputing
capability may do so but are responsible for designing
the software system.
The following topics which are pertinent to the multicomputing design
will be described in detail:
1. MicroVAX II Memory System
2. Interprocessor Communications Register
3. Arbiter/auxiliary Processor Differences
4. Bootstrapping an Auxiliary Processor
5. Configuration Rules
REFERENCES
The MicroVAX 630 CPU Module User's Guide (P/N
EK-KA630-UG) and MicroNote #22 have more detailed
information on all aspects of the MicroVAX II CPU and
memory system.
MicroVAX II Memory System
--------------------------
Before describing the details of the multicomputing feature it is
necessary to understand the memory system of a MicroVAX II.
The memory system of the MicroVAX II is unique. Unlike previous Q-bus
processors, the memory modules do not communicate to the CPU over the
Q-bus. Instead, the communication and transfers occur over a dedicated,
closely coupled interconnect. This design allows for very fast reads
and writes to memory (400 nsec). Not only are the transfers very fast,
but none of this CPU-memory activity appears on the Q-bus. Therefore
the Q-bus is strictly for I/O.
Since memory is not directly on the Q-bus it is necessary to allow DMA
devices on the Q-bus to access memory. This feature is called the Q-bus
I/O map. It is a mechanism which maps Q-bus addresses (22-bit physical
address) to the local memory system of a MicroVAX II (24-bit physical
address). This same concept is used on larger UNIBUS based VAXes.
When one or more auxiliary processors are added to a system, remember
that each has its own local memory system. Therefore it is intended
that they will each operate out of their own memory. This design is
Page 3
well suited for applications which can be easily partitioned between
multiple processors such that only messages are passed between them.
It would be possible for a processor to operate out of memory on the
Q-bus but this would be unconventional from a systems standpoint and
would decrease system performance significantly. Digital's operating
systems do not support the use of standard Q-bus memories in a MicroVAX
II.
With only one processor on the Q-bus, the system software has complete
control over DMA transfers. The software controls the contents of the
Q-bus map registers. First, it determines whether the register is
enabled and second, if it is enabled, where the DMA transfers are
directed in the local memory system.
In a multiple processor configuration (since each has its own set of
Q-bus I/O map registers) it is now possible for multiple map registers
to respond to a single Q-bus address. It is the responsibility of the
system software running on each processor to cooperate and assure that
one and only one map register will respond to any Q-bus address.
Otherwise multiple memory locations may respond and the results are
indeterminate (i.e. multiple BRPLYs on the Q-bus for a single address).
A similar situation occurs if standard Q-bus memories are added to a
MicroVAX II system. The system software is again responsible for
assuring that for any address on the Q-bus which is 'shared' by a Q-bus
memory board and a Q-bus I/O map register, the map register is disabled
to allow only a Q-bus memory reply.
Q-bus memories would not be added to 'typical' systems. However there
are some special applications which may require this capability. For
example, a graphics system where the Q-bus memory is the bit map of a
graphics display.
On power-up, the MicroVAX II boot ROM will check for the presence of
Q-bus memory and clear the valid bits of the corresponding mapping
registers. This will prevent the above situation from occurring as long
as the system software does not alter the state of these valid bits at a
later time.
Interprocessor Communications Register
--------------------------------------
The interprocessor communications register (ICR) is the primary
mechanism of the MicroVAX II CPU which enables multiple processors to
cooperate and reside on the same Q-bus. Figure 3 describes the ICR and
the bit definitions.
The ICR provides the following four functions:
1. Any processor on the bus may interrupt another without using
the Q-bus interrupt lines. This is done by setting the
appropriate bit in the ICR of the second processor. The CPU
will then interrupt at IPL 14 (HEX) with an interrupt vector of
204 (HEX).
Page 4
2. The ICR also has a bit which controls external access to local
memory via the Q-bus map. When set this bit allows external
devices to access the local memory system. When reset, this
bit prevents the local memory system from responding to any
Q-bus address references.
3. If a processor is an auxiliary then the ICR provides a
mechanism which allows it to be halted by other CPUs. This
feature is used at power-up to coordinate the bootstrap
process.
4. Parity errors are identified when one occurs during accesses to
the MicroVAX II's local memory.
The address for the ICR is located in the Q-bus I/O page address space
and may be accessed by any device which can become Q-bus master. The
ICR is byte-addressable.
Since it is possible to put a maximum of 4 MicroVAX II CPUs on one Q-bus
they each have their own unique ICR. Table 1 lists each processor and
the associated ICR address.
Table 1 - Interprocessor Communication Register Address Assignments
+--------------+----------------------+----------------------------+
| Processor | 22-bit Octal Address | 32-bit Hexadecimal Address |
+--------------+----------------------+----------------------------+
| Arbiter | 17 777 500 | 2000 1F40 |
+--------------+----------------------+----------------------------+
| Auxiliary #1 | 17 777 502 | 2000 1F42 |
+--------------+----------------------+----------------------------+
| Auxiliary #2 | 17 777 504 | 2000 1F44 |
+--------------+----------------------+----------------------------+
| Auxiliary #3 | 17 777 506 | 2000 1F46 |
+--------------+----------------------+----------------------------+
1 1 1 1 1 1
5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
+--+-----------------+--+--+--+--+-----------+--+
| | 0 0 0 0 0 0 | | | | | 0 0 0 0 | |
+--+-----------------+--+--+--+--+-----------+--+
| | | | |
DMA QPE ----+ | | | |
| | | |
AUX HLT -------------------------+ | | |
| | |
DBI IE -------------------------------+ | |
LM EAE ----------------------------------+ |
|
DBI RQ -------------------------------------------------+
Figure 3 - Interprocessor Communications Register
Page 5
Bit(s) Mnemonic Meaning
<15> DMA QPE DMA Q22-Bus Address Space Parity Error.
This read-only bit is set if Memory System
Error Register bit <04> (DMA QPE) is set.
The DMA QPE bit indicates that a parity
error occurred when an external device (or
CPU) was accessing the MicroVAX II CPU
local memory.
<14:09> - Unused. Read as zeros.
<08> AUX HLT Auxiliary Halt. On an auxiliary MicroVAX,
AUX HLT is a read-write bit. When set,
typically by the arbiter CPU, it causes
the on-board CPU to transfer program
control to the Halt Mode ROM Code. On
an arbiter MicroVAX II, AUX HLT is a
read-only bit which always reads as zero.
It has no effect on arbiter CPU operation.
<07> - Unused. Read as zero.
<06> DBI IE Doorbell Interrupt Enable. This bit, when
set, enables interprocessor doorbell
interrupt requests via ICR <00>. When the
on-board CPU is Q22-Bus master, DBI IE
is a read-write bit. When an external
device (or CPU) is bus master, DBI IE
is a read-only bit. DBI IE is cleared
by power up, by the negation of DCOK and
by writes to the Bus Initialize Register.
<05> LM EAE Local Memory External Access Enable. This
bit, when set, enables external access to
local memory (via the Q22-Bus map). When
the on-board CPU is Q22-Bus master, LM EAE
is a read-write bit. When an external
device (or CPU) is bus master, LM EAE is
a read-only bit. LM EAE is cleared by
power up, by the negation of DCOK and
by writes to the Bus Initialize Register.
<04:01> - Unused. Read as zeros.
<00> DBI RQ Doorbell Interrupt Request. If ICR <06>
(DBI IE) is set, writing a "1" to DBI RQ
sets DBI RQ, thus requesting a doorbell
interrupt. If ICR <06> is clear, writing
a "1" to DBI RQ has no effect. Writing a
"0" to DBI RQ has no effect. DBI RQ is
cleared when the CPU grants the doorbell
interrupt request. DBI RQ is held clear
whenever DBI IE is clear.
Page 6
When a processor is interrupted via its ICR the interrupt vector is 204
(Hex). This vector is used when the interrupting device is the arbiter,
auxiliary 1, auxiliary 2, auxiliary 3, or any device which may become
Q-bus master. Therefore it is the responsibility of the interrupt
service routine to determine which device caused the interrupt since the
same vector is used no matter what device set the DBI RQ bit. This
interrupt occurs at the same interrupt priority level (IPL) as IRQ4 on
the Q-bus.
NOTE
Following such an interrupt, the MicroVAX II sets the
IPL to 14 (Hex). This is different from what happens
after a standard Q-bus interrupt. When a Q-bus
interrupt occurs on IRQ4, the processor raises the IPL
to 17 (Hex) and it is the responsibility of the
interrupt service routine to lower the IPL later on.
Arbiter/Auxiliary Processor Differences
---------------------------------------
There are several differences between arbiter and auxiliary processors.
It is important to understand these differences when configuring
multiple processors into a system.
1. The arbiter MicroVAX II arbitrates bus mastership in accordance
with the Q22-Bus DMA protocol. The arbitration logic is
disabled on an auxiliary MicroVAX II.
2. Both the arbiter and auxiliary MicroVAX II request bus
mastership via the Q22-Bus DMA Request protocol.
a. They both assert BDMR on the Q22-Bus.
b. The arbiter MicroVAX II receives DMGI from its arbitration
logic. But an auxiliary receives DMGI from its Q22-Bus
BDMGI pin.
c. Only the auxiliary MicroVAX II actually asserts BSACK on
the Q22-Bus.
3. The arbiter MicroVAX II asserts the Q22-Bus BINIT signal when
DCOK is negated and when its CPU software writes to its Bus
Initialize Register. An auxiliary MicroVAX II never asserts
Q22-Bus BINIT, but receives BINIT and uses it to initialize the
MicroVAX chip and to clear all internal registers which are
cleared by the negation of DCOK.
4. The physical address of the Interprocessor Communication
Register is different for each of the four MicroVAX II
arbiter/auxiliary configurations.
Page 7
5. An auxiliary MicroVAX II can be halted by setting bit <08> (AUX
HLT) of its Interprocessor Communication Register. On an
arbiter MicroVAX II this feature is disabled and AUX HLT is a
read-only bit which always reads as zero.
6. The CPU halts are controlled by the external connector HLT ENB
input. However, the external halts which are affected differ
somewhat for the arbiter and auxiliary MicroVAX II modules. If
the HLT ENB signal is asserted (low) the a MicroVAX II CPU will
halt and enter the console program if:
a. The program executes a halt instruction in kernel mode.
b. The console detects a break character.
c. Arbiter CPU only - the Q-bus halt line is asserted.
d. Auxiliary CPU only - the Interprocessor Communication
Register AUX HLT bit is set.
If the HLT ENB signal is negated then:
a. The halt line and break character are ignored and the ROM
program responds to a halt instruction by restarting or
rebooting the system.
b. If the MicroVAX CPU is an auxiliary, the ROM program
responds to the assertion of the ICR AUX HLT bit by
rebooting.
The state of the HLT ENB bit can be read by software via the
Boot and Diagnostic Register.
7. Each arbiter or auxiliary MicroVAX II module can field
interrupt requests from its interval timer, from its console
device, and from its interprocessor doorbell. Only the arbiter
MicroVAX II can field interrupts from Q22-Bus interrupt request
lines IRQ7-4.
8. The arbiter asserts BIAKO on the Q22-Bus when it responds to a
Q22-Bus interrupt request. An auxiliary asserts BIAKO on the
Q22-Bus when it receives the assertion of BIAKI in order to
pass it through to devices after it.
9. Although both the arbiter and auxiliary MicroVAX II modules
contain the same time-of-year clock and battery back-up
circuitry, it is assumed that the auxiliary will be configured
without batteries and that its clock will never actually be
enabled. This configuration will ensure a single time base for
the system. If an auxiliary needs to set its time-of-year
clock it can do so by referencing the arbiter's clock.
10. An arbiter processor can bootstrap from a variety of devices
but an auxiliary will always boot using the ROM bootstrap
protocol. See the following section for a detailed description
Page 8
of the bootstrap process.
Bootstrapping an Auxiliary Processor
------------------------------------
Since there are distinct differences between the operation and
capabilities of arbiter and auxiliary processors, it is necessary to
bootstrap them differently. An arbiter processor bootstraps in the
conventional manner from one of several devices but an auxiliary boots
using only one method.
On power-up both processors initialize themselves and perform some self
tests. At this point the primary bootstrap (VMB) begins to execute. An
arbiter processor may bootstrap from any one of the following devices:
1. DU type device (RX50, RDxx, RC25 or RAxx)
2. TK50 tape
3. ROM bootstrap protocol
4. Ethernet (DEQNA)
An auxiliary processor will not attempt to boot from disk, tape or
ethernet. It will always boot via the ROM bootstrap protocol, which is
described below.
In order to synchronize the bootstrap process, after power-up and
initialization an auxiliary processor will not boot until it is allowed
to do so by the arbiter. The controlling device is not required to be
the arbiter, it could be another auxiliary or DMA device which is bus
master, but it is the most logical point of control.
The following steps summarize the bootstrap procedure for a system with
an arbiter and one (or more) auxiliaries:
1. Both types of processors initialize themselves after power-up.
2. Self-diagnostics execute to check out major sections of the
CPU.
3. The arbiter boots from one of the four device types listed
above.
a. While the arbiter is booting the auxiliary completes its
diagnostics and waits to start its bootstrap process.
b. The auxiliary clears the valid bit in each of its Q-bus map
registers to prevent accidental transfers to or from its
local memory.
Page 9
c. The auxiliary loops doing nothing until the AUX HLT bit in
its ICR is cleared. When this bit is finally cleared, the
auxiliary boots via the ROM bootstrap protocol.
4. Somewhere in the system the appropriate data structure has been
created to allow the auxiliary to boot via the ROM bootstrap
protocol. This can be done several ways:
a. The bootstrap code is actually in ROM on an MRV11-D module.
b. The bootstrap code is loaded into non-volatile RAM.
c. The arbiter CPU loads the bootstrap code into its own local
memory then sets up some Q-bus map registers so that this
data can be seen by the auxiliary CPU over the Q-bus. This
method is the most flexible since the bootstrap code can be
changed easily.
5. When the arbiter is ready and knows auxiliary boot code is
available, it allows the auxiliary CPU to bootstrap by clearing
the AUX HLT bit in its ICR.
ROM Bootstrap Protocol.
To locate a PROM bootstrap, VMB searches the Q22-bus address range from
high to low addresses by page (512 bytes per page) looking for readable
memory. If the first six longwords of any such page contains a valid
PROM "signature block" (see figure 4), VMB passes control directly to
the bootstrap code in the PROM, it does not copy the PROM code to local
memory for execution as it does for all other secondary bootstraps.
Note that while defined as an MRV11 PROM or equivalent bootstrap, VMB
does not actually require that the signature block or the bootstrap code
be in PROM. It could be in ROM, nonvolatile RAM or it could be loaded
into another MicroVAX II's RAM and mapped to the Q22-bus thus enabling
an auxiliary to see it.
RB:
+---------------+---------------+---------------+---------------+
+ 0: | check byte | any value | 0 | 18 (hex) |
+---------------+---------------+---------------+---------------+
+ 4: | any value | 1 | 0 |
+-------------------------------+---------------+---------------+
+ 8: | size of PROM in pages |
+---------------------------------------------------------------+
+ 12: | must be zero |
+---------------------------------------------------------------+
+ 16: | offset into PROM to start execution |
+---------------------------------------------------------------+
+ 20: | sum of previous three longwords |
+---------------------------------------------------------------+
Figure 4: PROM Bootstrap Memory Format
Page 10
RB + 0: This byte must be 18 (hex).
RB + 1: This byte must be 0.
RB + 2: This byte may be any value.
RB + 3: This byte must be the ones complement of the sum
of the previous three bytes.
RB + 4: This byte must be zero.
RB + 5: This byte must be 1.
RB + 6: These two bytes may be any value.
RB + 8: This longword contains the size in pages of the
PROM.
RB + 12: This longword must be zero.
RB + 16: This longword contains the byte offset into the
PROM where execution is to begin.
RB + 20: This entry is a longword containing the sum of
the previous three longwords.
Configuration Rules
-------------------
In order to configure multiple MicroVAX II CPUs onto one bus, 4 issues
must be addressed:
1. Q-bus slot requirements for the CPU and MS630 memory boards.
2. Compatible enclosures or boxes.
3. Bus termination.
4. Using a PDP-11 CPU as the Arbiter.
1. Q-bus slot requirements.
To operate properly both the MicroVAX II CPU and all versions of the
MS630 memory modules MUST be inserted in Q-bus slots which feature Q-bus
signals on the A/B side and the CD interconnect on the C/D side. The
MicroVAX II CPU and any expansion memory modules must all be adjacent in
the backplane.
WARNING
The MicroVAX II CPU WILL be damaged if inserted into a
slot which has Q-bus signals on the C/D side.
Page 11
2. Compatible enclosures or boxes.
Since Q/CD slots are required, there are only 4 enclosures which are
compatible with these modules. They are:
a. BA23 (8 slots total, first 3 are Q/CD)
b. BA123 (12 slots total, first 4 are Q/CD)
c. BA11-S (9 slots total, all 9 are Q/CD)
d. BA11-N (9 slots total, all 9 are Q/CD) This enclosure is
electrically compatible but since it is older it is only 18-bit
compatible and has not been tested for FCC compliance in a
system package. Since the BA11-S is the 22-bit, updated
version, the BA11-N enclosure is not recommended.
3. Bus termination.
Each MicroVAX II processor has 240 ohm termination. When multiple CPUs
are placed into one system, take care to abide by the Q-bus
specification regarding termination and configuring multiple box
systems.
REFERENCE
Reference MicroNote #29, entitled 'Q-bus Expansion
Concepts' and appendix A of the Microsystems Handbook
(P/N EB-26085-41) for more information concerning Q-bus
configuration rules.
The BA11-S enclosures are best suited for multiple CPUs because they
have backplanes with the Q22/CD configuration in all slots. Therefore
multiple CPUs and expansion memory can be accommodated most easily. The
other potential boxes, the BA23 and BA123, are not as flexible because
they only have 3 and 4 Q22/CD slots respectively.
When using multiple MicroVAX II CPUs in one system the goal should be to
configure each backplane with 120 ohm termination if the number of
backplanes used is one or two. In the three backplane case, only the
first and third backplanes should have 120 ohm termination. This
implies that no MicroVAX II CPUs should be configured into the middle
backplane of a three backplane system (three backplanes is the maximum
number allowed for the Q-bus).
The following configurations assume the BA11-S enclosures will be used.
Two CPU System:
A two CPU system is the easiest case to configure. As mentioned
earlier, each CPU has 240 ohm termination. With two CPUs in the same
backplane, the proper overall termination of 120 ohms is achieved. The
first CPU has 240 ohms in parallel with 240 ohms on the second CPU which
is 120 ohms, assuming there is no termination on the backplane (which is
Page 12
true for the BA11-S enclosure).
Three CPU System:
A three CPU system should be configured in 2 backplanes to maintain 120
ohm termination in each. The first 2 CPUs should be configured in the
first box which terminates that backplane properly as in the case above.
The third CPU should be placed in the second backplane. The modules of
the expansion cable set must be terminated differently. The module of
the expansion cable set in the first backplane should add no additional
termination to the system while the module in the second backplane
should be terminated with 240 ohms. Therefore the second backplane is
also terminated in 120 ohm (240 on the processor in parallel with 240 on
the expansion module).
Four CPU System:
A four CPU system should also be configured in 2 backplanes. The first
backplane will have 2 CPUs for 120 ohm termination and the second
backplane will also have 2 CPUs for 120 ohm termination. But the
expansion cable set connecting the two backplanes will not be the same
as in the three CPU system. In this case there should be no termination
on either of the expansion modules.
These same basic rules should be followed if the BA23 enclosure is used.
However the configurations become more constrained because of the
relatively few number of Q/CD slots. The configuration also becomes
more complicated because the BA23 has built in termination on the
backplane (termination resistors are socketed and therefore removable).
4. Using a PDP-11 CPU as the Arbiter.
The arbiter can be a Q-bus PDP-11 CPU as well as a MicroVAX II CPU. In
this configuration only the 3 auxiliary MicroVAXes could be added to the
system. Although there are tremendous architectural differences between
a PDP-11 and MicroVAX CPU, they both perform the arbiter function on the
Q-bus and therefore such a configuration is possible. Two issues arise
from this 'mixed' configuration.
1. All Q-bus PDP-11s to-date have their main memory on the Q-bus. In
order to talk to an auxiliary MicroVAX CPU's local memory, part of
the Q-bus physical address space must be reserved for mapping to
that local memory.
2. The PDP-11 processor will not have an interprocessor communication
register because that feature is unique to the MicroVAX II CPU.
Therefore, if it is necessary to have the capability for an
auxiliary CPU to interrupt the arbiter, an additional device must be
added to the system. This device will look like the interprocessor
communication register for an arbiter and will convert and ICR
interrupt into a conventional Q-bus interrupt (there is no
capability which allows an auxiliary to interrupt the arbiter via
the standard Q-bus interrupt protocol).
