Appendix A Hardware Overview
This appendix discusses general issues about hardware capable of supporting the Solaris 9 operating environment. This includes issues related to the processor, bus architectures, and memory models supported by the Solaris 9 operating environment, various device issues, and the PROM used in Sun platforms.
Note –
The information presented here is for informational purposes only and might be of help during driver debugging. However, the DDI/DKI interfaces hide many of these implementation details from device drivers.
This appendix provides information on the following subjects:
SPARC Processor Issues
This section describes a number of SPARC processor-specific topics including data alignment, byte ordering, register windows, and availability of floating-point instructions. For information on IA processor-specific topics, see IA Processor Issues.
Caution – Drivers should never perform floating-point operations, as they are not supported in the kernel.
SPARC Data Alignment
All quantities must be aligned on their natural boundaries. Using standard C data types:
-
short integers are aligned on 16-bit boundaries.
-
int integers are aligned on 32-bit boundaries.
-
long integers are aligned on either 32-bit boundaries or 64-bit boundaries, depending on whether the data model of the kernel is 64-bit or 32-bit. For information on data models, see Appendix C, Making a Device Driver 64-Bit Ready .
-
long long integers are aligned on 64-bit boundaries.
Usually, the compiler handles alignment issues. However, driver writers are more likely to be concerned about alignment as they must use the proper data types to access their device. Because device registers are commonly accessed through a pointer reference, drivers must ensure that pointers are properly aligned when accessing the device.
SPARC Structure Member Alignment
Because of the data alignment restrictions imposed by the SPARC processor, C structures also have alignment requirements. Structure alignment requirements are imposed by the most strictly-aligned structure component. For example, a structure containing only characters has no alignment restrictions, while a structure containing a long long member must be constructed to guarantee that this member falls on a 64-bit boundary.
SPARC Byte Ordering
The SPARC processor uses big-endian byte ordering; in other words, the most significant byte (MSB) of an integer is stored at the lowest address of the integer.
SPARC Register Windows
SPARC processors use register windows. Each register window consists of 8 in registers, 8 local registers, and 8 out registers (which are the in registers of the next window). There are also 8 global registers. The number of register windows ranges from 2 to 32, depending on the processor implementation.
Because drivers are normally written in C, the compiler usually hides the fact that register windows are used. However, you might have to use them when debugging the driver.
SPARC Multiply and Divide Instructions
The Version 7 SPARC processors do not have multiply or divide instructions. These instructions are emulated in software and should be avoided. Because a driver cannot determine whether it is running on a Version 7, Version 8, or Version 9 processor, intensive integer multiplication and division should be avoided if possible. Instead, use bitwise left and right shifts to multiply and divide by powers of two.
The SPARC Architecture Manual, Version 9, contains more specific information on the SPARC CPU. The SPARC Compliance Definition, Version 2.4, contains details of the SPARC V9 application binary interface (ABI). It describes the 32–bit SPARC V8 ABI and the 64–bit SPARC V9 ABI. You can obtain this document from SPARC International at http://www.sparc.com.
IA Processor Issues
There are no alignment restrictions on data types. However, extra memory cycles might be required for the IA processor to properly handle misaligned data transfers.
Caution – Drivers should not perform floating-point operations, as they are not supported in the kernel.
IA Byte Ordering
The IA processor uses little-endian byte ordering. The least significant byte (LSB) of an integer is stored at the lowest address of the integer.
IA Architecture Manuals
Intel Corporation publishes a number of books on the IA family of processors. See http://www.intel.com.
Endianness
To achieve the goal of multiple-platform, multiple-instruction-set architecture portability, host bus dependencies were removed from the drivers. The first dependency issue to be addressed was the endianness (or byte ordering) of the processor. For example, the IA processor family is little-endian while the SPARC architecture is big-endian.
Bus architectures display the same endianness types as processors. The PCI local bus, for example, is little-endian, the SBus is big-endian, the ISA bus is little-endian, and so on.
To maintain portability between processors and buses, DDI-compliant drivers must be endian neutral. Although drivers could conceivably manage their endianness by runtime checks or by preprocessor directives like #ifdef _LITTLE_ENDIAN or _BIG_ENDIAN statements in the source code, long-term maintenance would be troublesome. In some cases, the DDI framework performs the byte swapping using a software approach. In other cases, where byte swapping can be done by hardware (as in memory management unit (MMU) page-level swapping or by special machine instructions), the DDI framework will take advantage of the hardware features to improve performance.
Figure A–1 Byte Ordering Host Bus Dependency
Along with being endian-neutral, portable drivers must also be independent from data ordering of the processor. Under most circumstances, data must be transferred in the sequence instructed by the driver. However, sometimes data can be merged, batched, or reordered to streamline the data transfer, as illustrated in Figure A–2. For example, data merging can be applied to accelerate graphics display on frame buffers. Drivers have the option to advise the DDI framework to use other optimal data transfer mechanisms during the transfer.
Figure A–2 Data Ordering Host Bus Dependency
Store Buffers
To improve performance, the CPU uses internal store buffers to temporarily store data. This can affect the synchronization of device I/O operations. Therefore, the driver needs to take explicit steps to make sure that writes to registers are completed at the proper time.
For example, when access to device space (such as registers or a frame buffer) is synchronized by a lock, the driver needs to check that the store to the device space has actually completed before releasing the lock. Releasing the lock does not guarantee the flushing of I/O buffers.
To give another example, when acknowledging an interrupt, the driver usually sets or clears a bit in a device control register. The driver must ensure that the write to the control register has reached the device before the interrupt handler returns. Similarly, if the device requires a delay (the driver busy-waits) after writing a command to the control register, the driver must ensure that the write has reached the device before delaying.
If the device registers can be read without undesirable side effects, verification of a write can be as simple as reading the register immediately after writing to it. If that particular register cannot be read without undesirable side effects, another device register in the same register set can be used.
System Memory Model
The system memory model defines the semantics of memory operations such as load and store and specifies how the order in which these operations are issued by a processor is related to the order in which they reach memory. The memory model applies to both uniprocessors and shared-memory multiprocessors. Two memory models are supported: total store ordering (TSO) and partial store ordering (PSO).
Total Store Ordering (TSO)
TSO guarantees that the sequence in which store, FLUSH, and atomic load-store instructions appear in memory for a given processor is identical to the sequence in which they were issued by the processor.
Both IA and SPARC processors support TSO.
Partial Store Ordering (PSO)
PSO does not guarantee that the sequence in which store, FLUSH, and atomic load-store instructions appear in memory for a given processor is identical to the sequence in which they were issued by the processor. The processor can reorder the stores so that the sequence of stores to memory is not the same as the sequence of stores issued by the CPU.
SPARC processors support PSO; IA processors do not.
For SPARC processors, conformance between issuing order and memory order is provided by the system framework using the STBAR instruction: if two of the above instructions are separated by an STBAR in the issuing order of a processor, or if they reference the same location, the memory order of the two instructions is the same as the issuing order. Note that enforcement of strong data ordering in DDI-compliant drivers is provided by the ddi_regs_map_setup(9F) interface. Compliant drivers cannot use the STBAR instruction directly.
See the SPARC Architecture Manual, Version 9, for more details on the SPARC memory model.
Bus Architectures
This section describes device identification, device addressing, and interrupts.
Device Identification
Device identification is the process of determining which devices are present in the system. Some devices are self-identifying—the device itself provides information to the system so that it can identify the device driver that needs to be used. SBus and PCI local bus devices are examples of self-identifying devices. On SBus, the information is usually derived from a small Forth program stored in the FCode PROM on the device. Most PCI devices provide a configuration space containing device configuration information. See the sbus(4) and pci(4) man pages for more information.
All modern bus architectures require devices to be self-identifying.
Supported Interrupt Types
Solaris supports both polling and vectored interrupts. The Solaris 9 DDI/DKI interrupt model is the same for both. See Chapter 7, Interrupt Handlers for more information about interrupt handling.
Bus Specifics
This section covers addressing and device configuration issues specific to the buses that Solaris supports.
PCI Local Bus
The PCI local bus is a high-performance bus designed for high-speed data transfer. The PCI bus resides on the system board and is normally used as an interconnect mechanism between highly integrated peripheral components, peripheral add-on boards, and host processor or memory systems. The host processor, main memory, and the PCI bus itself are connected through a PCI host bridge, as shown in Figure A–3.
A tree structure of interconnected I/O buses is supported through a series of PCI bus bridges. Subordinate PCI bus bridges can be extended underneath the PCI host bridge to enable a single bus system to be expanded into a complex system with multiple secondary buses. PCI devices can be connected to one or more of these secondary buses. In addition, other bus bridges, such as SCSI or USB, can be connected.
Every PCI device has a unique vendor ID and device ID. Multiple devices of the same kind are further identified by their unique device numbers on the bus where they reside.
Figure A–3 Machine Block Diagram
The PCI host bridge provides an interconnect between the processor and peripheral components. Through the PCI host bridge, the processor can directly access main memory independent of other PCI bus masters. For example, while the CPU is fetching data from the cache controller in the host bridge, other PCI devices can also access the system memory through the host bridge. The advantage of this architecture lies in its separation of the I/O bus from the processor's host bus.
The PCI host bridge also provides data access mappings between the CPU and peripheral I/O devices. It maps every peripheral device to the host address domain so that the processor can access the device through programmed I/O. On the local bus side, the PCI host bridge maps the system memory to the PCI address domain so that the PCI device can access the host memory as a bus master. Figure A–3 shows the two address domains.
PCI Address Domain
The PCI address domain consists of three distinct address spaces: configuration, memory, and I/O space.
PCI Configuration Address Space
Configuration space is defined geographically; in other words, the location of a peripheral device is determined by its physical location within an interconnected tree of PCI bus bridges. A device is located by its bus number and device (slot) number. Each peripheral device contains a set of well-defined configuration registers in its PCI configuration space. The registers are used not only to identify devices but also to supply device configuration information to the configuration framework. For example, base address registers in the device configuration space must be mapped before a device can respond to data access.
The method for generating configuration cycles is host dependent. In IA machines, special I/O ports are used. On other platforms, the PCI configuration space can be memory-mapped to certain address locations corresponding to the PCI host bridge in the host address domain. When a device configuration register is accessed by the processor, the request is routed to the PCI host bridge. The bridge then translates the access into proper configuration cycles on the bus.
PCI Configuration Base Address Registers
The PCI configuration space consists of up to six 32-bit base address registers for each device. These registers provide both size and data type information. System firmware assigns base addresses in the PCI address domain to these registers.
Each addressable region can be either memory or I/O space. The value contained in bit 0 of the base address register identifies the type. A value of 0 in bit 0 indicates a memory space and a value of 1 indicates an I/O space. Figure A–4 shows two base address registers: one for memory; the other for I/O types.
Figure A–4 Base Address Registers for Memory and I/O
PCI Memory Address Space
PCI supports both 32-bit and 64-bit addresses for memory space. System firmware assigns regions of memory space in the PCI address domain to PCI peripherals. The base address of a region is stored in the base address register of the device's PCI configuration space. The size of each region must be a power of two, and the assigned base address must be aligned on a boundary equal to the size of the region. Device addresses in memory space are memory-mapped into the host address domain so that data access to any device can be performed by the processor's native load or store instructions.
PCI I/O Address Space
PCI supports 32-bit I/O space. I/O space can be accessed differently on different platforms. Processors with special I/O instructions, like the Intel processor family, access the I/O space with in and out instructions. Machines without special I/O instructions will map to the address locations corresponding to the PCI host bridge in the host address domain. When the processor accesses the memory-mapped addresses, an I/O request will be sent to the PCI host bridge, which then translates the addresses into I/O cycles and puts them on the PCI bus. Memory-mapped I/O is performed by the native load/store instructions of the processor.
PCI Hardware Configuration Files
Hardware configuration files should be unnecessary for PCI local bus devices. However, on some occasions drivers for PCI devices need to use hardware configuration files to augment the driver private information. See the driver.conf(4) and pci(4) man pages for further details.
SBus
Typical SBus systems consist of a motherboard (containing the CPU and SBus interface logic), a number of SBus devices on the motherboard itself, and a number of SBus expansion slots. An SBus can also be connected to other types of buses through an appropriate bus bridge.
The SBus is geographically addressed; each SBus slot exists at a fixed physical address in the system. An SBus card has a different address, depending on which slot it is plugged into. Moving an SBus device to a new slot causes the system to treat it as a new device.
The SBus uses polling interrupts. When an SBus device interrupts, the system only knows which of several devices might have issued the interrupt. The system interrupt handler must ask the driver for each device whether it is responsible for the interrupt.
SBus Physical Address Space
Table A–1 shows the physical address space layout of the Sun Ultra 2 computer. A physical address on the Ultra 2 consists of 41 bits. The 41-bit physical address space is further broken down into multiple 33-bit address spaces identified by PA(40:33).
Table A–1 Device Physical Space in the Ultra 2|
PA(40:33) |
33-bit Space |
Usage |
|---|---|---|
|
0x0 |
0x000000000 - 0x07FFFFFFF |
2 Gbytes Main memory |
|
0x80 – 0xDF |
Reserved on Ultra 2 |
Reserved on Ultra 2 |
|
0xE0 |
Processor 0 |
Processor 0 |
|
0xE1 |
Processor 1 |
Processor 1 |
|
0xE2 – 0xFD |
Reserved on Ultra 2 |
Reserved on Ultra 2 |
|
0xFE |
0x000000000 - 0x1FFFFFFFF |
UPA Slave (FFB) |
|
0xFF |
0x000000000 - 0x0FFFFFFFF |
System I/O space |
|
|
0x100000000 - 0x10FFFFFFF |
SBus Slot 0 |
|
|
0x110000000 - 0x11FFFFFFF |
SBus Slot 1 |
|
|
0x120000000 - 0x12FFFFFFF |
SBus Slot 2 |
|
|
0x130000000 - 0x13FFFFFFF |
SBus Slot 3 |
|
|
0x1D0000000 - 0x1DFFFFFFF |
SBus Slot D |
|
|
0x1E0000000 - 0x1EFFFFFFF |
SBus Slot E |
|
|
0x1F0000000 - 0x1FFFFFFFF |
SBus Slot F |
Physical SBus Addresses
The SBus has 32 address bits, as described in the SBus Specification. Table A–2 describes how the Ultra 2 uses the address bits.
Table A–2 Ultra 2 SBus Address Bits|
Bits |
Description |
|---|---|
|
0 - 27 |
These bits are the SBus address lines used by an SBus card to address the contents of the card. |
|
28 - 31 |
Used by the CPU to select one of the SBus slots. These bits generate the SlaveSelect lines. |
This addressing scheme yields the Ultra 2 addresses shown in Table A–1. Other implementations might use a different number of address bits.
The Ultra 2 has seven SBus slots, four of which are physical. Slots 0 through 3 are available for SBus cards. Slots 4-12 are reserved. The slots are used in the following way:
-
Slots 0–3 are physical slots that have DMA-master capability.
-
Slots D, E, and F are not actual physical slots, but refer to the onboard direct memory access (DMA), SCSI, Ethernet, and audio controllers. For convenience, these are viewed as being plugged into slots D, E, and F.
Note –Some SBus slots are slave-only slots. Drivers that require DMA capability should use ddi_slaveonly(9F) to determine if their device is in a DMA-capable slot. For an example of this function, see The attach() Entry Point.
SBus Hardware Configuration Files
Hardware configuration files are normally unnecessary for SBus devices. However, on some occasions, drivers for SBus devices need to use hardware configuration files to augment the information provided by the SBus card. See driver.conf(4) and sbus(4) for further details.
Device Issues
This section describes issues with special devices.
Timing-Critical Sections
While most driver operations can be performed without mechanisms for synchronization and protection beyond those provided by the locking primitives, some devices require that a sequence of events occur in order without interruption. In conjunction with the locking primitives, the function ddi_enter_critical(9F) asks the system to guarantee, to the best of its ability, that the current thread will neither be preempted nor interrupted. This stays in effect until a closing call to ddi_exit_critical(9F) is made. See the ddi_enter_critical(9F) man pages for details.
Delays
Many chips specify that they can be accessed only at specified intervals. For example, the Zilog Z8530 SCC has a “write recovery time” of 1.6 microseconds. This means that a delay must be enforced with drv_usecwait(9F) when writing characters with an 8530. In some instances, the specifications do not make explicit what delays are needed, so they must be determined empirically.
Be careful not to compound delays for parts of devices that might exist in large numbers, for example thousands of SCSI disk drives.
Internal Sequencing Logic
Devices with internal sequencing logic map multiple internal registers to the same external address. There are various kinds of internal sequencing logic:
-
The Intel 8251A and the Signetics 2651 alternate the same external register between two internal mode registers. Writing to the first internal register is accomplished by writing to the external register. This write, however, has the side effect of setting up the sequencing logic in the chip so that the next read/write operation refers to the second internal register.
-
The NEC PD7201 PCC has multiple internal data registers. To write a byte into a particular register, two steps must be performed. The first step is to write into register zero the number of the register into which the following byte of data will go. The data is then written to the specified data register. The sequencing logic automatically sets up the chip so that the next byte sent will go into data register zero.
-
The AMD 9513 timer has a data pointer register that points at the data register into which a data byte will go. When sending a byte to the data register, the pointer is incremented. The current value of the pointer register cannot be read.
Interrupt Issues
The following are some common interrupt-related issues:
-
A controller interrupt does not necessarily indicate that both the controller and one of its slave devices are ready. For some controllers, an interrupt can indicate that either the controller is ready or one of its devices is ready, but not both.
-
Not all devices power up with interrupts disabled and can begin interrupting at any time.
-
Some devices do not provide a way to determine that the board has generated an interrupt.
-
Not all interrupting boards shut off interrupts when told to do so or after a bus reset.
PROM on SPARC Machines
Some platforms have a PROM monitor that provides support for debugging a device without an operating system. This section describes how to use the PROM on SPARC machines to map device registers so that they can be accessed. Usually, the device can be exercised enough with PROM commands to determine if the device is working correctly.
See the boot(1M) man page for a description of the IA boot subsystem.
The PROM has several purposes, including:
-
Bringing the machine up from power on, or from a hard reset PROM reset command
-
Providing an interactive tool for examining and setting memory, device registers, and memory mappings
-
Booting the Solaris system or the kernel debugger kadb(1M)
Simply powering up the computer and attempting to use its PROM to examine device registers can fail. While the device might be correctly installed, those mappings are specific to the Solaris operating environment and do not become active until the Solaris kernel is booted. Upon power up, the PROM maps only essential system devices, such as the keyboard.
-
Taking a system crash dump using the sync command
Open Boot PROM 3
For complete documentation on the Open Boot PROM, see the Open Boot PROM Toolkit User's Guide and the monitor(1M) man page. The examples in this section refer to a Sun-4u architecture; other architectures might require different commands to perform actions.
Note –
The Open Boot PROM is currently used on Sun machines with an SBus or UPA/PCI. The Open Boot PROM uses an “ok” prompt. On older machines, you might have to type `n' to get the “ok” prompt.
If the PROM is in secure mode (the security-mode parameter is not set to none), the PROM password might be required (set in the security-password parameter).
The printenv command displays all parameters and their values.
Help is available with the help command.
EMACS-style command-line history is available. Use Control-N (next) and Control-P (previous) to traverse the history list.
Forth Commands
The Open Boot PROM uses the Forth programming language. This is a stack-based language; arguments must be pushed on the stack before running the correct command (called a word), and the result is left on the stack.
To place a number on the stack, type its value.
ok 57 ok 68
To add the two top values on the stack, use the + operator.
ok +
The result remains on the stack. The stack is shown with the .s word.
ok .s bf
The default base is hexadecimal. The hex and decimal words can be used to switch bases.
ok decimal ok .s 191
See the Forth User's Guide for more information.
Walking the PROMs Device Tree
The commands pwd, cd, and ls walk the PROM device tree to get to the device. The cd command must be used to establish a position in the tree before pwd will work. This example is from an Ultra 1 workstation with a cgsix frame buffer on an SBus.
ok cd /
To see the devices attached to the current node in the tree, use ls.
ok ls f006a064 SUNW,UltraSPARC@0,0 f00598b0 sbus@1f,0 f00592dc counter-timer@1f,3c00 f004eec8 virtual-memory f004e8e8 memory@0,0 f002ca28 aliases f002c9b8 options f002c880 openprom f002c814 chosen f002c7a4 packages
The full node name can be used:
ok cd sbus@1f,0 ok ls f006a4e4 cgsix@2,0 f0068194 SUNW,bpp@e,c800000 f0065370 ledma@e,8400010 f006120c espdma@e,8400000 f005a448 SUNW,pll@f,1304000 f005a394 sc@f,1300000 f005a24c zs@f,1000000 f005a174 zs@f,1100000 f005a0c0 eeprom@f,1200000 f0059f8c SUNW,fdtwo@f,1400000 f0059ec4 flashprom@f,0 f0059e34 auxio@f,1900000 f0059d28 SUNW,CS4231@d,c000000
Rather than using the full node name in the previous example, you could have used an abbreviation. The abbreviated command-line entry looks like this:
ok cd sbus
The name is actually device@slot,offset (for SBus devices). The cgsix device is in slot 2 and starts at offset 0. If an SBus device is displayed in this tree, the device has been recognized by the PROM.
The .properties command displays the PROM properties of a device. These can be examined to determine which properties the device exports. (This is useful later to ensure that the driver is looking for the correct hardware properties.) These are the same properties that can be retrieved with ddi_getprop(9F).
ok cd cgsix ok .properties character-set ISO8859-1 intr 00000005 00000000 interrupts 00000005 reg 00000002 00000000 01000000 dblbuf 00 00 00 00 vmsize 00 00 00 01 ...
The reg property defines an array of register description structures containing the following fields:
uint_t bustype; /* cookie for related bus type*/ uint_t addr; /* address of reg relative to bus */ uint_t size; /* size of this register set */
For the cgsix example, the address is 0.
Mapping the Device
To test the device, it must be mapped into memory. The PROM can then be used to verify proper operation of the device by using data-transfer commands to transfer bytes, words, and long words. If the device can be operated from the PROM, even in a limited way, the driver should also be able to operate the device.
To set up the device for initial testing, perform the following steps:
-
Determine the SBus slot number the device is in.
In this example, the cgsix device is located in slot 2.
-
Determine the offset within the physical address space used by the device.
The offset used is specific to the device. In the cgsix example, the video memory happens to start at an offset of 0x800000.
-
Use the select-dev word to select the Sbus device and the map-in word to map the device in.
The select-dev word takes a string of the device path as its argument. The map-in word takes an offset, a slot number, and a size as arguments to map. Like the offset, the size of the byte transfer is specific to the device. In the cgsix example, the size is set to 0x100000 bytes.
In the following code example, the sbus path is displayed as an argument to the select-dev word, and the offset, slot number, and size values for the frame buffer are displayed as arguments to the map-in word. Notice that there should be a space between the opening quote and / in the select-dev argument. The virtual address to use remains on top of the stack. The stack is shown using the .s word. It can be assigned a name with the constant operation.
ok "/sbus@1f,0" select-dev ok 800000 2 100000 map-in ok .s ffe98000 ok constant fb
Reading and Writing
The PROM provides a variety of 8-bit, 16-bit, and 32-bit operations. In general, a c (character) prefix indicates an 8-bit (one byte) operation; a w (word) prefix indicates a 16-bit (two byte) operation; and an L (longword) prefix indicates a 32-bit (four byte) operation.
A suffix of ! indicates a write operation. The write operation takes the first two items off the stack; the first item is the address, and the second item is the value.
ok 55 ffe98000 c!
A suffix of @ indicates a read operation. The read operation takes one argument (the address) off the stack.
ok ffe98000 c@ ok .s 55
A suffix of ? is used to display the value, without affecting the stack.
ok ffe98000 c? 55
Be careful when trying to query the device. If the mappings are not set up correctly, trying to read or write could cause errors. Special words are provided to handle these cases. cprobe, wprobe, and lprobe, for example, read from the given address but return zero if the location does not respond, or nonzero if it does.
ok fffa4000 c@ Data Access Error ok fffa4000 cprobe ok .s0 ok ffe98000 cprobe ok .s 0 ffffffffffffffff
A region of memory can be shown with the dump word. This takes an address and a length, and displays the contents of the memory region in bytes.
In the following example, the fill word is used to fill video memory with a pattern. fill takes the address, the number of bytes to fill, and the byte to use (there is also a wfill and an Lfill for words and longwords). This causes the cgsix to display simple patterns based on the byte passed.
ok “ /sbus” select-dev ok 800000 2 100000 map-in ok constant fb ok fb 10000 ff fill ok fb 20000 0 fill ok fb 18000 55 fill ok fb 15000 3 fill ok fb 10000 5 fillok fb 5000 f9 fill
- © 2010, Oracle Corporation and/or its affiliates