Writing Device Drivers

Appendix A Hardware Overview

This appendix discusses general issues about hardware capable of supporting the Solaris 9 operating environment. This includes issues related to the processor, bus architectures, and memory models supported by the Solaris 9 operating environment, various device issues, and the PROM used in Sun platforms.

Note –

The information presented here is for informational purposes only and might be of help during driver debugging. However, the DDI/DKI interfaces hide many of these implementation details from device drivers.

This appendix provides information on the following subjects:

SPARC Processor Issues

This section describes a number of SPARC processor-specific topics including data alignment, byte ordering, register windows, and availability of floating-point instructions. For information on IA processor-specific topics, see IA Processor Issues.

Caution – Caution –

Drivers should never perform floating-point operations, as they are not supported in the kernel.

SPARC Data Alignment

All quantities must be aligned on their natural boundaries. Using standard C data types:

SPARC Structure Member Alignment

Because of the data alignment restrictions imposed by the SPARC processor, C structures also have alignment requirements. Structure alignment requirements are imposed by the most strictly-aligned structure component. For example, a structure containing only characters has no alignment restrictions, while a structure containing a long long member must be constructed to guarantee that this member falls on a 64-bit boundary.

SPARC Byte Ordering

The SPARC processor uses big-endian byte ordering; in other words, the most significant byte (MSB) of an integer is stored at the lowest address of the integer.

Diagram shows how bytes are ordered in big-endian architectures; that is, byte 0 is the most significant byte.

SPARC Register Windows

SPARC processors use register windows. Each register window consists of 8 in registers, 8 local registers, and 8 out registers (which are the in registers of the next window). There are also 8 global registers. The number of register windows ranges from 2 to 32, depending on the processor implementation.

Because drivers are normally written in C, the compiler usually hides the fact that register windows are used. However, you might have to use them when debugging the driver.

SPARC Multiply and Divide Instructions

The Version 7 SPARC processors do not have multiply or divide instructions. These instructions are emulated in software and should be avoided. Because a driver cannot determine whether it is running on a Version 7, Version 8, or Version 9 processor, intensive integer multiplication and division should be avoided if possible. Instead, use bitwise left and right shifts to multiply and divide by powers of two.

The SPARC Architecture Manual, Version 9, contains more specific information on the SPARC CPU. The SPARC Compliance Definition, Version 2.4, contains details of the SPARC V9 application binary interface (ABI). It describes the 32–bit SPARC V8 ABI and the 64–bit SPARC V9 ABI. You can obtain this document from SPARC International at http://www.sparc.com.

IA Processor Issues

There are no alignment restrictions on data types. However, extra memory cycles might be required for the IA processor to properly handle misaligned data transfers.

Caution – Caution –

Drivers should not perform floating-point operations, as they are not supported in the kernel.

IA Byte Ordering

The IA processor uses little-endian byte ordering. The least significant byte (LSB) of an integer is stored at the lowest address of the integer.

Diagram shows how bytes are ordered in little-endian architectures; that is, byte 0 is the least significant byte.

IA Architecture Manuals

Intel Corporation publishes a number of books on the IA family of processors. See http://www.intel.com.


To achieve the goal of multiple-platform, multiple-instruction-set architecture portability, host bus dependencies were removed from the drivers. The first dependency issue to be addressed was the endianness (or byte ordering) of the processor. For example, the IA processor family is little-endian while the SPARC architecture is big-endian.

Bus architectures display the same endianness types as processors. The PCI local bus, for example, is little-endian, the SBus is big-endian, the ISA bus is little-endian, and so on.

To maintain portability between processors and buses, DDI-compliant drivers must be endian neutral. Although drivers could conceivably manage their endianness by runtime checks or by preprocessor directives like #ifdef _LITTLE_ENDIAN or _BIG_ENDIAN statements in the source code, long-term maintenance would be troublesome. In some cases, the DDI framework performs the byte swapping using a software approach. In other cases, where byte swapping can be done by hardware (as in memory management unit (MMU) page-level swapping or by special machine instructions), the DDI framework will take advantage of the hardware features to improve performance.

Figure A–1 Byte Ordering Host Bus Dependency

Diagram shows byte swapping to reverse endianness.

Along with being endian-neutral, portable drivers must also be independent from data ordering of the processor. Under most circumstances, data must be transferred in the sequence instructed by the driver. However, sometimes data can be merged, batched, or reordered to streamline the data transfer, as illustrated in Figure A–2. For example, data merging can be applied to accelerate graphics display on frame buffers. Drivers have the option to advise the DDI framework to use other optimal data transfer mechanisms during the transfer.

Figure A–2 Data Ordering Host Bus Dependency

Diagram shows reordering of bytes by CPU.

Store Buffers

To improve performance, the CPU uses internal store buffers to temporarily store data. This can affect the synchronization of device I/O operations. Therefore, the driver needs to take explicit steps to make sure that writes to registers are completed at the proper time.

For example, when access to device space (such as registers or a frame buffer) is synchronized by a lock, the driver needs to check that the store to the device space has actually completed before releasing the lock. Releasing the lock does not guarantee the flushing of I/O buffers.

To give another example, when acknowledging an interrupt, the driver usually sets or clears a bit in a device control register. The driver must ensure that the write to the control register has reached the device before the interrupt handler returns. Similarly, if the device requires a delay (the driver busy-waits) after writing a command to the control register, the driver must ensure that the write has reached the device before delaying.

If the device registers can be read without undesirable side effects, verification of a write can be as simple as reading the register immediately after writing to it. If that particular register cannot be read without undesirable side effects, another device register in the same register set can be used.

System Memory Model

The system memory model defines the semantics of memory operations such as load and store and specifies how the order in which these operations are issued by a processor is related to the order in which they reach memory. The memory model applies to both uniprocessors and shared-memory multiprocessors. Two memory models are supported: total store ordering (TSO) and partial store ordering (PSO).

Total Store Ordering (TSO)

TSO guarantees that the sequence in which store, FLUSH, and atomic load-store instructions appear in memory for a given processor is identical to the sequence in which they were issued by the processor.

Both IA and SPARC processors support TSO.

Partial Store Ordering (PSO)

PSO does not guarantee that the sequence in which store, FLUSH, and atomic load-store instructions appear in memory for a given processor is identical to the sequence in which they were issued by the processor. The processor can reorder the stores so that the sequence of stores to memory is not the same as the sequence of stores issued by the CPU.

SPARC processors support PSO; IA processors do not.

For SPARC processors, conformance between issuing order and memory order is provided by the system framework using the STBAR instruction: if two of the above instructions are separated by an STBAR in the issuing order of a processor, or if they reference the same location, the memory order of the two instructions is the same as the issuing order. Note that enforcement of strong data ordering in DDI-compliant drivers is provided by the ddi_regs_map_setup(9F) interface. Compliant drivers cannot use the STBAR instruction directly.

See the SPARC Architecture Manual, Version 9, for more details on the SPARC memory model.

Bus Architectures

This section describes device identification, device addressing, and interrupts.

Device Identification

Device identification is the process of determining which devices are present in the system. Some devices are self-identifying—the device itself provides information to the system so that it can identify the device driver that needs to be used. SBus and PCI local bus devices are examples of self-identifying devices. On SBus, the information is usually derived from a small Forth program stored in the FCode PROM on the device. Most PCI devices provide a configuration space containing device configuration information. See the sbus(4) and pci(4) man pages for more information.

All modern bus architectures require devices to be self-identifying.

Supported Interrupt Types

Solaris supports both polling and vectored interrupts. The Solaris 9 DDI/DKI interrupt model is the same for both. See Chapter 7, Interrupt Handlers for more information about interrupt handling.

Bus Specifics

This section covers addressing and device configuration issues specific to the buses that Solaris supports.

PCI Local Bus

The PCI local bus is a high-performance bus designed for high-speed data transfer. The PCI bus resides on the system board and is normally used as an interconnect mechanism between highly integrated peripheral components, peripheral add-on boards, and host processor or memory systems. The host processor, main memory, and the PCI bus itself are connected through a PCI host bridge, as shown in Figure A–3.

A tree structure of interconnected I/O buses is supported through a series of PCI bus bridges. Subordinate PCI bus bridges can be extended underneath the PCI host bridge to enable a single bus system to be expanded into a complex system with multiple secondary buses. PCI devices can be connected to one or more of these secondary buses. In addition, other bus bridges, such as SCSI or USB, can be connected.

Every PCI device has a unique vendor ID and device ID. Multiple devices of the same kind are further identified by their unique device numbers on the bus where they reside.

Figure A–3 Machine Block Diagram

Diagram shows how a PCI host bridge connects the CPU and main memory to a PCI bus.

The PCI host bridge provides an interconnect between the processor and peripheral components. Through the PCI host bridge, the processor can directly access main memory independent of other PCI bus masters. For example, while the CPU is fetching data from the cache controller in the host bridge, other PCI devices can also access the system memory through the host bridge. The advantage of this architecture lies in its separation of the I/O bus from the processor's host bus.

The PCI host bridge also provides data access mappings between the CPU and peripheral I/O devices. It maps every peripheral device to the host address domain so that the processor can access the device through programmed I/O. On the local bus side, the PCI host bridge maps the system memory to the PCI address domain so that the PCI device can access the host memory as a bus master. Figure A–3 shows the two address domains.

PCI Address Domain

The PCI address domain consists of three distinct address spaces: configuration, memory, and I/O space.

PCI Configuration Address Space

Configuration space is defined geographically; in other words, the location of a peripheral device is determined by its physical location within an interconnected tree of PCI bus bridges. A device is located by its bus number and device (slot) number. Each peripheral device contains a set of well-defined configuration registers in its PCI configuration space. The registers are used not only to identify devices but also to supply device configuration information to the configuration framework. For example, base address registers in the device configuration space must be mapped before a device can respond to data access.

The method for generating configuration cycles is host dependent. In IA machines, special I/O ports are used. On other platforms, the PCI configuration space can be memory-mapped to certain address locations corresponding to the PCI host bridge in the host address domain. When a device configuration register is accessed by the processor, the request is routed to the PCI host bridge. The bridge then translates the access into proper configuration cycles on the bus.

PCI Configuration Base Address Registers

The PCI configuration space consists of up to six 32-bit base address registers for each device. These registers provide both size and data type information. System firmware assigns base addresses in the PCI address domain to these registers.

Each addressable region can be either memory or I/O space. The value contained in bit 0 of the base address register identifies the type. A value of 0 in bit 0 indicates a memory space and a value of 1 indicates an I/O space. Figure A–4 shows two base address registers: one for memory; the other for I/O types.

Figure A–4 Base Address Registers for Memory and I/O

Diagram shows how bit 0 in a base address indicates a memory or I/O space.

PCI Memory Address Space

PCI supports both 32-bit and 64-bit addresses for memory space. System firmware assigns regions of memory space in the PCI address domain to PCI peripherals. The base address of a region is stored in the base address register of the device's PCI configuration space. The size of each region must be a power of two, and the assigned base address must be aligned on a boundary equal to the size of the region. Device addresses in memory space are memory-mapped into the host address domain so that data access to any device can be performed by the processor's native load or store instructions.

PCI I/O Address Space

PCI supports 32-bit I/O space. I/O space can be accessed differently on different platforms. Processors with special I/O instructions, like the Intel processor family, access the I/O space with in and out instructions. Machines without special I/O instructions will map to the address locations corresponding to the PCI host bridge in the host address domain. When the processor accesses the memory-mapped addresses, an I/O request will be sent to the PCI host bridge, which then translates the addresses into I/O cycles and puts them on the PCI bus. Memory-mapped I/O is performed by the native load/store instructions of the processor.

PCI Hardware Configuration Files

Hardware configuration files should be unnecessary for PCI local bus devices. However, on some occasions drivers for PCI devices need to use hardware configuration files to augment the driver private information. See the driver.conf(4) and pci(4) man pages for further details.


Typical SBus systems consist of a motherboard (containing the CPU and SBus interface logic), a number of SBus devices on the motherboard itself, and a number of SBus expansion slots. An SBus can also be connected to other types of buses through an appropriate bus bridge.

The SBus is geographically addressed; each SBus slot exists at a fixed physical address in the system. An SBus card has a different address, depending on which slot it is plugged into. Moving an SBus device to a new slot causes the system to treat it as a new device.

The SBus uses polling interrupts. When an SBus device interrupts, the system only knows which of several devices might have issued the interrupt. The system interrupt handler must ask the driver for each device whether it is responsible for the interrupt.

SBus Physical Address Space

Table A–1 shows the physical address space layout of the Sun Ultra 2 computer. A physical address on the Ultra 2 consists of 41 bits. The 41-bit physical address space is further broken down into multiple 33-bit address spaces identified by PA(40:33).

Table A–1 Device Physical Space in the Ultra 2


33-bit Space 



0x000000000 - 0x07FFFFFFF

2 Gbytes Main memory 

0x80 – 0xDF 

Reserved on Ultra 2

Reserved on Ultra 2 


Processor 0

Processor 0 


Processor 1

Processor 1 

0xE2 – 0xFD 

Reserved on Ultra 2

Reserved on Ultra 2 


0x000000000 - 0x1FFFFFFFF

UPA Slave (FFB) 


0x000000000 - 0x0FFFFFFFF

System I/O space 


0x100000000 - 0x10FFFFFFF

SBus Slot 0 


0x110000000 - 0x11FFFFFFF

SBus Slot 1 


0x120000000 - 0x12FFFFFFF

SBus Slot 2 


0x130000000 - 0x13FFFFFFF

SBus Slot 3 


0x1D0000000 - 0x1DFFFFFFF

SBus Slot D 


0x1E0000000 - 0x1EFFFFFFF

SBus Slot E 


0x1F0000000 - 0x1FFFFFFFF

SBus Slot F 

Physical SBus Addresses

The SBus has 32 address bits, as described in the SBus Specification. Table A–2 describes how the Ultra 2 uses the address bits.

Table A–2 Ultra 2 SBus Address Bits



0 - 27 

These bits are the SBus address lines used by an SBus card to address the contents of the card. 

28 - 31 

Used by the CPU to select one of the SBus slots. These bits generate the SlaveSelect lines. 

This addressing scheme yields the Ultra 2 addresses shown in Table A–1. Other implementations might use a different number of address bits.

The Ultra 2 has seven SBus slots, four of which are physical. Slots 0 through 3 are available for SBus cards. Slots 4-12 are reserved. The slots are used in the following way:

SBus Hardware Configuration Files

Hardware configuration files are normally unnecessary for SBus devices. However, on some occasions, drivers for SBus devices need to use hardware configuration files to augment the information provided by the SBus card. See driver.conf(4) and sbus(4) for further details.

Device Issues

This section describes issues with special devices.

Timing-Critical Sections

While most driver operations can be performed without mechanisms for synchronization and protection beyond those provided by the locking primitives, some devices require that a sequence of events occur in order without interruption. In conjunction with the locking primitives, the function ddi_enter_critical(9F) asks the system to guarantee, to the best of its ability, that the current thread will neither be preempted nor interrupted. This stays in effect until a closing call to ddi_exit_critical(9F) is made. See the ddi_enter_critical(9F) man pages for details.


Many chips specify that they can be accessed only at specified intervals. For example, the Zilog Z8530 SCC has a “write recovery time” of 1.6 microseconds. This means that a delay must be enforced with drv_usecwait(9F) when writing characters with an 8530. In some instances, the specifications do not make explicit what delays are needed, so they must be determined empirically.

Be careful not to compound delays for parts of devices that might exist in large numbers, for example thousands of SCSI disk drives.

Internal Sequencing Logic

Devices with internal sequencing logic map multiple internal registers to the same external address. There are various kinds of internal sequencing logic:

Interrupt Issues

The following are some common interrupt-related issues:

PROM on SPARC Machines

Some platforms have a PROM monitor that provides support for debugging a device without an operating system. This section describes how to use the PROM on SPARC machines to map device registers so that they can be accessed. Usually, the device can be exercised enough with PROM commands to determine if the device is working correctly.

See the boot(1M) man page for a description of the IA boot subsystem.

The PROM has several purposes, including:

Open Boot PROM 3

For complete documentation on the Open Boot PROM, see the Open Boot PROM Toolkit User's Guide and the monitor(1M) man page. The examples in this section refer to a Sun-4u architecture; other architectures might require different commands to perform actions.

Note –

The Open Boot PROM is currently used on Sun machines with an SBus or UPA/PCI. The Open Boot PROM uses an “ok” prompt. On older machines, you might have to type `n' to get the “ok” prompt.

If the PROM is in secure mode (the security-mode parameter is not set to none), the PROM password might be required (set in the security-password parameter).

The printenv command displays all parameters and their values.

Help is available with the help command.

EMACS-style command-line history is available. Use Control-N (next) and Control-P (previous) to traverse the history list.

Forth Commands

The Open Boot PROM uses the Forth programming language. This is a stack-based language; arguments must be pushed on the stack before running the correct command (called a word), and the result is left on the stack.

To place a number on the stack, type its value.

ok 57
ok 68

To add the two top values on the stack, use the + operator.

ok +

The result remains on the stack. The stack is shown with the .s word.

ok .s

The default base is hexadecimal. The hex and decimal words can be used to switch bases.

ok decimal
ok .s

See the Forth User's Guide for more information.

Walking the PROMs Device Tree

The commands pwd, cd, and ls walk the PROM device tree to get to the device. The cd command must be used to establish a position in the tree before pwd will work. This example is from an Ultra 1 workstation with a cgsix frame buffer on an SBus.

ok cd /

To see the devices attached to the current node in the tree, use ls.

ok ls
f006a064 SUNW,UltraSPARC@0,0
f00598b0 sbus@1f,0
f00592dc counter-timer@1f,3c00
f004eec8 virtual-memory
f004e8e8 memory@0,0
f002ca28 aliases
f002c9b8 options
f002c880 openprom
f002c814 chosen
f002c7a4 packages

The full node name can be used:

ok cd sbus@1f,0
ok ls
f006a4e4 cgsix@2,0
f0068194 SUNW,bpp@e,c800000
f0065370 ledma@e,8400010
f006120c espdma@e,8400000
f005a448 SUNW,pll@f,1304000
f005a394 sc@f,1300000
f005a24c zs@f,1000000
f005a174 zs@f,1100000
f005a0c0 eeprom@f,1200000
f0059f8c SUNW,fdtwo@f,1400000
f0059ec4 flashprom@f,0
f0059e34 auxio@f,1900000
f0059d28 SUNW,CS4231@d,c000000

Rather than using the full node name in the previous example, you could have used an abbreviation. The abbreviated command-line entry looks like this:

ok cd sbus

The name is actually device@slot,offset (for SBus devices). The cgsix device is in slot 2 and starts at offset 0. If an SBus device is displayed in this tree, the device has been recognized by the PROM.

The .properties command displays the PROM properties of a device. These can be examined to determine which properties the device exports. (This is useful later to ensure that the driver is looking for the correct hardware properties.) These are the same properties that can be retrieved with ddi_getprop(9F).

ok cd cgsix
ok .properties
character-set            ISO8859-1
intr                     00000005 00000000
interrupts               00000005
reg                      00000002 00000000 01000000
dblbuf                   00 00 00 00
vmsize                   00 00 00 01

The reg property defines an array of register description structures containing the following fields:

uint_t        bustype;       /* cookie for related bus type*/
uint_t        addr;          /* address of reg relative to bus */
uint_t        size;          /* size of this register set */

For the cgsix example, the address is 0.

Mapping the Device

To test the device, it must be mapped into memory. The PROM can then be used to verify proper operation of the device by using data-transfer commands to transfer bytes, words, and long words. If the device can be operated from the PROM, even in a limited way, the driver should also be able to operate the device.

To set up the device for initial testing, perform the following steps:

  1. Determine the SBus slot number the device is in.

    In this example, the cgsix device is located in slot 2.

  2. Determine the offset within the physical address space used by the device.

    The offset used is specific to the device. In the cgsix example, the video memory happens to start at an offset of 0x800000.

  3. Use the select-dev word to select the Sbus device and the map-in word to map the device in.

    The select-dev word takes a string of the device path as its argument. The map-in word takes an offset, a slot number, and a size as arguments to map. Like the offset, the size of the byte transfer is specific to the device. In the cgsix example, the size is set to 0x100000 bytes.

    In the following code example, the sbus path is displayed as an argument to the select-dev word, and the offset, slot number, and size values for the frame buffer are displayed as arguments to the map-in word. Notice that there should be a space between the opening quote and / in the select-dev argument. The virtual address to use remains on top of the stack. The stack is shown using the .s word. It can be assigned a name with the constant operation.

    ok "/sbus@1f,0" select-dev
    ok 800000 2 100000 map-in
    ok .s
    ok constant fb

Reading and Writing

The PROM provides a variety of 8-bit, 16-bit, and 32-bit operations. In general, a c (character) prefix indicates an 8-bit (one byte) operation; a w (word) prefix indicates a 16-bit (two byte) operation; and an L (longword) prefix indicates a 32-bit (four byte) operation.

A suffix of ! indicates a write operation. The write operation takes the first two items off the stack; the first item is the address, and the second item is the value.

ok 55 ffe98000 c!

A suffix of @ indicates a read operation. The read operation takes one argument (the address) off the stack.

ok ffe98000 c@
ok .s

A suffix of ? is used to display the value, without affecting the stack.

ok ffe98000 c?

Be careful when trying to query the device. If the mappings are not set up correctly, trying to read or write could cause errors. Special words are provided to handle these cases. cprobe, wprobe, and lprobe, for example, read from the given address but return zero if the location does not respond, or nonzero if it does.

ok fffa4000 c@
Data Access Error

ok fffa4000 cprobe
ok .s0

ok ffe98000 cprobe
ok .s
0 ffffffffffffffff

A region of memory can be shown with the dump word. This takes an address and a length, and displays the contents of the memory region in bytes.

In the following example, the fill word is used to fill video memory with a pattern. fill takes the address, the number of bytes to fill, and the byte to use (there is also a wfill and an Lfill for words and longwords). This causes the cgsix to display simple patterns based on the byte passed.

ok “ /sbus” select-dev
ok 800000 2 100000 map-in
ok constant fb
ok fb 10000 ff fill
ok fb 20000 0 fill
ok fb 18000 55 fill
ok fb 15000 3 fill
ok fb 10000 5 fillok fb 5000 f9 fill