JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Writing Device Drivers     Oracle Solaris 11.1 Information Library
search filter icon
search icon

Document Information

Preface

Part I Designing Device Drivers for the Oracle Solaris Platform

1.  Overview of Oracle Solaris Device Drivers

2.  Oracle Solaris Kernel and Device Tree

3.  Multithreading

4.  Properties

5.  Managing Events and Queueing Tasks

6.  Driver Autoconfiguration

7.  Device Access: Programmed I/O

8.  Interrupt Handlers

Interrupt Handler Overview

Device Interrupts

High-Level Interrupts

Legacy Interrupts

Standard and Extended Message-Signaled Interrupts

MSI Interrupts

MSI-X Interrupts

Software Interrupts

DDI Interrupt Functions

Interrupt Capability Functions

Interrupt Initialization and Destruction Functions

Priority Management Functions

Soft Interrupt Functions

Interrupt Function Examples

Registering Interrupts

Registering Legacy Interrupts

Registering MSI Interrupts

Interrupt Resource Management

The Interrupt Resource Management Feature

Callback Interfaces

Register a Callback Handler Function

Unregister a Callback Handler Function

Callback Handler Function

Interrupt Request Interfaces

Allocate an Interrupt

Modify Number of Interrupt Vectors Requested

Interrupt Usage and Flexibility

Example Implementation of Interrupt Resource Management

Interrupt Handler Functionality

Handling High-Level Interrupts

High-Level Mutexes

High-Level Interrupt Handling Example

9.  Direct Memory Access (DMA)

10.  Mapping Device and Kernel Memory

11.  Device Context Management

12.  Power Management

13.  Hardening Oracle Solaris Drivers

14.  Layered Driver Interface (LDI)

Part II Designing Specific Kinds of Device Drivers

15.  Drivers for Character Devices

16.  Drivers for Block Devices

17.  SCSI Target Drivers

18.  SCSI Host Bus Adapter Drivers

19.  Drivers for Network Devices

20.  USB Drivers

21.  SR-IOV Drivers

Part III Building a Device Driver

22.  Compiling, Loading, Packaging, and Testing Drivers

23.  Debugging, Testing, and Tuning Device Drivers

24.  Recommended Coding Practices

Part IV Appendixes

A.  Hardware Overview

B.  Summary of Oracle Solaris DDI/DKI Services

C.  Making a Device Driver 64-Bit Ready

D.  Console Frame Buffer Drivers

E.  pci.conf File

Index

Interrupt Resource Management

This section discusses how a driver for a device that can generate many different interruptible conditions can utilize the Interrupt Resource Management feature to optimize its allocation of interrupt vectors.

The Interrupt Resource Management Feature

The Interrupt Resource Management feature can enable a device driver to use more interrupt resources by dynamically managing the driver's interrupt configuration. When the Interrupt Resource Management feature is not used, configuration of interrupt handling usually only occurs in a driver's attach(9E) routine. The Interrupt Resource Management feature monitors the system for changes, recalculates the number of interrupt vectors to grant to each device in response to those changes, and notifies each affected participating driver of the driver's new allocation of interrupt vectors. A participating driver is a driver that has registered a callback handler as described in Callback Interfaces. Changes that can cause interrupt vector reallocation include adding or removing devices, or an explicit request as described in Modify Number of Interrupt Vectors Requested.

The Interrupt Resource Management feature is not available on every Oracle Solaris platform. The feature is only available to PCIe devices that utilize MSI-X interrupts.


Note - A driver that utilizes the Interrupt Resource Management feature must be able to adapt correctly when the feature is not available.


When the Interrupt Resource Management feature is available, it can enable a driver to gain access to more interrupt vectors than the driver might otherwise be allocated. A driver might process interrupt conditions more efficiently when utilizing a larger number of interrupt vectors.

The Interrupt Resource Management feature dynamically adjusts the number of interrupt vectors granted to each participating driver depending upon the following constraints:

The number of interrupt vectors made available to a device at any given time can vary:

A driver must provide the following support to use the Interrupt Resource Management feature:

Callback Interfaces

A driver must use the following interfaces to register its callback support.

Table 8-1 Callback Support Interfaces

Interface
Data Structures
Description
ddi_cb_register()
ddi_cb_flags_t, ddi_cb_handle_t
Register a callback handler function to receive specific types of actions.
ddi_cb_unregister()
ddi_cb_handle_t
Unregister a callback handler function.
(*ddi_cb_func_t)()
ddi_cb_action_t
Receive callback actions and specific arguments relevant to each action to be processed.

Register a Callback Handler Function

Use the ddi_cb_register(9F) function to register a callback handler function for a driver.

int
ddi_cb_register (dev_info_t *dip, ddi_cb_flags_t cbflags,
                 ddi_cb_func_t cbfunc, void *arg1, void *arg2,
                 ddi_cb_handle_t *ret_hdlp);

The driver can register only one callback function. This one callback function is used to handle all individual callback actions. The cbflags parameter determines which types of actions should be received by the driver when they occur. The cbfunc() routine is called whenever a relevant action should be processed by the driver. The driver specifies two private arguments (arg1 and arg2) to send to itself during each execution of its cbfunc() routine.

The cbflags() parameter is an enumerated type that specifies which actions the driver supports.

typedef enum {
        DDI_CB_FLAG_INTR
} ddi_cb_flags_t;

To register support for Interrupt Resource Management actions, a driver must register a handler and include the DDI_CB_FLAG_INTR flag. When the callback handler is successfully registered, an opaque handle is returned through the ret_hdlp parameter. When the driver is finished with the callback handler, the driver can use the ret_hdlp parameter to unregister the callback handler.

Register the callback handler in the driver's attach(9F) entry point. Save the opaque handle in the driver's soft state. Unregister the callback handler in the driver's detach(9F) entry point.

Unregister a Callback Handler Function

Use the ddi_cb_unregister(9F) function to unregister a callback handler function for a driver.

int
ddi_cb_unregister (ddi_cb_handle_t hdl);

Make this call in the driver's detach(9F) entry point. After this call, the driver no longer receives callback actions.

The driver also loses any additional support from the system that it gained from having a registered callback handling function. For example, some interrupt vectors previously made available to the driver are immediately taken back when the driver unregisters its callback handling function. Before returning successfully, the ddi_cb_unregister() function notifies the driver of any final actions that result from losing support from the system.

Callback Handler Function

Use the registered callback handling function to receive callback actions and receive arguments that are specific to each action to be processed.

typedef int (*ddi_cb_func_t)(dev_info_t *dip, ddi_cb_action_t cbaction,
                             void *cbarg, void *arg1, void *arg2);

The cbaction parameter specifies what action the driver is receiving a callback to process.

typedef enum {
        DDI_CB_INTR_ADD,
        DDI_CB_INTR_REMOVE
} ddi_cb_action_t;

A DDI_CB_INTR_ADD action means that the driver now has more interrupts available to use. A DDI_CB_INTR_REMOVE action means that the driver now has fewer interrupts available to use. Cast the cbarg parameter to an int to determine the number of interrupts added or removed. The cbarg value represents the change in the number of interrupts that are available.

For example, get the change in the number of interrupts available:

count = (int)(uintptr_t)cbarg;

If the cbaction is DDI_CB_INTR_ADD, add cbarg number of interrupt vectors. If the cbaction is DDI_CB_INTR_REMOVE, free cbarg number of interrupt vectors.

See ddi_cb_register(9F) for an explanation of arg1 and arg2.

The callback handling function must be able to perform correctly for the entire time that the function is registered. The callback function cannot depend upon any data structures that might be destroyed before the callback function is successfully unregistered.

The callback handling function must return one of the following values:

Interrupt Request Interfaces

A driver must use the following interfaces to request interrupt vectors from the system.

Table 8-2 Interrupt Vector Request Interfaces

Interface
Data Structures
Description
ddi_intr_alloc()
ddi_intr_handle_t
Allocate interrupts.
ddi_intr_set_nreq()
Change number of interrupt vectors requested.

Allocate an Interrupt

Use the ddi_intr_alloc(9F) function to initially allocate interrupts.

int
ddi_intr_alloc (dev_info_t *dip, ddi_intr_handle_t *h_array, int type,
                int inum, int count, int *actualp, int behavior);

Before calling this function, the driver must allocate an empty handle array large enough to contain the number of interrupts requested. The ddi_intr_alloc() function attempts to allocate count number of interrupt handles, and initialize the array with the assigned interrupt vectors beginning at the offset specified by the inum parameter. The actualp parameter returns the actual number of interrupt vectors that were allocated.

A driver can use the ddi_intr_alloc() function in two ways:

If you are using the Interrupt Resource Management feature, call ddi_intr_alloc() one time to allocate all interrupt vectors at once. The count parameter is the total number of interrupt vectors requested by the driver. If the value in actualp is less than the value of count, then the system is not able to fulfill the request completely. The Interrupt Resource Management feature saves this request (count becomes nreq - see below) and might be able to allocate more interrupt vectors to this driver at a later time.


Note - When you use the Interrupt Resource Management feature, additional calls to ddi_intr_alloc() do not change the total number of interrupt vectors requested. Use the ddi_intr_set_nreq(9F) function to change the number of interrupt vectors requested.


Modify Number of Interrupt Vectors Requested

Use the ddi_intr_set_nreq(9F) function to change the number of interrupt vectors requested.

int
ddi_intr_set_nreq (dev_info_t *dip, int nreq);

When the Interrupt Resource Management feature is available, a driver can use the ddi_intr_set_nreq() function to dynamically adjust the total number of interrupt vectors requested. The driver might do this in response to the actual load that exists once the driver is attached.

A driver must first call ddi_intr_alloc(9F) to request an initial number of interrupt vectors. Any time after the ddi_intr_alloc()call, the driver can call ddi_intr_set_nreq() to change its request size. The specified nreq value is the driver's new total number of requested interrupt vectors. The Interrupt Resource Management feature might rebalance the number of interrupts allocated to each driver in the system in response to this new request. Whenever the Interrupt Resource Management feature rebalances the number of interrupts allocated to drivers, each affected driver receives a callback notification that more or fewer interrupt vectors are available for the driver to use.

A driver might dynamically adjust its total number of requested interrupt vectors if, for example, it uses interrupts in conjunction with specific transactions that it is processing. A storage driver might associate a DMA engine with each ongoing transaction, thus requiring interrupt vectors for that reason. A driver might make calls to ddi_intr_set_nreq() in its open(9F) and close(9F) routines to scale its interrupt usage in response to actual use of the driver.

Interrupt Usage and Flexibility

A driver for a device that supports many different interruptible conditions must be able to map those conditions to an arbitrary number of interrupt vectors. The driver cannot assume that interrupt vectors that are allocated will remain available. Some currently available interrupts might later be taken back by the system to accommodate the needs of other drivers in the system.

A driver must be able to:

In summary, the driver must be able to select a mixture of interrupt handling functions and program its hardware to generate interrupts according to need and interrupt availability. In some cases multiple interrupts might be targeted to the same vector, and the interrupt handler for that interrupt vector must determine which interrupts occurred. The performance of the device can be affected by how well the driver maps interrupts to interrupt vectors.

Example Implementation of Interrupt Resource Management

One type of device driver that is an excellent candidate for interrupt resource management is a network device driver. The network device hardware supports multiple transmit and receive channels.

The network device generates a unique interrupt condition whenever the device receives a packet on one of its receive channels or transmits a packet on one of its transmit channels. The hardware can send a specific MSI-X interrupt for each event that can occur. A table in the hardware determines which MSI-X interrupt to generate for each event.

To optimize performance, the driver requests enough interrupts from the system to give each separate interrupt its own interrupt vector. The driver makes this request when it first calls ddi_intr_alloc(9F) in its attach(9F) routine.

The driver then evaluates the actual number of interrupts it received from ddi_intr_alloc() in actualp. It might receive all the interrupts it requested, or it might receive fewer interrupts.

A separate function inside the driver uses the total number of available interrupts to calculate which MSI-X interrupts to generate for each event. This function programs the table in the hardware accordingly.

The driver should have two different interrupt handler functions.

In the example driver in this section, the function xx_setup_interrupts() uses the number of available interrupt vectors to program the hardware and calls the appropriate interrupt handler for each of those interrupt vectors. The xx_setup_interrupts() function is called in two places: after ddi_intr_alloc() is called in xx_attach(), and after interrupt vector allocations are adjusted in the xx_cbfunc() callback handler function.

int
xx_setup_interrupts(xx_state_t *statep, int navail, xx_intrs_t *xx_intrs_p);

The xx_setup_interrupts() function is called with an array of xx_intrs_t data structures.

typedef struct {
        ddi_intr_handler_t      inthandler;
        void                    *arg1;
        void                    *arg2;
} xx_intrs_t;

This xx_setup_interrupts() functionality must exist in the driver independent of whether the Interrupt Resource Management feature is available. Drivers must be able to function with fewer interrupt vectors than the number requested during attach. If the Interrupt Resource Management feature is available, you can modify the driver to dynamically adjust to a new number of available interrupt vectors.

Other functionality that the driver must provide independent of whether the Interrupt Resource Management feature is available includes the ability to quiesce the hardware and resume the hardware. Quiesce and resume are needed for certain events related to power management and hotplugging. Quiesce and resume also are required to handle interrupt callback actions.

The quiesce function is called in xx_detach().

int
xx_quiesce(xx_state_t *statep);

The resume function is called in xx_attach().

int
xx_resume(xx_state_t *statep);

Make the following modifications to enhance this device driver to use the Interrupt Resource Management feature:

/*
 * attach(9F) routine.
 *
 * Creates soft state, registers callback handler, initializes
 * hardware, and sets up interrupt handling for the driver.
 */
xx_attach(dev_info_t *dip, ddi_attach_cmd_t cmd)
{
        xx_state_t              *statep = NULL;
        xx_intr_t               *intrs = NULL;
        ddi_intr_handle_t       *hdls;
        ddi_cb_handle_t         cb_hdl;
        int                     instance;
        int                     type;
        int                     types;
        int                     nintrs;
        int                     nactual;
        int                     inum;

        /* Get device instance */
        instance = ddi_get_instance(dip);

        switch (cmd) {
        case DDI_ATTACH:

                /* Get soft state */
                if (ddi_soft_state_zalloc(state_list, instance) != 0)
                        return (DDI_FAILURE);
                statep = ddi_get_soft_state(state_list, instance);
                ddi_set_driver_private(dip, (caddr_t)statep);
                statep->dip = dip;

                /* Initialize hardware */
                xx_initialize(statep);

                /* Register callback handler */
                if (ddi_cb_register(dip, DDI_CB_FLAG_INTR, xx_cbfunc,
                    statep, NULL, &cb_hdl) != 0) {
                        ddi_soft_state_free(state_list, instance);
                        return (DDI_FAILURE);
                }
                statep->cb_hdl = cb_hdl;

                /* Select interrupt type */
                ddi_intr_get_supported_types(dip, &types);
                if (types & DDI_INTR_TYPE_MSIX) {
                        type = DDI_INTR_TYPE_MSIX;
                } else if (types & DDI_INTR_TYPE_MSI) {
                        type = DDI_INTR_TYPE_MSI;
                } else {
                        type = DDI_INTR_TYPE_FIXED;
                }
                statep->type = type;

                /* Get number of supported interrupts */
                ddi_intr_get_nintrs(dip, type, &nintrs);

                /* Allocate interrupt handle array */
                statep->hdls_size = nintrs * sizeof (ddi_intr_handle_t);
                statep->hdls = kmem_zalloc(statep->hdls_size, KMEM_SLEEP);

                /* Allocate interrupt setup array */
                statep->intrs_size = nintrs * sizeof (xx_intr_t);
                statep->intrs = kmem_zalloc(statep->intrs_size, KMEM_SLEEP);

                /* Allocate interrupt vectors */
                ddi_intr_alloc(dip, hdls, type, 0, nintrs, &nactual, 0);
                statep->nactual = nactual;

                /* Configure interrupt handling */
                xx_setup_interrupts(statep, statep->nactual, statep->intrs);

                /* Install and enable interrupt handlers */
                for (inum = 0; inum < nactual; inum++) {
                        ddi_intr_add_handler(&hdls[inum],
                            intrs[inum].inthandler,
                            intrs[inum].arg1, intrs[inum].arg2);
                        ddi_intr_enable(hdls[inum]);
                }

                break;

        case DDI_RESUME:

                /* Get soft state */
                statep = ddi_get_soft_state(state_list, instance);
                if (statep == NULL)
                        return (DDI_FAILURE);

                /* Resume hardware */
                xx_resume(statep);

                break;
        }

        return (DDI_SUCESS);
}

/*
 * detach(9F) routine.
 *
 * Stops the hardware, disables interrupt handling, unregisters
 * a callback handler, and destroys the soft state for the driver.
 */
xx_detach(dev_info_t *dip, ddi_detach_cmd_t cmd)
{
        xx_state_t      *statep = NULL;
        int             instance;
        int             inum;

        /* Get device instance */
        instance = ddi_get_instance(dip);

        switch (cmd) {
        case DDI_DETACH:

                /* Get soft state */
                statep = ddi_get_soft_state(state_list, instance);
                if (statep == NULL)
                        return (DDI_FAILURE);

                /* Stop device */
                xx_uninitialize(statep);

                /* Disable and free interrupts */
                for (inum = 0; inum < statep->nactual; inum++) {
                        ddi_intr_disable(statep->hdls[inum]);
                        ddi_intr_remove_handler(statep->hdls[inum]);
                        ddi_intr_free(statep->hdls[inum]);
                }

                /* Unregister callback handler */
                ddi_cb_unregister(statep->cb_hdl);

                /* Free interrupt handle array */
                kmem_free(statep->hdls, statep->hdls_size);

                /* Free interrupt setup array */
                kmem_free(statep->intrs, statep->intrs_size);

                /* Free soft state */
                ddi_soft_state_free(state_list, instance);

                break;

        case DDI_SUSPEND:

                /* Get soft state */
                statep = ddi_get_soft_state(state_list, instance);
                if (statep == NULL)
                        return (DDI_FAILURE);

                /* Suspend hardware */
                xx_quiesce(statep);

                break;
        }

        return (DDI_SUCCESS);
}

/*
 * (*ddi_cbfunc)() routine.
 *
 * Adapt interrupt usage when availability changes.
 */
int
xx_cbfunc(dev_info_t *dip, ddi_cb_action_t cbaction, void *cbarg,
    void *arg1, void *arg2)
{
        xx_state_t      *statep = (xx_state_t *)arg1;
        int             count;
        int             inum;
        int             nactual;

        switch (cbaction) {
        case DDI_CB_INTR_ADD:
        case DDI_CB_INTR_REMOVE:

                /* Get change in availability */
                count = (int)(uintptr_t)cbarg;

                /* Suspend hardware */
                xx_quiesce(statep);

                /* Tear down previous interrupt handling */
                for (inum = 0; inum < statep->nactual; inum++) {
                        ddi_intr_disable(statep->hdls[inum]);
                        ddi_intr_remove_handler(statep->hdls[inum]);
                }

                /* Adjust interrupt vector allocations */
                if (cbaction == DDI_CB_INTR_ADD) {

                        /* Allocate additional interrupt vectors */
                        ddi_intr_alloc(dip, statep->hdls, statep->type,
                            statep->nactual, count, &nactual, 0);

                        /* Update actual count of available interrupts */
                        statep->nactual += nactual;

                } else {

                        /* Free removed interrupt vectors */
                        for (inum = statep->nactual - count;
                            inum < statep->nactual; inum++) {
                                ddi_intr_free(statep->hdls[inum]);
                        }

                        /* Update actual count of available interrupts */
                        statep->nactual -= count;
                }

                /* Configure interrupt handling */
                xx_setup_interrupts(statep, statep->nactual, statep->intrs);

                /* Install and enable interrupt handlers */
                for (inum = 0; inum < statep->nactual; inum++) {
                        ddi_intr_add_handler(&statep->hdls[inum],
                            statep->intrs[inum].inthandler,
                            statep->intrs[inum].arg1,
                            statep->intrs[inum].arg2);
                        ddi_intr_enable(statep->hdls[inum]);
                }

                /* Resume hardware */
                xx_resume(statep);

                break;

        default:
                return (DDI_ENOTSUP);
        }

        return (DDI_SUCCESS);
}