Writing Device Drivers

Managing DMA Resources

This section describes how to manage DMA resources.

Object Locking

Before allocating the DMA resources for a memory object, the object must be prevented from moving. Otherwise, the system can remove the object from memory while the device is writing to it, causing the data transfer to fail, and possibly corrupting the system. The process of preventing memory objects from moving during a DMA transfer is known as locking down the object.

The following object types do not require explicit locking:

Buffers coming from the file system through strategy(9E). These buffers are already locked by the file system.
Kernel memory allocated within the device driver, such as that allocated by ddi_dma_mem_alloc(9F).

For other objects (such as buffers from user space), physio(9F) or ddi_umem_lock(9F) must be used to lock down the objects. This is usually performed in the read(9E) or write(9E) routines of a character device driver. See Data Transfer Methods for an example.

Allocating a DMA Handle

A DMA handle is an opaque object that is used as a reference to subsequently allocated DMA resources. It is usually allocated in the driver's attach entry point using ddi_dma_alloc_handle(9F). ddi_dma_alloc_handle(9F) takes the device information referred to by dip and the device's DMA attributes described by a ddi_dma_attr(9S) structure as parameters. ddi_dma_alloc_handle(9F) has the following syntax:

int ddi_dma_alloc_handle(dev_info_t *dip,
    ddi_dma_attr_t *attr, int (*callback)(caddr_t),
    caddr_t arg, ddi_dma_handle_t *handlep);

dip: Pointer to the device's dev_info structure
attr: Pointer to a ddi_dma_attr(9S) structure as described in DMA Attributes
callback: Address of the callback function for handling resource allocation failures
arg: Argument to be passed to the callback function
handlep: Pointer to a DMA handle to store the returned handle

Allocating DMA Resources

Two interfaces allocate DMA resources:

ddi_dma_buf_bind_handle(9F) – Used with buf(9S) structures

ddi_dma_addr_bind_handle(9F) – Used with virtual addresses

DMA resources are usually allocated in the driver's xxstart() routine, if one exists. See Asynchronous Data Transfers (Block Drivers) for discussion of xxstart. These two interfaces have the following syntax:

int ddi_dma_addr_bind_handle(ddi_dma_handle_t handle,
    struct as *as, caddr_t addr,
    size_t len, uint_t flags, int (*callback)(caddr_t),
    caddr_t arg, ddi_dma_cookie_t *cookiep, uint_t *ccountp);

int ddi_dma_buf_bind_handle(ddi_dma_handle_t handle,
    struct buf *bp, uint_t flags,
    int (*callback)(caddr_t), caddr_t arg,
    ddi_dma_cookie_t *cookiep, uint_t *ccountp);

The following arguments are common to both ddi_dma_addr_bind_handle(9F) and ddi_dma_buf_bind_handle(9F):

handle: DMA handle and the object for allocating resources
flags: Set of flags indicating the transfer direction and other attributes. DDI_DMA_READ indicates a data transfer from device to memory. DDI_DMA_WRITE indicates a data transfer from memory to device. See the ddi_dma_addr_bind_handle(9F) or ddi_dma_buf_bind_handle(9F) man pages for a complete discussion of the allowed flags.
callback: Address of callback function for handling resource allocation failures. See the ddi_dma_addr_bind_handle(9F) man page.
arg: Argument to pass to the callback function.
cookiep: Pointer to the first DMA cookie for this object.
ccountp: Pointer to the number of DMA cookies for this object.

For ddi_dma_addr_bind_handle(9F), the object is described by an address range, where as is a pointer to an address space structure (this must be NULL), addr is the base kernel address of the object, and len is the length of the object in bytes.
For ddi_dma_buf_bind_handle(9F), the object is described by a buf(9S) structure pointed to by bp.

Device Register Structure

DMA capable devices have more registers than have been used in previous examples. This section adds the following fields to the device register structure to support DMA-capable device examples.

For DMA engines without scatter-gather support:

uint32_t      dma_addr;      /* starting address for DMA */
uint32_t      dma_size;      /* amount of data to transfer */

For DMA engines with scatter-gather support:

struct sglentry {
    uint32_t        dma_addr;
    uint32_t        dma_size;
} sglist[SGLLEN];

caddr_t      iopb_addr;      /* When written informs device of the next */
                             /* command's parameter block address. */
                             /* When read after an interrupt,contains */
                             /* the address of the completed command. */

DMA Callback Example

In Example 8–1, xxstart() is used as the callback function and the per-device state structure is given as its argument. xxstart() attempts to start the command. If the command cannot be started because resources are not available, xxstart() is scheduled to be called sometime later, when resources might be available.

Because xxstart() is used as a DMA callback, it must follow these rules imposed on DMA callbacks:

It must not assume that resources are available (it must try to allocate them again).
It must indicate to the system whether allocation succeeded by returning DDI_DMA_CALLBACK_RUNOUT if it fails to allocate resources (and needs to be called again later) or DDI_DMA_CALLBACK_DONE indicating success (so no further callback is necessary).

Example 8–1 DMA Callback Example

static int
xxstart(caddr_t arg)
{
    struct xxstate *xsp = (struct xxstate *)arg;
    struct device_reg *regp;
    int flags;
    mutex_enter(&xsp->mu);
    if (xsp->busy) {
            /* transfer in progress */
            mutex_exit(&xsp->mu);
            return (DDI_DMA_CALLBACK_RUNOUT);
    }
    xsp->busy = 1;
    regp = xsp->regp;
    if (transfer is a read) {
            flags = DDI_DMA_READ;
    } else {
            flags = DDI_DMA_WRITE;
    }
    mutex_exit(&xsp->mu);
    if (ddi_dma_buf_bind_handle(xsp->handle,xsp->bp,flags,
xxstart,
            (caddr_t)xsp, &cookie, &ccount) != DDI_DMA_MAPPED) {
            /* really should check all return values in a switch */
                mutex_enter(&xsp->mu);
                xsp->busy=0;
                mutex_exit(&xsp->mu);
            return (DDI_DMA_CALLBACK_RUNOUT);
    }
    ...
    program the DMA engine
    ...
    return (DDI_DMA_CALLBACK_DONE);
}

Determining Maximum Burst Sizes

Drivers specify the DMA burst sizes that their device supports in the dma_attr_burstsizes field of the ddi_dma_attr(9S) structure. This is a bitmap of the supported burst sizes. However, when DMA resources are allocated, the system might impose further restrictions on the burst sizes that might be actually used by the device. The ddi_dma_burstsizes(9F) routine can be used to obtain the allowed burst sizes. It returns the appropriate burst size bitmap for the device. When DMA resources are allocated, a driver can ask the system for appropriate burst sizes to use for its DMA engine.

Example 8–2 Determining Burst Size

#define BEST_BURST_SIZE 0x20 /* 32 bytes */

     if (ddi_dma_buf_bind_handle(xsp->handle,xsp->bp, flags, xxstart,
         (caddr_t)xsp, &cookie, &ccount) != DDI_DMA_MAPPED) {
             /* error handling */
      }
    burst = ddi_dma_burstsizes(xsp->handle);
    /* check which bit is set and choose one burstsize to */
     /* program the DMA engine */
     if (burst & BEST_BURST_SIZE) {
         program DMA engine to use this burst size
     } else {
         other cases
     }

Allocating Private DMA Buffers

Some device drivers might need to allocate memory for DMA transfers to or from a device, besides doing transfers requested by user threads and the kernel. Examples of this are setting up shared memory for communication with the device and allocating intermediate transfer buffers. Use ddi_dma_mem_alloc(9F) to allocate memory for DMA transfers.

int ddi_dma_mem_alloc(ddi_dma_handle_t handle, size_t length,
    ddi_device_acc_attr_t *accattrp, uint_t flags,
    int (*waitfp)(caddr_t), caddr_t arg, caddr_t *kaddrp,
    size_t *real_length, ddi_acc_handle_t *handlep);

handle: DMA handle
length: Length in bytes of the desired allocation
accattrp: Pointer to a device access attribute structure
flags: Data transfer mode flags; possible values are: DDI_DMA_CONSISTENT and DDI_DMA_STREAMING
waitfp: Address of callback function for handling resource allocation failures. See the ddi_dma_mem_alloc(9F) man page
arg: Argument to pass to the callback function
kaddrp: Pointer (on a successful return) that contains the address of the allocated storage
real_length: Length in bytes that was allocated
handlep: Pointer to a data access handle

flags should be set to DDI_DMA_CONSISTENT if the device accesses in a nonsequential fashion, or if synchronization steps using ddi_dma_sync(9F) should be as lightweight as possible (because of frequent use on small objects). This type of access is commonly known as consistent access. I/O parameter blocks that are used for communication between a device and the driver are set up this way.

On the IA platform, to allocate memory for DMA using physically contiguous pages, set the length of the scatter/gather list dma_attr_sgllen in the ddi_dma_attr(9S) structure to 1, and do not specify DDI_DMA_PARTIAL which would otherwise permit partial resource allocation.

Example 8–3 shows how to allocate IOPB memory and the necessary DMA resources to access it. DMA resources must still be allocated, and the DDI_DMA_CONSISTENT flag must be passed to the allocation function.

Example 8–3 Using `ddi_dma_mem_alloc(9F)`

if (ddi_dma_mem_alloc(xsp->iopb_handle, size, &accattr,
        DDI_DMA_CONSISTENT, DDI_DMA_SLEEP, NULL, &xsp->iopb_array,
        &real_length, &xsp->acchandle) != DDI_SUCCESS) {
        error handling
        goto failure;
}
if (ddi_dma_addr_bind_handle(xsp->iopb_handle, NULL,
        xsp->iopb_array, real_length,
        DDI_DMA_READ | DDI_DMA_CONSISTENT, DDI_DMA_SLEEP,
        NULL, &cookie, &count) != DDI_DMA_MAPPED) {
        error handling
        ddi_dma_mem_free(&xsp->acchandle);
        goto failure;
}

flags should be set to DDI_DMA_STREAMING if the device is doing sequential, unidirectional, block-sized and block-aligned transfers to or from memory. This type of access is commonly known as streaming access.

For example, if an I/O transfer can be sped up by using an I/O cache, which at a minimum transfers (flushes) one cache line, ddi_dma_mem_alloc(9F) will round the size to a multiple of the cache line to avoid data corruption.

ddi_dma_mem_alloc(9F) returns the actual size of the allocated memory object. Because of padding and alignment requirements, the actual size might be larger than the requested size. ddi_dma_addr_bind_handle(9F) requires the actual length.

ddi_dma_mem_free(9F) is used to free the memory allocated by ddi_dma_mem_alloc(9F).

Note –

If the memory is not properly aligned, the transfer will succeed but the system will choose a different (and possibly less efficient) transfer mode that requires fewer restrictions. For this reason, ddi_dma_mem_alloc(9F) is preferred over kmem_alloc(9F) when allocating memory for the device to access.

Handling Resource Allocation Failures

The resource-allocation routines provide the driver with several options when handling allocation failures. The waitfp argument indicates whether the allocation routines will block, return immediately, or schedule a callback, as shown in Table 8–1.

Table 8–1 Resource Allocation Handling


`waitfp` value	Indicated Action
`DDI_DMA_DONTWAIT`	Driver does not want to wait for resources to become available
`DDI_DMA_SLEEP`	Driver is willing to wait indefinitely for resources to become available
Other values	The address of a function to be called when resources are likely to be available

Programming the DMA Engine

When the resources have been successfully allocated, the device must be programmed. Although programming a DMA engine is device specific, all DMA engines require a starting address and a transfer count. Device drivers retrieve these two values from the DMA cookie returned by a successful call from ddi_dma_addr_bind_handle(9F), ddi_dma_buf_bind_handle(9F), or ddi_dma_getwin(9F). These functions all return the first DMA cookie and a cookie count indicating whether the DMA object consists of more than one cookie. If the cookie count N is greater than 1, ddi_dma_nextcookie(9F) has to be called N-1 times to retrieve all the remaining cookies.

A cookie is of type ddi_dma_cookie(9S) and has the following fields:

uint64_t        _dmac_ll;       /* 64-bit DMA address */
uint32_t        _dmac_la[2];    /* 2 x 32-bit address */
size_t          dmac_size;      /* DMA cookie size */
uint_t          dmac_type;      /* bus specific type bits */

The dmac_laddress specifies a 64-bit I/O address appropriate for programming the device's DMA engine. If a device has a 64-bit DMA address register, a driver should use this field to program the DMA engine. The dmac_address field specifies a 32-bit I/O address that should be used for devices that have a 32-bit DMA address register. dmac_size contains the transfer count. Depending on the bus architecture, the dmac_type field in the cookie might be required by the driver. The driver should not perform any manipulations, such as logical or arithmetic, on the cookie.

Example 8–4 `ddi_dma_cookie(9S)` Example

ddi_dma_cookie_t                    cookie;

     if (ddi_dma_buf_bind_handle(xsp->handle,xsp->bp, flags, xxstart,
         (caddr_t)xsp, &cookie, &xsp->ccount) != DDI_DMA_MAPPED) {
             /* error handling */
      }
     sglp = regp->sglist;
     for (cnt = 1; cnt <= SGLLEN; cnt++, sglp++) {
         /* store the cookie parms into the S/G list */
         ddi_put32(xsp->access_hdl, &sglp->dma_size,
             (uint32_t)cookie.dmac_size);
         ddi_put32(xsp->access_hdl, &sglp->dma_addr,
             cookie.dmac_address);
         /* Check for end of cookie list */
         if (cnt == xsp->ccount)
             break;
         /* Get next DMA cookie */
         (void) ddi_dma_nextcookie(xsp->handle, &cookie);
     }
        /* start DMA transfer */
     ddi_put8(xsp->access_hdl, &regp->csr,
         ENABLE_INTERRUPTS | START_TRANSFER);

Note –

ddi_dma_addr_bind_handle(9F) and ddi_dma_buf_bind_handle(9F) can return more DMA cookies than fit into the scatter-gather list. In this case, the driver has to continue the transfer in the interrupt routine and reprogram the scatter-gather list with the remaining DMA cookies. You must handle sgllen cookies at a time.

Freeing the DMA Resources

After a DMA transfer is completed (usually in the interrupt routine), the driver can release DMA resources by calling ddi_dma_unbind_handle(9F).

As described in Synchronizing Memory Objects, ddi_dma_unbind_handle(9F) calls ddi_dma_sync(9F), eliminating the need for any explicit synchronization. After calling ddi_dma_unbind_handle(9F), the DMA resources become invalid, and further references to them have undefined results. Example 8–5 shows how to use ddi_dma_unbind_handle(9F).

Example 8–5 Freeing DMA Resources

static uint_t
xxintr(caddr_t arg)
{
     struct xxstate *xsp = (struct xxstate *)arg;
     uint8_t    status;
        volatile   uint8_t   temp;
     mutex_enter(&xsp->mu);
     /* read status */
     status = ddi_get8(xsp->access_hdl, &xsp->regp->csr);
     if (!(status & INTERRUPTING)) {
            mutex_exit(&xsp->mu);
            return (DDI_INTR_UNCLAIMED);
     }
     ddi_put8(xsp->access_hdl, &xsp->regp->csr, CLEAR_INTERRUPT);
      /* for store buffers */
     temp = ddi_get8(xsp->access_hdl, &xsp->regp->csr);
     ddi_dma_unbind_handle(xsp->handle);
     ...
         /* check for errors */
     ...
     xsp->busy = 0;
     mutex_exit(&xsp->mu);
     if (pending transfers) {
            (void) xxstart((caddr_t)xsp);
     }
     return (DDI_INTR_CLAIMED);
}

The DMA resources should be released and reallocated if a different object will be used in the next transfer. However, if the same object is always used, the resources can be allocated once and continually reused as long as there are intervening calls to ddi_dma_sync(9F).

Freeing the DMA Handle

When the driver is detached, the DMA handle must be freed. ddi_dma_free_handle(9F) destroys the DMA handle and any residual resources the system is caching on the handle. Any further references of the DMA handle will have undefined results.

Canceling DMA Callbacks

DMA callbacks cannot be canceled. This requires some additional code in the drivers detach(9E) routine, as it must not return DDI_SUCCESS if there are any outstanding callbacks. (See Example 8–6.) When DMA callbacks occur, the detach(9E) routine must wait for the callback to run and must prevent it from rescheduling itself. This can be done using additional fields in the state structure, as shown below.

Example 8–6 Canceling DMA Callbacks

static int
xxdetach(dev_info_t *dip, ddi_detach_cmd_t cmd)
{
     ...
     mutex_enter(&xsp->callback_mutex);
     xsp->cancel_callbacks = 1;
     while (xsp->callback_count > 0) {
            cv_wait(&xsp->callback_cv, &xsp->callback_mutex);
     }
     mutex_exit(&xsp->callback_mutex);
     ...
 }

static int
xxstrategy(struct buf *bp)
{
     ...
     mutex_enter(&xsp->callback_mutex);
       xsp->bp = bp;
     error = ddi_dma_buf_bind_handle(xsp->handle, xsp->bp, flags,
                 xxdmacallback, (caddr_t)xsp, &cookie, &ccount);
     if (error == DDI_DMA_NORESOURCES)
           xsp->callback_count++;
     mutex_exit(&xsp->callback_mutex);
     ...
}

static int
xxdmacallback(caddr_t callbackarg)
{
     struct xxstate *xsp = (struct xxstate *)callbackarg;
     ...
     mutex_enter(&xsp->callback_mutex);
     if (xsp->cancel_callbacks) {
            /* do not reschedule, in process of detaching */
            xsp->callback_count--;
            if (xsp->callback_count == 0)
               cv_signal(&xsp->callback_cv);
            mutex_exit(&xsp->callback_mutex);
            return (DDI_DMA_CALLBACK_DONE);        /* don't reschedule it */
     }
     /*
        * Presumably at this point the device is still active
        * and will not be detached until the DMA has completed.
        * A return of 0 means try again later
        */
     error = ddi_dma_buf_bind_handle(xsp->handle, xsp->bp, flags,
                 DDI_DMA_DONTWAIT, NULL, &cookie, &ccount);
     if (error == DDI_DMA_MAPPED) {
            ...
            /* program the DMA engine */
             ...
            xsp->callback_count--;
            mutex_exit(&xsp->callback_mutex);
            return (DDI_DMA_CALLBACK_DONE);
     }
     if (error != DDI_DMA_NORESOURCES) {
            xsp->callback_count--;
            mutex_exit(&xsp->callback_mutex);
            return (DDI_DMA_CALLBACK_DONE);
     }
     mutex_exit(&xsp->callback_mutex);
     return (DDI_DMA_CALLBACK_RUNOUT);
}

Synchronizing Memory Objects

At various points when the memory object is accessed (including the time of removal of the DMA resources), the driver might need to synchronize the memory object with respect to various caches. This section gives guidelines on when and how to synchronize memory objects.

Cache

Cache is a very high-speed memory that sits between the CPU and the system's main memory (CPU cache), or between a device and the system's main memory (I/O cache), as shown in Figure 8–1.

Figure 8–1 CPU and System I/O Caches

Diagram shows how the cache is used to speed data transfers involving devices.

When an attempt is made to read data from main memory, if the associated cache determines whether it contains the requested data, it quickly satisfies the request. If the cache does not have the data, it retrieves the data from main memory, passes the data on to the requestor, and saves the data in case that data is requested again.

Similarly, on a write cycle, the data is stored in the cache quickly and the CPU or device is allowed to continue executing (transferring). This takes much less time than if the CPU or device had to wait for the data to be written to memory.

With this model, after a device transfer has been completed, the data can still be in the I/O cache but not yet in main memory. If the CPU accesses the memory, it might read the wrong data from the CPU cache. To ensure a consistent view of the memory for the CPU, the driver must call a synchronization routine to flush the data from the I/O cache and update the CPU cache with the new data. Similarly, a synchronization step is required if data modified by the CPU is to be accessed by a device.

There might also be additional caches and buffers between the device and memory, such as caches associated with bus extenders or bridges. Use ddi_dma_sync(9F) to synchronize all applicable caches.

`ddi_dma_sync(9F)`

If a memory object has multiple mappings—such as for a device (through the DMA handle) and for the CPU—and one mapping is used to modify the memory object, the driver needs to call ddi_dma_sync(9F) to ensure that the modification of the memory object is complete before accessing the object through another mapping. ddi_dma_sync(9F) can also inform other mappings of the object that any cached references to the object are now stale. Additionally, ddi_dma_sync(9F) flushes or invalidates stale cache references as necessary.

Generally, the driver has to call ddi_dma_sync(9F) when a DMA transfer completes. The exception to this is that deallocating the DMA resources with ddi_dma_unbind_handle(9F), does an implicit ddi_dma_sync(9F) on behalf of the driver.

int ddi_dma_sync(ddi_dma_handle_t handle, off_t off,
size_t length, uint_t type);

If the object is going to be read by the DMA engine of the device, the device's view of the object must be synchronized by setting type to DDI_DMA_SYNC_FORDEV. If the DMA engine of the device has written to the memory object, and the object is going to be read by the CPU, the CPU's view of the object must be synchronized by setting type to DDI_DMA_SYNC_FORCPU.

Here is an example of synchronizing a DMA object for the CPU:

if (ddi_dma_sync(xsp->handle, 0, length, DDI_DMA_SYNC_FORCPU)
    == DDI_SUCCESS) {
    /* the CPU can now access the transferred data */
    ...        
} else {
         error handling
}

If the only mapping that concerns the driver is one for the kernel (such as memory allocated by ddi_dma_mem_alloc(9F)), the flag DDI_DMA_SYNC_FORKERNEL can be used. If the system can synchronize the kernel's view faster than the CPU's view, it will do so; otherwise, it acts the same as DDI_DMA_SYNC_FORCPU.