JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Writing Device Drivers     Oracle Solaris 10 1/13 Information Library
search filter icon
search icon

Document Information

Preface

Part I Designing Device Drivers for the Oracle Solaris Platform

1.  Overview of Oracle Solaris Device Drivers

2.  Oracle Solaris Kernel and Device Tree

3.  Multithreading

4.  Properties

5.  Managing Events and Queueing Tasks

6.  Driver Autoconfiguration

7.  Device Access: Programmed I/O

8.  Interrupt Handlers

9.  Direct Memory Access (DMA)

10.  Mapping Device and Kernel Memory

11.  Device Context Management

12.  Power Management

13.  Hardening Oracle Solaris Drivers

14.  Layered Driver Interface (LDI)

Part II Designing Specific Kinds of Device Drivers

15.  Drivers for Character Devices

16.  Drivers for Block Devices

17.  SCSI Target Drivers

18.  SCSI Host Bus Adapter Drivers

19.  Drivers for Network Devices

20.  USB Drivers

21.  SR-IOV Drivers

Introduction to SR-IOV

Benefits of SR-IOV

Supported Platforms

Glossary

Overview of SR-IOV Device Driver

Physical Function (PF) Driver

Virtual Function (VF) Driver

Device Configuration Parameters

pci.conf File

Setting Device Configuration Parameters

SR-IOV Configuration on Sparc OVM Platform

SR-IOV Configuration on Bare Metal Platforms

Boot Configuration Sequence

SR-IOV Interfaces Summary

Driver Ioctls

Interfaces for SR-IOV Drivers

pci_param_get() Interface

pci_param_get_ioctl() Interface

pci_plist_get() Interface

pci_plist_getvf() Interface

pciv_vf_config() Interface

pci_plist_lookup() Interface

pci_param_free() Interface

pciv_send() Interface

SR-IOV Driver Ioctls

Data Structures

iov_param_ver_info Structure

iov_param_validate Structure

iov_param_desc Structure

IOV_GET_VER_INFO Ioctl

IOV_GET_PARAM_INFO Ioctl

IOV_VALIDATE_PARAM Ioctl

Driver Callbacks

Sample Code for Driver Ioctls

Part III Building a Device Driver

22.  Compiling, Loading, Packaging, and Testing Drivers

23.  Debugging, Testing, and Tuning Device Drivers

24.  Recommended Coding Practices

Part IV Appendixes

A.  Hardware Overview

B.  Summary of Solaris DDI/DKI Services

C.  Making a Device Driver 64-Bit Ready

D.  Console Frame Buffer Drivers

E.  pci.conf File

Index

Interfaces for SR-IOV Drivers

The following sections describe the interfaces for SR-IOV drivers.

pci_param_get() Interface

SR-IOV drivers must use the pci_param_get(9F) interface to obtain the list of currently configured parameters. This interface is called during driver attachment or at any other appropriate time. The returned data, which is a pointer to the parameter list, contains the name-value information of both the PF and its corresponding VF devices.

int pci_param_get(dev_info_t *dip, pci_param_t *php)

where:

dip

A pointer to the dev_info structure

php

A pointer to param handle, pci_param_t

The device driver should perform the following steps to obtain the list of parameters of PF and VF after calling the pci_param_get interface:

  1. Call the pci_plist_get(9F) interface to obtain the list of parameters of the PF device and the pci_plist_getvf(9F) interface to obtain the list of parameters of the configured VFs.

  2. Call the pci_plist_lookup(9F) interface to obtain the device parameters.

  3. Validate all the PF and VF parameters

  4. If the parameters do not match the current configuration, the driver should fail the device attachment.

  5. Call the pci_param_free(9F) interface to free the pointer to the parameter handle obtaining the parameters for PF and the configured VF devices. See Example 21-2


Note - Validation of the parameters should be completed before the VFs are configured.


The name-value pairs are defined on a per–device basis. There is one set of name-value pairs for the PF and one set each for the configured VFs. The name-value pairs are optional and may be absent for any or all of the devices.

Example 21-2 SR-IOV pci_param_get(9F) Routine

    pci_param_t my_params;
pci_plist_t pf_plist;
pci_plist_t  vf_plist[8];
labelp = NULL;
rval = pci_param_get(dip,&my_params);
if (rval || (my_params == NULL)) {
   cmn_err(CE_NOTE, "No params available\n");
goto continue_with_attach;
}
rval = pci_plist_get(my_params, &pf_list);
if (rval || (pf_plist == NULL)) {
        cmn_err(CE_NOTE, "No params for PF \n");
goto continue_with_attach;
}
for (i = 0; i < 8; i++) {
    rval = pci_plist_getvf(my_params, i, &vf_plist[i]);
    if (rval || (vf_plist[i] == NULL)) {
     cmn_err(CE_WARN, "No params for VF %d\n", i);
     continue;
 }
}
pci_param_free(my_params);
/*
* Validate the PF and VF params lists.
* Fail the attach if the params are incompatible or exceed the
* resources available.
*/
continue_with_attach:

pci_param_get_ioctl() Interface

SR-IOV device drivers can use the pci_param_get_ioctl(9F) interface to extract the parameters for the PF and VF devices from the arg argument if they implement the IOV_VALIDATE_PARAM ioctl.

int pci_param_get_ioctl(dev_info_t *dip, intptr_t arg, int mode,pci_param_t *php)

where:

dip

A pointer to the dev_info structure

arg

Argument obtained from the driver's ioctl call

mode

Argument obtained from the driver's ioctl call

php

A pointer to the param handle, pci_param_t, which is a handle obtained from calling the pci_param_get()or pci_param_get_ioctl() interfaces

The driver should call the pci_param_free() interface to free the param handle returned in this call after retrieving the parameters.

pci_plist_get() Interface

The pci_plist_get(9F) interface is used to obtain the parameter list from the param handle obtained from either the pci_param_get(9F) call or the pci_param_get_ioctl(9F) call.

int pci_plist_get(pci_param_t param, pci_plist_t *plist_p)

where:

param

A handle obtained from the pci_param_get()or pci_param_get_ioctl() interfaces.

plist_p

A pointer to pci_plist_t where a non-NULL plist is returned on a successful return.

The plist that is returned from the pci_plist_get() call is only for the PF function. The structure pci_plist_t supports arrays of the following data types:

pci_plist_getvf() Interface

The pci_plist_getvf(9F) interface is used to obtain the name-value pair list for VF devices.

int pciv_plist_getvf (pci_param_t param, uint16_t vf_index, pci_plist_t *vfplist_p)

where:

param

A handle obtained from pci_param_get()or pci_param_get_ioctl() interfaces.

vf_index

A value between 0 through #VFS - 1.

*vfplist_p

A pointer to the pci_plist_t structure.

pciv_vf_config() Interface

The pciv_vf_config(9F) interface is used by the SR-IOV drivers to obtain configuration info about the VFs and is also used to configure the VFs during the attachment of the driver.

#include <sys/sunddi.h>
int pciv_vf_config(dev_info_t *dip, pciv_config_vf_t *vfcfg_p)

where:

dip

A pointer to the dev_info structure.

vfcfg_p

A pointer to the pciv_config_vf structure.

typedef enum { 
                 PCIV_VFCFG_PARAM,
                 PCIV_VF_ENABLE,
                          PCIV_VF_DISABLE
                 PCIV_EVT_VFENABLE_PRE,
                 PCIV_EVT_VFENABLE_POST,
                 PCIV_EVT_VFDISABLE_PRE,
                 PCIV_EVT_VFDISABLE_POST
             } pciv_vf_config_cmd_t;

The pciv_config_vf structure contains the following fields:

typedef struct pciv_config_vf {
               int version;
                pciv_vf_config_cmd_t cmd;
                uint16_t num_vf;
                uint16_t first_vf_offset;
                uint16_t vf_stride;
                boolean_t ari_cap;
                uint32_t page_size;
     } pciv_config_vf_t;

where:

version

Version number.

cmd

Used to indicate whether this interface is called to obtain configuration information or to attach the VFs.

  • PCIV_VFCFG_PARAM – Obtain configuration information

  • PCIV_VF_ENABLE – Enable the VFs

    • PCIV_EVT_VFENABLE_PRE

    • PCIV_EVT_VFDISABLE_PRE

    • PCIV_EVT_VFENABLE_POST

    • PCIV_EVT_VFDISABLE_POST

num_vf

Number of VFs defined in the backend.

vf_stride

Distance between the VFs.

first_vf_offset

Offset between the first VF and PF.

ari_cap

ARI-capable hierarchy.

page_size

Specifies system page size.

The driver should first call the pciv_vfconfig() interface with the cmd field set to PCIV_VFCFG_PARAM to obtain the configuration information. The driver should then call this interface again with the cmd field set to PCIV_VF_ENABLE to configure the VFs.

The driver can return one of the following error codes:

DDI_SUCCESS

DDI_FAILURE

PCIV_REQRESET

PCIV_REQREATTACH

The driver does not have to register the callbacks with the DDI_CB_FLAG_SRIOV bit set before calling the pciv_vf_config() interface to enable the VFs. However in order to receive notifications when the VFs are unconfigured by the SR-IOV framework, the drivers must register callbacks with DDI_CB_FLAG_SRIOV bit set after the VFs are enabled. See Driver Callbacks for additional information.

All PF drivers that can support VFs should inform the PCIe framework of their capability by calling the ddi_cb_register(9F) with DDI_CB_FLAG_SRIOV flag set in the flags argument. The ddi_cb_register() function must be called in the driver's attach routine. If the PF device driver calls the pciv_vf_config() function to enable VFs in its attach routine, then the PF driver should call the ddi_cb_register() function after enabling the VFs.

The DDI_CB_FLAG_SRIOV flag is required by the framework to perform the following actions:

pci_plist_lookup() Interface

The pci_plist_lookup(9F) interface can be used by the drivers to look up name-value pairs of the various data types that are supported. The functions find the nvpair (name-value pair) that matches the name and type as indicated by the interface name. If found, nelem and val are modified to contain the number of elements in value and the starting address of data, respectively.

The following data types are supported by the pci_plist_lookup() interface:

where:

plist

A pointer to the pci_plist_t structure to be processed.

name

Name of the name-value pair to search.

nelem

Address to store the number of elements in value.

val

Starting address of data.

The pci_plist_lookup() function return 0 on success and an error value on failure. The following error values are supported:

DDI_EINVAL

Invalid argument

ENOENT

No matching name-value pair found

ENOTSUP

An encode or decode method is not supported

pci_param_free() Interface

The pci_param_free(9F) interface must be called by the driver after obtaining the device parameters using the param handle. This call frees up the resources allocated by the pci_param_get() and pci_param_get_ioctl() interfaces.

int pci_param_free (pci_param_t param)

where param is a handle obtained from pci_param_get()or pci_param_get_ioctl() interfaces.

pciv_send() Interface

The pciv_send(9F) interface is used by SR-IOV capable PF and VF drivers to communicate with each other. A PF driver can communicate with any of its VF drivers although a VF driver can only communicate with its PF driver.

int pciv_send(dev_info_t *dip, pciv_pvp_req_t *req

where:

dip

A pointer to the dev_info structure.

req

A pointer to the pciv_pvp_req_t structure.

The structure of the pciv_pvp_req_t is :

typedef struct pciv_pvp_req {
int pvp_dstfunc;
caddr_t pvp_buf;
size_t pvp_nbyte;
buf_cb_t pvp_cb;
caddr_t pvp_cb_arg;
uint_t pvp_flag;
} pciv_pvp_req_t;

where:

pvp_dstfunc

VF index ranges from 1 to num_vf if called by PF driver. If the caller is a VF driver it should always be PCIV_PF.

pvp_buf

Buffer address of caller's buffer to be sent.

pvp_nbyte

Number of bytes to be transmitted, which must be less than 8k.

pvp_cb

Call back function pointer if the pvp_flag is set as PCIV_NOWAIT.

If pvp_flag is set to PCIV_NOWAIT the call returns immediately and the callback routine in pvp_cb is called before data in pvp_buf has been transmitted to the destination. The caller is then allowed to free the buffer in its callback routine.

typedef void (*buf_cb_t)(int rc, caddr_t buf, size_t size, caddr_t cb_arg);

where:

rc

DDI return code for the transmission.

buf

Buffer address of caller's buffer to be sent.

size

Number of bytes to be transmitted.

cb_arg

Input argument the caller has set when calling the routine.

pvp_cb_arg

Call back input argument for pvp_cb if the pvp_flag is set as PCIV_NOWAIT.

pvp_flag
  • PCIV_NOWAIT – Do not wait for receiver's response.

  • PCIV_WAIT – This is the default state. Wait until receiver acknowledges the transmission.

The pciv_send() interface returns one of the following return values:

DDI_SUCCESS

Buffer has been sent successfully.

DDI_ENOTSUP

Device driver does not support this operation. Caller may use other mechanisms, such as hardware mailbox.

DDI_EINVAL

The pvp_nbyte or pvp_dstfunc is invalid.

DDI_ENOMEM

Operation failed due to lack of resources.

DDI_ETRANSPORT

The remote end did not register a call back to handle incoming transmission.

DDI_FAILURE

Failed due to unspecified reasons.