SR-IOV Drivers - Writing Device Drivers in Oracle® Solaris 11.4

Writing Device Drivers in Oracle^® Solaris 11.4

Document Information

Using This Documentation

Product Documentation Library

Feedback

Part I Designing Device Drivers for the Oracle Solaris Platform

Chapter 1 Overview of Oracle Solaris Device Drivers

What's New in Oracle Solaris 11

Device Driver Basics

What Is a Device Driver?

What Is a Device Driver Entry Point?

Device Driver Entry Points

Entry Points Common to All Drivers

Device Access Entry Points

Loadable Module Entry Points

Autoconfiguration Entry Points

Kernel Statistics Entry Points

Power Management Entry Point

System Quiesce Entry Point

Summary of Common Entry Points

Entry Points for Block Device Drivers

Entry Points for Character Device Drivers

Entry Points for STREAMS Device Drivers

Entry Points for Memory Mapped Devices

Entry Points for Network Device Drivers

Entry Points for SCSI HBA Drivers

Considerations in Device Driver Design

Software State Management

Programmed I/O Device Access

Direct Memory Access (DMA)

Layered Driver Interfaces

Driver Context

Returning Errors

Dynamic Memory Allocation

Hotplugging Drivers

Firmware Compatibility

Chapter 2 Oracle Solaris Kernel and Device Tree

What Is the Kernel?

Multithreaded Execution Environment

Virtual Memory

Devices as Special Files

DDI/DKI Interfaces

Overview of the Device Tree

Device Tree Components

Displaying the Device Tree

libdevinfo Library

prtconf Command

/devices Directory

Binding a Driver to a Device

Generic Device Names

Chapter 3 Multithreading in Oracle Solaris

Locking Primitives

Storage Classes of Driver Data

Mutual-Exclusion Locks

Thread Synchronization

Condition Variables in Thread Synchronization

Initializing Condition Variables

Waiting for the Condition

Signaling the Condition

cv_wait() and cv_timedwait() Functions

cv_wait_sig() Function

cv_timedwait_sig() Function

Choosing a Locking Scheme

Potential Locking Pitfalls

Threads Unable to Receive Signals

Chapter 4 Device Properties

Device Properties and Property Values

Device Property Names

Creating and Updating Properties

Looking Up Properties

Changes to the driver.conf File

prop_op() Entry Point

Chapter 5 Managing Events and Queueing Tasks

Managing Events

Introduction to Events

Using ddi_log_sysevent() to Log Events

ddi_log_sysevent() Syntax

Sample Code for Logging Events

Defining Event Attributes

Queueing Tasks

Introduction to Task Queues

Task Queue Interfaces

Observing Task Queues

Task Queue Kernel Statistics Counters

Task Queue DTrace SDT Probes

Chapter 6 Driver Autoconfiguration

Driver Loading and Unloading

Data Structures Required for Drivers

Loadable Driver Interfaces

_init() Example

_fini() Example

_info() Example

Device Configuration Concepts

Device Instances and Instance Numbers

Minor Nodes and Minor Numbers

probe() Entry Point

attach() Entry Point

Driver Soft-State Management

Lock Variable and Conditional Variable Initialization

Creating Minor Device Nodes

Deferred Attach

detach() Entry Point

getinfo() Entry Point

Using Device IDs

Registering Device IDs

Registering a Device-Supplied ID

Registering a Fabricated ID

Unregistering Device IDs

Chapter 7 Device Access: Programmed I/O

Device Memory

Managing Differences in Device and Host Endianness

Managing Data Ordering Requirements

ddi_device_acc_attr Structure

Mapping Device Memory

Mapping Setup Example

Device Access Functions

Alternate Device Access Interfaces

Memory Space Access

I/O Space Access

PCI Configuration Space Access

Chapter 8 Interrupt Handlers

Interrupt Handler Overview

Device Interrupts

High-Level Interrupts

Legacy Interrupts

Standard and Extended Message-Signaled Interrupts

MSI Interrupts

MSI-X Interrupts

Software Interrupts

DDI Interrupt Functions

Interrupt Capability Functions

Interrupt Initialization and Destruction Functions

Priority Management Functions

Soft Interrupt Functions

Interrupt Function Examples

Registering Interrupts

Registering Legacy Interrupts

Registering MSI Interrupts

Interrupt Resource Management

The Interrupt Resource Management Feature

Callback Interfaces

Unregister a Callback Handler Function

Callback Handler Function

Interrupt Request Interfaces

Allocate an Interrupt

Modify Number of Interrupt Vectors Requested

Interrupt Usage and Flexibility

Example Implementation of Interrupt Resource Management

Interrupt Handler Functionality

Handling High-Level Interrupts

High-Level Mutexes

High-Level Interrupt Handling Example

Chapter 9 Direct Memory Access (DMA)

Types of Host Platform DMA

DMA Software Components: Handles, Windows, and Cookies

DMA Operations

Performing Bus-Master DMA Transfers

Performing First-Party DMA Transfers

Performing Third-Party DMA Transfers

DMA Attributes

ddi_dma_attr Structure

ISA Bus Example

Managing DMA Resources

Object Locking

Allocating a DMA Handle

Allocating DMA Resources

Device Register Structure

DMA Callback Example

Determining Maximum Burst Sizes

Allocating Private DMA Buffers

Handling Resource Allocation Failures

Programming the DMA Engine

Freeing the DMA Resources

Freeing the DMA Handle

Canceling DMA Callbacks

Synchronizing Memory Objects

CPU and I/O Cache

ddi_dma_sync() Function

DMA Windows

Chapter 10 Mapping Device and Kernel Memory

Memory Mapping Overview

Exporting the Mapping

The segmap(9E) Entry Point

The devmap(9E) Entry Point

Associating Device Memory With User Mappings

Associating Kernel Memory With User Mappings

Allocating Kernel Memory for User Access

Exporting Kernel Memory to Applications

Freeing Kernel Memory Exported for User Access

Chapter 11 Device Context Management

Introduction to Device Context

What Is a Device Context?

Context Management Model

Context Management Operation

devmap_callback_ctl Structure

Entry Points for Device Context Management

devmap_map() Entry Point

devmap_access() Entry Point

devmap_contextmgt() Entry Point

devmap_dup() Entry Point

devmap_unmap() Entry Point

Associating User Mappings With Driver Notifications

Managing Mapping Accesses

devmap_load() Entry Point

devmap_unload() Entry Point

Chapter 12 Power Management

Power Management Framework

Device Power Management

System Power Management

Device Power Management Model

Power Management Components

Multiple Power Management Components

Power Management States

Power Levels

Power Management Dependencies

Automatic Power Management for Devices

Device Power Management Interfaces

Busy-Idle State Transitions

Device Power State Transitions

power() Entry Point

System Power Management Model

Autoshutdown Threshold

Busy State

Hardware State

Automatic Power Management for Systems

Entry Points Used by System Power Management

detach() Entry Point

attach() Entry Point

Power Management Device Access Example

Power Management Flow of Control

Changes to Power Management Interfaces

Chapter 13 Hardening Oracle Solaris Drivers

Oracle Fault Management Architecture I/O Fault Services

What Is Predictive Self-Healing?

Oracle Solaris Fault Manager

Diagnosis, Suspect Lists, and Fault Events

Response Agents

Message IDs and Dictionary Files

System Topology

Error Handling

Declaring Fault Management Capabilities

Cleaning Up Fault Management Resources

Getting the Fault Management Capability Bit Mask

Reporting Errors

Chapter 14 Layered Driver Interface (LDI)

LDI Overview

Kernel Interfaces

Layered Identifiers – Kernel Device Consumers

Layered Driver Handles – Target Devices

Opening and Closing Target Devices

Accessing Target Devices

Retrieving Target Device Information

Retrieving Target Device Property Values

Receiving Asynchronous Device Event Notification

LDI Kernel Interfaces Example

Device Configuration File

Driver Source File

How to Build and Load the Layered Driver

Test the Layered Driver

User Interfaces

Device Information Library Interfaces

Print System Configuration Command Interfaces

Device User Command Interfaces

Part II Designing Specific Kinds of Device Drivers

Chapter 15 Drivers for Character Devices

Overview of the Character Driver Structure

Character Device Autoconfiguration

Device Access (Character Drivers)

open() Entry Point (Character Drivers)

close() Entry Point (Character Drivers)

I/O Request Handling

User Addresses

Vectored I/O

Differences Between Synchronous and Asynchronous I/O

Data Transfer Methods

Programmed I/O Transfers

DMA Transfers (Synchronous)

DMA Transfers (Asynchronous)

minphys() Entry Point

strategy() Entry Point

Mapping Device Memory

Multiplexing I/O on File Descriptors

Miscellaneous I/O Control

ioctl() Entry Point (Character Drivers)

I/O Control Support for 64-Bit Capable Device Drivers

Handling copyout() Overflow

32-bit and 64-bit Data Structure Macros

How Do the Structure Macros Work?

When to Use Structure Macros

Declaring and Initializing Structure Handles

Operations on Structure Handles

Other Operations

Chapter 16 Drivers for Block Devices

Block Driver Structure Overview

File I/O

Block Device Autoconfiguration

Controlling Device Access

open() Entry Point (Block Drivers)

close() Entry Point (Block Drivers)

strategy() Entry Point

buf Structure

bp_mapin Structure

Synchronous Data Transfers (Block Drivers)

Asynchronous Data Transfers (Block Drivers)

Checking for Invalid buf Requests

Enqueuing the Request

Starting the First Transfer

Handling the Interrupting Device

dump() and print() Entry Points

dump() Entry Point (Block Drivers)

print() Entry Point (Block Drivers)

Disk Device Drivers

Disk ioctls

Disk Performance

Chapter 17 SCSI Target Drivers

Introduction to Target Drivers

Sun Common SCSI Architecture Overview

General Flow of Control

SCSA Functions

Hardware Configuration File

Declarations and Data Structures

scsi_device Structure

scsi_pkt Structure (Target Drivers)

Autoconfiguration for SCSI Target Drivers

probe() Entry Point (SCSI Target Drivers)

attach() Entry Point (SCSI Target Drivers)

detach() Entry Point (SCSI Target Drivers)

getinfo() Entry Point (SCSI Target Drivers)

Resource Allocation

scsi_init_pkt() Function

scsi_sync_pkt() Function

scsi_destroy_pkt() Function

scsi_alloc_consistent_buf() Function

scsi_free_consistent_buf() Function

Building and Transporting a Command

Building a Command

Setting Target Capabilities

Transporting a Command

Synchronous scsi_transport() Function

Command Completion

Reuse of Packets

Auto-Request Sense Mode

Dump Handling

SCSI Options

Chapter 18 SCSI Host Bus Adapter Drivers

Introduction to Host Bus Adapter Drivers

SCSI Interface

SCSA HBA Interfaces

SCSA HBA Entry Point Summary

SCSA HBA Data Structures

scsi_hba_tran() Structure

scsi_address Structure

scsi_device Structure

scsi_pkt Structure (HBA)

Per-Target Instance Data

Transport Structure Cloning

SCSA HBA Functions

HBA Driver Dependency and Configuration Issues

Declarations and Structures

Per-Command Structure

Entry Points for Module Initialization

_init() Entry Point (SCSI HBA Drivers)

_fini() Entry Point (SCSI HBA Drivers)

Autoconfiguration Entry Points

attach() Entry Point (SCSI HBA Drivers)

detach() Entry Point (SCSI HBA Drivers)

Entry Points for SCSA HBA Drivers

Target Driver Instance Initialization

tran_tgt_init() Entry Point

tran_tgt_probe() Entry Point

tran_tgt_free() Entry Point

Resource Allocation

tran_init_pkt() Entry Point

Allocation and Initialization of a Structure

Allocation of DMA Resources

Reallocation of DMA Resources for Data Transfer

tran_destroy_pkt() Entry Point

tran_sync_pkt() Entry Point

tran_dmafree() Entry Point

Command Transport

tran_start() Entry Point

Interrupt Handler and Command Completion

Timeout Handler

Capability Management

tran_getcap() Entry Point

tran_setcap() Entry Point

Abort and Reset Management

tran_abort() Entry Point

tran_reset() Entry Point

tran_bus_reset() Entry Point

tran_reset_notify() Entry Point

Dynamic Reconfiguration

SCSI HBA Driver Specific Issues

Installing HBA Drivers

HBA Configuration Properties

scsi-reset-delay Property

scsi-options Property

Per-Target scsi-options

x86 Target Driver Configuration Properties

Support for Queuing

Chapter 19 Drivers for Network Devices

GLDv3 Network Device Driver Framework

GLDv3 MAC Registration

GLDv3 MAC Registration Process

GLDv3 MAC Registration Functions

GLDv3 MAC Registration Data Structures

GLDv3 Capabilities

MAC Rings Capability

Hardware Checksum Offload

Large Segment (or Send) Offload

GLDv3 Data Paths

Transmit Data Path

Receive Data Path

GLDv3 State Change Notifications

GLDv3 Network Statistics

GLDv3 Properties

Summary of GLDv3 Interfaces

GLDv2 Network Device Driver Framework

GLDv2 Device Support

Ethernet V2 and ISO 8802-3 (IEEE 802.3)

TPR and FDDI: SNAP Processing

TPR: Source Routing

GLDv2 DLPI Providers

GLDv2 DLPI Primitives

GLDv2 I/O Control Functions

GLDv2 Driver Requirements

GLDv2 Network Statistics

GLDv2 Declarations and Data Structures

gld_mac_info Structure

gld_stats Structure

GLDv2 Function Arguments

GLDv2 Entry Points

gldm_reset() Entry Point

gldm_start() Entry Point

gldm_stop() Entry Point

gldm_set_mac_addr() Entry Point

gldm_set_multicast() Entry Point

gldm_set_promiscuous() Entry Point

gldm_send() Entry Point

gldm_intr() Entry Point

gldm_get_stats() Entry Point

gldm_ioctl() Entry Point

GLDv2 Return Values

GLDv2 Service Routines

gld_mac_alloc() Function

gld_mac_free() Function

gld_register() Function

gld_unregister() Function

gld_recv() Function

gld_sched() Function

gld_intr() Function

Chapter 20 USB Drivers

USB in the Oracle Solaris Environment

USBA 2.0 Framework

USB Client Drivers

Binding Client Drivers

How USB Devices Appear to the System

USB Devices and the Oracle Solaris Device Tree

Compatible Device Names

Devices With Multiple Interfaces

Devices With Interface-Association Descriptors

Checking Device Driver Bindings

Basic Device Access

Before the Client Driver Is Attached

The Descriptor Tree

Registering Drivers to Gain Device Access

Synchronous and Asynchronous Transfers and Callbacks

Request Allocation and Deallocation

Flushing Pipes

Device State Management

Hotplugging USB Devices

Device Power Management

System Power Management

Serialization Strategy

Utility Functions

Device Configuration Facilities

Getting Interface Numbers

Managing Entire Devices

Multiple-Configuration Devices

Modifying or Getting the Alternate Setting

Other Utility Functions

Retrieving a String Descriptor

Pipe Private Data Facility

Clearing a USB Condition

Getting Device, Interface, or Endpoint Status

Getting the Bus Address of a Device

Sample USB Device Drivers

Chapter 21 SR-IOV Drivers

Introduction to SR-IOV

Benefits of SR-IOV

Supported Platforms

Glossary

Overview of SR-IOV Device Driver

Physical Function (PF) Driver

Virtual Function (VF) Driver

Device Configuration Parameters

pci.conf File

Setting Device Configuration Parameters

SR-IOV Configuration on Sparc OVM Platform

SR-IOV Configuration on Bare Metal Platforms

Boot Configuration Sequence

SR-IOV Interfaces Summary

Driver Ioctls

Interfaces for SR-IOV Drivers

pci_param_get() Interface

pci_param_get_ioctl() Interface

pci_plist_get() Interface

pci_plist_getvf() Interface

pciv_vf_config() Interface

pci_plist_lookup() Interface

pci_param_free() Interface

pciv_send() Interface

SR-IOV Driver Ioctls

Data Structures

iov_param_ver_info Structure

iov_param_validate Structure

iov_param_desc Structure

IOV_GET_VER_INFO Ioctl

IOV_GET_PARAM_INFO Ioctl

IOV_VALIDATE_PARAM Ioctl

Sample Code for Driver Ioctls

Driver Callbacks

Fibre Channel SR-IOV Devices

Configuring a Fibre Channel Virtual Function

Updating a Fibre Channel Virtual Function

Unconfiguring a Fibre Channel Virtual Function

Bandwidth Configuration for Fibre Channel Virtual Functions

Configuring SR-IOV Fibre Channel Devices on SPARC Platforms

Configuring SR-IOV Fibre Channel Devices on x86 Platforms

Part III Building a Device Driver

Chapter 22 Compiling, Loading, Packaging, and Testing Drivers

Driver Development Summary

Preparing for Driver Installation

Compiling and Linking the Driver

Module Dependencies

Writing a Hardware Configuration File

Installing, Updating, and Removing Drivers

Copying the Driver to a Module Directory

Installing Drivers with add_drv

Updating Driver Information

Removing the Driver

Loading and Unloading Drivers

Driver Packaging

Criteria for Testing Drivers

Configuration Testing

Functionality Testing

Error Handling

Testing Loading and Unloading

Stress, Performance, and Interoperability Testing

DDI/DKI Compliance Testing

Installation and Packaging Testing

Testing Specific Types of Drivers

Testing Tape Drivers

Testing Disk Drivers

Asynchronous Communication Drivers

Testing Network Drivers

Testing SR-IOV Drivers

Chapter 23 Debugging, Testing, and Tuning Device Drivers

Testing Drivers

Using the Deadman Feature

Testing With a Serial Connection

To Set Up the Host for a tip Connection

Setting Up a Target System on the SPARC Platform

Setting Up a Target System on the x86 Platform

Setting Up Test Modules

Setting Kernel Variables

Loading and Unloading Test Modules

Setting kmem_flags Debugging Flags

Avoiding Data Loss on a Test System

Using an Alternate Boot Environment

Booting With an Alternate Kernel

Consider Alternative Back-Up Plans

Capture System Crash Dumps

Debugging Tools

Using KADI

Postmortem Debugging

Using the kmdb Kernel Debugger

Booting kmdb With an Alternate Kernel on the SPARC Platform

Booting kmdb With an Alternate Kernel on the x86 Platform

Setting Breakpoints in kmdb

kmdb Macros for Driver Developers

Using the mdb Modular Debugger

Getting Started With the Modular Debugger

Useful Debugging Tasks With kmdb and mdb

Exploring System Registers With kmdb

Detecting Kernel Memory Leaks

Writing Debugger Commands With mdb

Obtaining Kernel Data Structure Information

Obtaining Device Tree Information

Retrieving Driver Soft State Information

Modifying Kernel Variables

Tuning Drivers

Kernel Statistics

Optional Kernel Statistics

Optional Kstats Persistence

Kernel Statistics Structure Members

Kernel Statistics Structures

Kernel Statistics Functions

Kernel Statistics for Oracle Solaris Ethernet Drivers

Trace for Dynamic Instrumentation

Chapter 24 Recommended Coding Practices

Debugging Preparation Techniques

Use a Unique Prefix to Avoid Kernel Symbol Collisions

Use cmn_err() to Log Driver Activity

Use ASSERT() to Catch Invalid Assumptions

Use mutex_owned() to Validate and Document Locking Requirements

Use Conditional Compilation to Toggle Costly Debugging Features

Declaring a Variable Volatile

Driver Serviceability

Periodic Health Checks

Part IV Appendixes for Writing Device Drivers

Appendix A Hardware Overview

SPARC Processor Issues

SPARC Data Alignment

Member Alignment in SPARC Structures

SPARC Byte Ordering

SPARC Register Windows

SPARC Multiply and Divide Instructions

x86 Processor Issues

x86 Byte Ordering

x86 Architecture Manuals

Endianness, Byte Ordering

Store Buffers

System Memory Model

Total Store Ordering (TSO)

Partial Store Ordering (PSO)

Bus Architectures

Device Identification

Supported Interrupt Types

Bus Specifics

PCI Local Bus

PCI Address Domain

PCI Configuration Address Space

PCI Configuration Base Address Registers

PCI Memory Address Space

PCI I/O Address Space

PCI Hardware Configuration Files

PCI Express

Device Issues

Timing-Critical Sections

Chip Delays

Internal Sequencing Logic

Interrupt Issues

PROM on SPARC Systems

Open Boot PROM 3

Forth Commands

Walking the PROMs Device Tree

Mapping the Device

Reading and Writing

Appendix B Summary of Oracle Solaris DDI/DKI Services

Module Functions

Device Information Tree Node (dev_info_t) Functions

Device (dev_t) Functions

Property Functions

Device Software State Functions

Memory Allocation and Deallocation Functions

Kernel Thread Control and Synchronization Functions

Task Queue Management Functions

Interrupt Functions

Programmed I/O Functions

Direct Memory Access (DMA) Functions

User Space Access Functions

User Process Event Functions

User Process Information Functions

User Application Kernel and Device Access Functions

Time-Related Functions

Power Management Functions

Fault Management Functions

Kernel Statistics Functions

Kernel Logging and Printing Functions

Buffered I/O Functions

Virtual Memory Functions

Device ID Functions

SCSI Functions

Resource Map Management Functions

System Global State

Utility Functions

Appendix C Making a Device Driver 64-Bit Ready

Introduction to 64-Bit Driver Design

General Conversion Steps

Use Fixed-Width Types for Hardware Registers

Use Fixed-Width Common Access Functions

Check and Extend Use of Derived Types

Check Changed Fields in DDI Data Structures

buf Structure Changes

ddi_dma_attr Structure

ddi_dma_cookie Structure Changes

csi_arq_status Structure Changes

scsi_pkt Structure Changes

Check Changed Arguments of DDI Functions

getrbuf() Argument Changes

drv_getparm() Argument Changes

delay() and timeout() Argument Changes

rmallocmap() and rmallocmap_wait() Argument Changes

scsi_alloc_consistent_buf() Argument Changes

uiomove() Argument Changes

cv_timedwait() and cv_timedwait_sig() Argument Changes

ddi_device_copy() Argument Changes

ddi_device_zero() Argument Changes

ddi_dma_mem_alloc() Argument Changes

Modify Routines That Handle Data Sharing

Data Sharing in ioctl()

Data Sharing in devmap()

Data Sharing in mmap()

Check Structures with 64-bit Long Data Types on x86-Based Platforms

Well Known ioctl Interfaces

Device Sizes

Appendix D Console Frame Buffer Drivers

Oracle Solaris Consoles and the Kernel Terminal Emulator

x86 Platform Console Communication

SPARC Platform Console Communication

Console Visual I/O Interfaces

I/O Control Interfaces

Polled I/O Interfaces

Video Mode Change Callback Interface

Implementing the Visual I/O Interfaces in Console Frame Buffer Drivers

VIS_DEVINIT Macro

VIS_DEFINI Macro

VIS_CONSDISPLAY Macro

Implementing Polled I/O in Console Frame Buffer Drivers

Frame Buffer Specific Configuration Module

The X Window System Frame Buffer Specific DDX Module

Developing, Testing, and Debugging Console Frame Buffer Drivers

Testing the I/O Control Interfaces

Testing the Polled I/O Interfaces

Testing the Video Mode Change Callback Function

Additional Suggestions for Testing Console Frame Buffer Drivers

Appendix E pci.conf File

pci.conf Description

System Configuration Section

Device Configuration Section

pci.conf Syntax

pci.conf References

Appendix F Supported FC-HBA API

Supported Fibre Channel HBA API

Appendix G Equivalent kstat2 APIs for kstat

Equivalent kstat2 APIs for kstat

kstat2 Data Structures

Index

Index Numbers and Symbols