Go to main content

Writing Device Drivers for Oracle® Solaris 11.3

Exit Print View

Updated: March 2019

Overview of SR-IOV Device Driver

The SR-IOV functionality is comprised of the Physical Function (PF) driver and the Virtual Function (VF) driver. The following sections describe the PF and VF drivers and the details of the required device configuration.

Physical Function (PF) Driver

The PF driver of a SR-IOV device is used to manage the Physical Function (PF) of an SR-IOV capable device. A PCI Function that supports the SR-IOV capabilities is defined in the SR-IOV specification. A PF contains the SR-IOV capability structure and is used to manage the SR-IOV functionality. PFs are fully featured PCIe functions which can be discovered, managed, and manipulated like any other PCIe device. PFs have full configuration resources, and can be used to configure or control the PCIe device. A PF driver exhibits the following characteristics:

  • Visible in Root Domain only

  • May or may not possess data movement capabilities. The PF driver should function even in SR-IOV mode.

  • Controls the enabling and disabling of the SR-IOV capability through APIs provided by the Oracle Solaris IOV Framework.

  • The number of VFs to be configured for a given PF is determined by the system administrator. This number defined either in Machine Descriptors (MD) on the SPARC OVM platforms or a configuration file in the bare metal environment.

  • The PF driver enables the VFs during the attachment phase by calling the Oracle Solaris IOV framework through the DDI interface. If the PF driver does not enable VFs during attachment, the Oracle Solaris IOV framework will attempt to configure VFs after the attachment as long as the driver callback flags indicates that the driver has IOV capability is supported.

  • PFs can enable and disable each associated VF individually through a device specific mechanism

Virtual Function (VF) Driver

    A function that is associated with a Physical Function. A VF is a lightweight PCIe function that shares one or more physical resources with the Physical Function and with other VFs that are associated with the same PF. A VF driver exhibits the following characteristics:

  • Visible in both the root domain and I/O domain

  • Can initiate communication with its PF either through the HW mailbox or an OS–provided interface

  • Not visible in the root domain until the following conditions are satisfied:

    • The root domain has booted up

    • The PF driver attaches and invokes the configuration of the VF

    • The VFs are enabled by the root domain's Oracle Solaris IOV framework

    • The system firmware allocates resources to the VFs

  • Not visible in the I/O Domain until the following conditions are satisfied:

    • VFs are already enabled and are visible in the root domain

    • VFs are assigned to the I/O Domain

    • VFs are probed in the I/O Domain by the Oracle Solaris firmware (OBP)

Note - The SR-IOV Capable PF and VF drivers have to register Interrupt Resource Management (IRM) callbacks and provide support for this feature. See Interrupt Handlers for details and usage of IRM interfaces.

Note - If the VF is a network VF, the following parameters can be configured after the numVFs are enabled. The configuration should be completed before the VFs are enabled.
  • mac-addr

  • vlan (Virtual LAN) ID

  • port-vlan-id

  • alt-mac-addrs

  • mtu

Device Configuration Parameters

The PF driver must support the configuration parameters listed in the following table. These parameters may be exported to the SPARC OVM Manager. Configuration is complete only when all the parameters are configured.

Table 22  Configuration Parameters Definition
Configuration Parameters
Standards related configuration parameters
Number of VFs it can support

Note -  Changes to the number of VFs require the PF device to detach and attach again.

max-config-vfs – Maximum number of VFs that can actually be configured. A PF driver can export this parameter when the maximum number of VFs that a PF driver supports is different from the capability indicated by the SR-IOV functionality.
Resource and device-specific parameters
Bandwidth, pools, and Q-pairs. Changes to these parameters can affect both PF and VF drivers.
The framework may not be aware of device-specific parameters and they may be known only to the PF driver. These parameters should be known before enabling the VFs so that a PF driver initializes its hardware appropriately
See igb(7D) and ixgbe(7D) to learn how to obtain the device specific parameters that are exported to the IOV framework.
  • pvid-exclusive – Indicates that a port-vlan-id and vlan-ids cannot be supported at the same time.

  • max-vf-mtu – Maximum MTU allowed for a VF.

  • max-vlans – Maximum number of vlan slots supported by the network class PF driver.

Class-specific parameters
Common properties based on the class of the device. For example, Ethernet devices may have properties such as MAC addresses, VLAN-Ids, Port-VLAN-ID, bandwidth and so on.
Class specific configuration is expected to be used and may define the behavior of each of the parameters.

Note -  When device configuration parameters are changed, the devices should be reattached.

Note -  Configure the parameters in the following order before enabling the VFs. Class specific parameters will be based on the class specific configurator.
  1. Standards-related parameters

  2. Resource and device-specific parameters

  3. Class-specific parameters

pci.conf File

    The PCI configuration information file, /etc/pci.conf enables the system to save PCI configurations such as the number of VFs of a particular PF. The pci.conf file provides the following:

  • To persist the PCI configuration so that VFs can be created automatically upon booting of the system.

  • Since the configuration file is part of the boot_archive, VFs can be used during the booting of the system.

Note -  Add the /etc/pci.conf to the /boot/solaris/filelist.ramdisk file in order to include the /etc/pci.conf file into the Oracle Solaris boot process.

See pci.conf File for more information.

Setting Device Configuration Parameters

  • SPARC: The parameters can be set through the ldm command. See the ldm (1M) man page for details.

  • x86: The class-specific parameters can be specified by the pci.conf file. The following example shows the parameters set in a pci.conf file.

Example 122  Setting Device Configuration Parameters

VF[0] = {
		primary-mac-addr = 0xaabbccddeeff
		alt-mac-addrs = 0x102233445556, 0x102233445557
		vlan-id = 20, 30

VF[1] = {
		primary-mac-addr = 0xaabbccddeef1
		alt-mac-addrs = 0x102233445568
		vlan-id = 20, 30, 40, 50

SR-IOV Configuration on SPARC OVM Platform

    The SPARC OVM Manager is responsible for the SR-IOV configuration on all SPARC OVM platforms. The SPARC OVM Manager is responsible for the following operations:

  • Obtain a list of PFs that have SR-IOV capable drivers

  • Obtain the device specific parameters supported by the drivers

  • Validate a specific device configuration

  • Update the Machine Descriptor (MD) file with all valid configuration details along with the assignment and removal of VFs

The following figure shows a high-level view of the SPARC OVM configuration.

Figure 27  High-Level View of SPARC OVM Configuration

image:Figure shows a high-level view of the SPARC OVM Configuration

SR-IOV Configuration on Bare Metal Platforms

At the time of Oracle Solaris 11 release, no configuration tool configures SR-IOV on bare metal platforms, including x86.