In order for a Fibre Channel PF driver to add support for SR-IOV, it must call ddi_cb_register() and include the DDI_CB_FLAG_SRIOV flag. See section Driver Callbacks for more information.
A Fibre Channel PF driver that has registered for SR-IOV callbacks will receive callbacks with DDI_CB_PCIV_CLASS_CONFIG as the action parameter. The callback includes the cbarg parameter, a pointer to a fciov_conf_t structure:
typedef struct fciov_conf
{
fciov_config_cmd_t fci_cmd; /* configuration op to be performed */
uint16_t fci_vf_id; /* Index of the VF to be configured */
fciov_cfg_flags fci_flags; /* Flags, indicate which info is valid */
uint8_t fci_node_wwn[8];/* node WWN for the VF */
uint8_t fci_port_wwn[8];/* port WWN for the VF */
uint16_t fci_bandwidth; /* percentile bandwidth for the VF */
} fciov_conf_t;
Indicates what operation to perform for the specified VF. Valid values are defined as follows:
typedef enum
{
FCIOV_VF_CONFIG = 0x1, /* Configure a FC VF */
FCIOV_VF_UPDCONFIG = 0x2,/* Update configuration of FC VF */
FCIOV_VF_UNCONFIG = 0x4 /* Unconfigure a FC VF*/
}fciov_config_cmd_t;
An index number of a VF for the PF that is receiving the callback. Valid values for this field are 0 to N-1, where N is the number of virtual functions enabled in the receiving PF.
A bitmask that indicates which of the included data fields are valid. Supported values are defined as follows:
typedef enum
{
FCIOV_NODE_WWN = 0x1, /* node WWN info is valid */
FCIOV_PORT_WWN = 0x2, /* port WWN info is valid */
FCIOV_BANDWIDTH = 0x4 /* bandwidth info is valid */
}fciov_cfg_flags;
The node world wide name (WWN) of the specified VF.
The port world wide name (WWN) of the specified VF.
Indicates the bandwidth in percentile that needs to be allocated to the virtual function. For more information, see Bandwidth Configuration for Fibre Channel Virtual Functions.
At some point after an fibre channel PF driver has successfully attached and has enabled the required number of virtual functions, the IOV framework will issue a callback to the PF driver so that it can complete the configuration of a specified VF. In this callback, the action parameter is set to DDI_CB_PCIV_CLASS_CONFIG, the cbarg parameter points to a fciov_conf_t structure containing information about the VF, and the fc_cmd field of the fciov_conf_t structure is set to FCIOV_VF_CONFIG.
There is no guarantee of the timing or order in which this callback is received for a set of virtual functions associated with a PF. The IOV framework ensures that this callback is issued for a VF before it is made available for use. A callback to configure a fibre channel VF must have a valid node WWN and a valid port WWN. Thus, the fci_flags will always have FCIOV_NODE_WWN and FCIOV_PORT_WWN set. The bandwidth specification is optional. See, Bandwidth Configuration for Fibre Channel Virtual Functions for details on how to handle the case where the bandwidth is not specified for a configure operation.
If the configuration of an fibre channel VF is modified some time after it has been enabled and configured, the IOV framework will issue a callback to the PF driver. In this callback, the action parameter is set to DDI_CB_PCIV_CLASS_CONFIG, the cbarg parameter points to a fciov_conf_t structure containing information about the VF, and the fc_cmd field of the fciov_conf_t structure is set to FCIOV_VF_UPDCONFIG.
Upon receiving this callback, the PF driver must examine the fci_flags field to determine which of the VF attributes has changed. Based on the settings of this filed, the appropriate information must be retrieved from the data fields and used to update the VF configuration.
When a fibre channel VF is to be unconfigured, the IOV framework will issue a callback to the PF driver. In this callback, the action parameter is set to DDI_CB_PCIV_CLASS_CONFIG, the cbarg parameter points to a fciov_conf_t structure containing information about the VF, and the fc_cmd field of the fciov_conf_t structure is set to FCIOV_VF_UNCONFIG.
Upon receiving this callback, the PF driver must unconfigure all the attributes of the VF and mark it as not usable. Under normal circumstances, this callback will be followed by additional callbacks with the DDI_CB_PCIV_VF_CONFIG action to disable the VF. See section Driver Callbacks for more information.
The bandwidth value provided in a fibre channel SR-IOV callback is specified in a percentile format rather than an absolute bandwidth percentage. Specifying a percentile provides flexibility for the PF driver and the HBA firmware to tune the hardware to meet the quality of service (QoS) that is close to the specified percentile. The devices are expected to provide QoS as accurately as possible.
The bandwidth percentile applies only to the bandwidth that is available to the set of virtual functions associated with a PF. Bandwidth used by the PF itself must be reserved through some other means.
The bandwidth configuration is specified as the minimum cap that must be reserved for a particular VF. The PF driver can choose to allocate more available bandwidth to a VF if it does not cause other virtual functions to fall below their specified minimum. The following guidelines describes how a PF driver interprets the bandwidth parameters to a callback:
All virtual functions that have no explicit bandwidth percentile set or a bandwidth percentile of zero should get a fair share allocation of the bandwidth that is not reserved for any other VF.
For example, if 60% of bandwidth is allocated for one or more virtual functions, then the rest (40%) of the bandwidth will be used for fair share allocation between all virtual functions that have no bandwidth percentile specified.
The percentile of the available bandwidth for virtual functions associated with this PF.
The PF driver must ensure that if there are any virtual functions that have no bandwidth percentile specified, some bandwidth is available for those virtual functions. A situation in which a VF has no available bandwidth should be prevented.
Bandwidth allocations are not intended to be a hard requirement that must be met. A best effort attempt by the PF driver and HBA firmware is all that is expected.
On SPARC platforms, fibre channel SR-IOV is configured using OVM Server for SPARC. Use the ldm list-io command to display the fibre channel SR-IOV physical functions. These devices are identified by the value of PF in the TYPE column. In the NAME column for fibre channel physical functions, the string includes IOVFC. To create a fibre channel VF, use the ldm create-vf command:
ldm create-vf [port-wwn=<value> node-wwn=<value>] [bw_percent=[<value>]] <fc_pf_name>
The port-wwn and node-wwn arguments must both be present or both be omitted. If omitted, the Logical Domains Manager will generate valid values that do not collide with other existing WWN values. The format for these parameters are similar to the colon separated list of hex-bytes used for MAC addresses, but there must be 8 bytes rather than 6 bytes. If bw_percent is not specified, a value of 0 will be used. Valid values are 0-100. After the VF parameters are created, the ldm set-io command is used to modify them:
ldm set-io [port-wwn=<value> node-wwn=<value>]
[bw-percent=<value>] <fc_vf_name>
The ldm destroy-vf command is used to destroy a VF:
ldm destroy-vf <vf_name>
More details on this an other commands used to work with fibre channel SR-IOV can be found in the Oracle VM Server for SPARC 3.1 Administration Guide.
On x86 platforms, the /etc/pci.conf file can be used during development to specify and configure fibre channel virtual functions. All VF configuration is contained in a section called Device_Configuration. In this section, a new entry is added for each VF and the properties port-wwn, node-wwn, and bw_percent are set. This file is added to /boot/solaris/filelist.ramdisk and the system rebooted in order for the changes to take effect.
[Device_Configuration]
[[path=/pci@0,0/pci8086,340c@5/pci10df,e20e@0]]
VF[0] = {
port_wwn=0x2000123456789018
node_wwn=0x1000123456789018
bandwidth = 80
}
VF[1] = {
port_wwn=0x2000123456789019
node_wwn=0x1000123456789019
bandwidth = 20
}
[[path=/pci@0,0/pci8086,340c@5/pci10df,e20e@0,1]]
VF[0] = {
port_wwn=0x2000123456789028
node_wwn=0x1000123456789028
bandwidth = 20
}
VF[1] = {
port_wwn=0x2000123456789029
node_wwn=0x1000123456789029
bandwidth = 80
}