This procedure describes how to create an InfiniBand SR-IOV virtual function.
primary# ldm start-reconf root-domain-name
Perform this step only if I/O virtualization is not enabled already for the bus that has the physical function.
primary# ldm set-io iov=on bus
primary# ldm create-vf pf-name
You can run this command for each virtual function that you want to create. You can also use the –n option to create more than one virtual function from the same physical function in a single command. See Example 10 and the ldm(1M) man page.
Run one of the following commands:
primary# ldm stop-domain -r root-domain
primary# shutdown -i6 -g0 -y
The following example shows information about the /SYS/MB/RISER1/PCIE4/IOVIB.PF0 physical function:
This physical function is in PCIE slot 4.
The IOVIB string indicates that the physical function is an InfiniBand SR-IOV device.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_0 BUS pci_0 primary niu_0 NIU niu_0 primary /SYS/MB/RISER0/PCIE0 PCIE pci_0 primary EMP /SYS/MB/RISER1/PCIE1 PCIE pci_0 primary EMP /SYS/MB/RISER2/PCIE2 PCIE pci_0 primary EMP /SYS/MB/RISER0/PCIE3 PCIE pci_0 primary OCC /SYS/MB/RISER1/PCIE4 PCIE pci_0 primary OCC /SYS/MB/RISER2/PCIE5 PCIE pci_0 primary EMP /SYS/MB/SASHBA0 PCIE pci_0 primary OCC /SYS/MB/SASHBA1 PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/NET2 PCIE pci_0 primary OCC /SYS/MB/RISER0/PCIE3/IOVIB.PF0 PF pci_0 primary /SYS/MB/RISER1/PCIE4/IOVIB.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_0 primary
The following command shows more details about the specified physical function. The maxvfs value indicates the maximum number of virtual functions that are supported by the device.
primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0 PF pci_0 primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
maxvfs = 64
The following example shows how to create a static virtual function. First, initiate a delayed reconfiguration on the primary domain and enable I/O virtualization on the pci_0 PCIe bus. Because the pci_0 bus has been assigned already to the primary root domain, use the ldm set-io command to enable I/O virtualization.
primary# ldm start-reconf primary Initiating a delayed reconfiguration operation on the primary domain. All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. primary# ldm set-io iov=on pci_0 ----------------------------------------------------------------------- Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. -----------------------------------------------------------------------
Now, use the ldm create-vf command to create a virtual function from the /SYS/MB/RISER1/PCIE4/IOVIB.PF0 physical function.
primary# ldm create-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0 ----------------------------------------------------------------------- Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ---------------------------------------------------------------------- Created new vf: /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0
Note that you can create more than one virtual function during the same delayed reconfiguration. The following command creates a second virtual function:
primary# ldm create-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0 ----------------------------------------------------------------------- Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ----------------------------------------------------------------------- Created new vf: /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1
Finally, reboot the primary root domain to make the changes take effect.
primary# shutdown -i6 -g0 -y Shutdown started. Changing to init state 6 - please wait ...
This procedure describes how to destroy an InfiniBand SR-IOV virtual function.
A virtual function can be destroyed if it is not currently assigned to a domain. A virtual function can be destroyed only in the reverse sequential order of creation, so only the last virtual function that was created can be destroyed. The resulting configuration is validated by the physical function driver.
primary# ldm start-reconf root-domain-name
primary# ldm destroy-vf vf-name
You can run this command for each virtual function that you want to destroy. You can also use the –n option to destroy more than one virtual function from the same physical function in a single command. See Example 12 and the ldm(1M) man page.
Run one of the following commands:
primary# ldm stop-domain -r root-domain
primary# shutdown -i6 -g0 -y
The following example shows how to destroy a static InfiniBand virtual function, /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1.
The ldm list-io command shows information about the buses, physical functions, and virtual functions.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ pci_0 BUS pci_0 primary IOV ... /SYS/MB/RISER1/PCIE4/IOVIB.PF0 PF pci_0 primary ... /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0 VF pci_0 /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1 VF pci_0
You can obtain more details about the physical function and related virtual functions by using the ldm list-io -l command.
primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0 PF pci_0 primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
maxvfs = 64
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0 VF pci_0
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,1]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1 VF pci_0
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,2]
A virtual function can be destroyed only if it is unassigned to a domain. The DOMAIN column of the ldm list-io -l output shows the name of any domain to which a virtual function is assigned. Also, virtual functions must be destroyed in the reverse order of their creation. Therefore, in this example, you must destroy the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1 virtual function before you can destroy the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0 virtual function.
After you identify the proper virtual function, you can destroy it. First, initiate a delayed reconfiguration.
primary# ldm start-reconf primary Initiating a delayed reconfiguration operation on the primary domain. All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. primary# ldm destroy-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1 ----------------------------------------------------------------------- Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. -----------------------------------------------------------------------
You can issue more than one ldm destroy-vf command while in delayed reconfiguration. Thus, you could also destroy the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0.
Finally, reboot the primary root domain to make the changes take effect.
primary# shutdown -i6 -g0 -y Shutdown started. Changing to init state 6 - please wait ...