If the PCIe card is assigned to an I/O domain on a logical domain, use these steps to avoid a configuration that is unsupported by the SR-IOV feature.
If the PCIe card is not assigned to an I/O domain on a logical domain, go to Remove a PCIe Card Carrier and Card From the Server.
In this example, pci_45 represents the I/O domain name.
# ldm ls-io -l pci_45
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
pci_45 BUS pci_45 primary IOV
[pci@32d]
/SYS/CMIOU9/PCIE2 PCIE pci_45 primary OCC
[pci@32d/pci@1]
SUNW,qlc@0/fp/disk
SUNW,qlc@0/fp@0,0
SUNW,qlc@0,1/fp/disk
SUNW,qlc@0,1/fp@0,0/ssd@w20350080e518b846,1f
fibre-channel@0,2
fibre-channel@0,3
fibre-channel@0,4
fibre-channel@0,5
...
/SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 VF pci_45 ldg4
[pci@32d/pci@1/SUNW,qlc@0,2]
Class properties [FIBRECHANNEL]
port-wwn = 10:00:00:14:4f:f8:74:85
node-wwn = 20:00:00:14:4f:f8:74:85
bw-percent = 0
...
/SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 VF pci_45 ldg4
[pci@32d/pci@1/SUNW,qlc@0,5]
Class properties [FIBRECHANNEL]
port-wwn = 10:00:00:14:4f:fb:27:29
node-wwn = 20:00:00:14:4f:fb:27:29
bw-percent = 0
#
Before removing any virtual functions, save the original configuration in a file you can use to restore the configuration after you replace the card.
# ldm ls-io slot-name>file-name
where:
slot-name is a format similar to /SYS/CMIOU9/PCIE2
file-name is a format similar to /var/tmp/before_removal.txt
Display these details from the backup file you created in the previous step, for example:
# cat /var/tmp/before_removal.txt NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ /SYS/CMIOU9/PCIE2 PCIE pci_45 primary OCC /SYS/CMIOU9/PCIE2/IOVFC.PF0 PF pci_45 primary /SYS/CMIOU9/PCIE2/IOVFC.PF1 PF pci_45 primary /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 VF pci_45 ldg4 /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF1 VF pci_45 ldg4 /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF2 VF pci_45 ldg4 /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 VF pci_45 ldg4
Typically, a card has two or more physical functions, so you must ensure that all virtual functions from all physical functions are removed from the I/O domains. As shown in the preceding example, these virtual functions are indicated under the TYPE column as VF.
# ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 ldg4 # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF1 ldg4 # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF2 ldg4 # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 ldg4
# ldm ls-io /SYS/CMIOU9/PCIE2 NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ /SYS/CMIOU9/PCIE2 PCIE pci_45 primary OCC /SYS/CMIOU9/PCIE2/IOVFC.PF0 PF pci_45 primary /SYS/CMIOU9/PCIE2/IOVFC.PF1 PF pci_45 primary /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 VF pci_45 /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF1 VF pci_45 /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF2 VF pci_45 /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 VF pci_45
In this example, the virtual functions listed under the TYPE column now display no domain name.
# ldm stop ldg4 LDom ldg4 stopped # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 ldg4 # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF1 ldg4 # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF2 ldg4 # ldm rm-io /SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 ldg4
If the replacement PCIe card is the same type and model as the original card, and if virtual functions were created from physical functions on that card, save the properties of any virtual function instance for the PCIe card. You can use the file you create to reproduce the original configuration on the new card.
# ldm ls-io -l slot-name>file-name
where:
slot-name is a format similar to /SYS/CMIOU9/PCIE2
file-name is a format similar to /var/tmp/vf_config.txt
Display the details from the backup file you created in the previous step, for example:
# cat /var/tmp/vf_config.txt
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
pci_45 BUS pci_45 primary IOV
[pci@32d]
/SYS/CMIOU9/PCIE2 PCIE pci_45 primary OCC
[pci@32d/pci@1]
SUNW,qlc@0/fp/disk
SUNW,qlc@0/fp@0,0
SUNW,qlc@0,1/fp/disk
SUNW,qlc@0,1/fp@0,0/ssd@w20350080e518b846,1f
fibre-channel@0,2
fibre-channel@0,3
fibre-channel@0,4
fibre-channel@0,5
...
/SYS/CMIOU9/PCIE2/IOVFC.PF0.VF0 VF pci_45 ldg4
[pci@32d/pci@1/SUNW,qlc@0,2]
Class properties [FIBRECHANNEL]
port-wwn = 10:00:00:14:4f:f8:74:85
node-wwn = 20:00:00:14:4f:f8:74:85
bw-percent = 0
...
/SYS/CMIOU9/PCIE2/IOVFC.PF0.VF3 VF pci_45 ldg4
[pci@32d/pci@1/SUNW,qlc@0,5]
Class properties [FIBRECHANNEL]
port-wwn = 10:00:00:14:4f:fb:27:29
node-wwn = 20:00:00:14:4f:fb:27:29
bw-percent = 0
#
You can do this dynamically while the root domain is running.
# ldm destroy-vf -n max physical-function-name
where physical-function-name is a format similar to /SYS/CMIOU9/PCIE2/IOVFC.PF0.
Display the details for the slot, for example:
# ldm ls-io -l /SYS/CMIOU9/PCIE2
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
/SYS/CMIOU9/PCIE2 PCIE pci_45 primary OCC
[pci@32d/pci@1]
SUNW,qlc@0/fp/disk
SUNW,qlc@0/fp@0,0
SUNW,qlc@0,1/fp/disk
SUNW,qlc@0,1/fp@0,0/ssd@w20350080e518b846,1f
/SYS/CMIOU9/PCIE2/IOVFC.PF0 PF pci_45 primary
[pci@32d/pci@1/SUNW,qlc@0]
maxvfs = 16
/SYS/CMIOU9/PCIE2/IOVFC.PF1 PF pci_45 primary
[pci@32d/pci@1/SUNW,qlc@0,1]
maxvfs = 16
#
# ldm start-reconf primary Initiating a delayed reconfiguration operation on the primary domain. All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. # # ldm destroy-vf -n max /SYS/CMIOU9/PCIE2/IOVFC.PF0 ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ # shutdown -i6 -g0 -y
Use the ldm stop -r command, instead of the shutdown command, to stop the other root domain.
See Remove a PCIe Card Carrier and Card From the Server.
For more information about making PCIe hardware changes, refer to the Oracle VM Server for SPARC documentation at http://www.oracle.com/goto/vm-sparc/docs.