How to Configure a Resilient I/O Domain
Use only the PCIe cards that support the I/O domain resiliency feature. See https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=REFERENCE&id=1325454.1.
Ensure that the I/O domain, root domain, service domain, and primary
domain
are all running at least the Oracle Solaris 11.4 SRU 13 OS.
Example 8-27 Using IPMP to Configure Multipathing With Ethernet SR-IOV Functions
This example shows how to use IPMP to configure network virtual-function devices for a resilient I/O domain. For more information, see Administering TCP/IP Networks, IPMP, and IP Tunnels in Oracle Solaris 11.4.
-
Identify two Ethernet SR-IOV physical functions that are assigned to different root domains.
In this example, the
root-1
androot-2
root domains have Ethernet SR-IOV physical functions.primary# ldm list-io | grep root-1 | grep PF /SYS/PCI-EM8/IOVNET.PF0 PF pci_1 root-1 primary# ldm list-io | grep root-2 | grep PF /SYS/RIO/NET2/IOVNET.PF0 PF pci_2 root-2
-
Create two Ethernet virtual functions on each of the specified physical functions.
primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0 Created new vf: /SYS/PCI-EM8/IOVNET.PF0.VF0 primary# ldm create-vf /SYS/RIO/NET2/IOVNET.PF0 Created new vf: /SYS/RIO/NET2/IOVNET.PF0.VF0
-
Assign the Ethernet virtual functions to the
io-1
I/O domain.primary# ldm add-io /SYS/PCI-EM8/IOVNET.PF0.VF0 io-1 primary# ldm add-io /SYS/RIO/NET2/IOVNET.PF0.VF0 io-1
-
Configure the Ethernet virtual functions into an IPMP group on the I/O domain.
-
Identify the newly added network devices,
net1
andnet2
, on the I/O domain.i0-1# dladm show-phys LINK MEDIA STATE SPPED DUPLEX DEVICE net0 Ethernet up 1000 full vnet0 net1 Ethernet up 1000 full igbvf0 net2 Ethernet up 1000 full igbvf1
-
Create IP interfaces for the newly added network devices.
io-1# ipadm create-ip net1 io-1# ipadm create-ip net2
-
Create the
ipmp0
IPMP group for the two network interfaces.io-1# ipadm create-ipmp -i net1 -i net2 ipmp0
-
Assign an IP address to the IPMP group.
This example configures the DHCP option.
io-1# ipadm create-addr -T dhcp ipmp0/v4
-
Check the status of the IPMP group interface.
io-1# ipmpstat -g
-
Example 8-28 Using Oracle Solaris I/O Multipathing to Configure Multipathing With Fibre Channel SR-IOV Functions
This example shows how to use Oracle Solaris I/O multipathing to configure Fibre Channel virtual-function devices for a resilient I/O domain. For more information, see Managing SAN Devices and I/O Multipathing in Oracle Solaris 11.4.
-
Identify two Fibre Channel SR-IOV physical functions that are assigned to different root domains.
In this example, the
root-1
androot-2
root domains have Fibre Channel SR-IOV physical functions.primary# ldm list-io | grep root-1 | grep PF /SYS/PCI-EM4/IOVFC.PF0 PF pci_1 root-1 primary# ldm list-io | grep root-2 | grep PF /SYS/PCI-EM15/IOVFC.PF0 PF pci_2 root-2
-
Create two virtual functions on each of the specified physical functions.
For more information, see How to Create a Fibre Channel SR-IOV Virtual Function.
primary# ldm create-vf port-wwn=10:00:00:14:4f:fc:60:00 \ node-wwn=20:00:00:14:4f:fc:60:00 /SYS/PCI-EM4/IOVFC.PF0 Created new vf: /SYS/PCI-EM4/IOVFC.PF0.VF0 primary# ldm create-vf port-wwn=10:00:00:14:4f:fc:70:00 \ node-wwn=20:00:00:14:4f:fc:70:00 /SYS/PCI-EM15/IOVFC.PF0 Created new vf: /SYS/PCI-EM15/IOVFC.PF0.VF0
-
Add the newly created virtual functions to the
io-1
I/O domain.primary# ldm add-io /SYS/PCI-EM4/IOVFC.PF0.VF0 io-1 primary# ldm add-io /SYS/PCI-EM15/IOVFC.PF0.VF0 io-1
-
Determine whether Oracle Solaris I/O multipathing is enabled on the I/O domain by using the
prtconf -v
command.If the output for the
fp
device includes the following device property setting, Oracle Solaris I/O multipathing is enabled:mpxio-disable="no"
If the
mpxio-disable
property is set toyes
, update the property value tono
in the/etc/driver/drv/fp.conf
file and then reboot the I/O domain.If the
mpxio-disable
device property does not appear in theprtconf -v
output, add thempxio-disable="no"
entry to the/etc/driver/drv/fp.conf
file and then reboot the I/O domain. -
Check the status of Oracle Solaris I/O multipathing group.
io-1# mpathadm show LU Logical Unit: /dev/rdsk/c0t600A0B80002A384600003D6B544EECD0d0s2 mpath-support: libmpscsi_vhci.so Vendor: SUN Product: CSM200_R Revision: 0660 Name Type: unknown type Name: 600a0b80002a384600003d6b544eecd0 Asymmetric: yes Current Load Balance: round-robin Logical Unit Group ID: NA Auto Failback: on Auto Probing: NA Paths: Initiator Port Name: 100000144ffc6000 Target Port Name: 201700a0b82a3846 Override Path: NA Path State: OK Disabled: no Initiator Port Name: 100000144ffc7000 Target Port Name: 201700a0b82a3846 Override Path: NA Path State: OK Disabled: no Target Port Groups: ID: 1 Explicit Failover: yes Access State: active Target Ports: Name: 201700a0b82a3846 Relative ID: 0