How to Create an I/O Domain by Assigning a PCIe Endpoint Device
Plan all DIO deployments ahead of time to minimize downtime.
 | Caution - The primary domain loses access to the on-board DVD device if you assign the /SYS/MB/SASHBA1 slot on a SPARC T3-1 or a SPARC T4-1 system to a DIO domain. The SPARC T3-1 and SPARC T4-1 systems include two DIO slots for on-board storage, which are represented by the /SYS/MB/SASHBA0 and /SYS/MB/SASHBA1 paths. In addition to hosting multiheaded on-board disks, the /SYS/MB/SASHBA1 slot hosts the on-board DVD device. So, if you assign /SYS/MB/SASHBA1 to a DIO domain, the primary domain loses access to the on-board DVD device. The SPARC T3-2 and SPARC T4-2 systems have a single SASHBA slot that hosts all on-board disks as well as the on-board DVD device. So, if you assign SASHBA to a DIO domain, the on-board disks and the on-board DVD device are loaned to the DIO domain and unavailable to the primary domain. |
For an example of adding a PCIe endpoint device to create an I/O domain, see Planning PCIe Endpoint Device Configuration.
Note - In this release, use the DefaultFixed NCP to configure datalinks and network interfaces on Oracle Solaris 11 systems.
DefaultFixed – Enables you to use the dladm or ipadm command to manage networking
Automatic – Enables you to use the netcfg or netadm command to manage networking
The Oracle Solaris 11 OS includes the following NCPs:
Ensure that the DefaultFixed NCP is enabled by using the netadm list command. See Chapter 7, Using Datalink and Interface Configuration Commands on Profiles, in Oracle Solaris Administration: Network Interfaces and Network Virtualization .
- Identify and archive the devices that are currently installed on the system.
The output of the ldm list-io -l command shows how the I/O devices are currently configured. You can obtain more detailed information by using the prtdiag -v command.
Note - After the devices are assigned to I/O domains, the identity of the devices can be determined only in the I/O domains.primary# ldm list-io -l NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary [niu@480] niu_1 NIU niu_1 primary [niu@580] pci_0 BUS pci_0 primary [pci@400] pci_1 BUS pci_1 primary [pci@500] /SYS/MB/PCIE0 PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@8] SUNW,emlxs@0/fp/disk SUNW,emlxs@0/fp/tape SUNW,emlxs@0/fp@0,0 SUNW,emlxs@0,1/fp/disk SUNW,emlxs@0,1/fp/tape SUNW,emlxs@0,1/fp@0,0 /SYS/MB/PCIE2 PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@4] pci/scsi/disk pci/scsi/tape pci/scsi/disk pci/scsi/tape /SYS/MB/PCIE4 PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@0] ethernet@0 ethernet@0,1 SUNW,qlc@0,2/fp/disk SUNW,qlc@0,2/fp@0,0 SUNW,qlc@0,3/fp/disk SUNW,qlc@0,3/fp@0,0 /SYS/MB/PCIE6 PCIE pci_0 primary EMP [pci@400/pci@1/pci@0/pci@8] /SYS/MB/PCIE8 PCIE pci_0 primary EMP [pci@400/pci@1/pci@0/pci@c] /SYS/MB/SASHBA PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@e] scsi@0/iport@1 scsi@0/iport@2 scsi@0/iport@4 scsi@0/iport@8 scsi@0/iport@80/cdrom@p7,0 scsi@0/iport@v0 /SYS/MB/NET0 PCIE pci_0 primary OCC [pci@400/pci@1/pci@0/pci@4] network@0 network@0,1 /SYS/MB/PCIE1 PCIE pci_1 primary OCC [pci@500/pci@2/pci@0/pci@a] SUNW,qlc@0/fp/disk SUNW,qlc@0/fp@0,0 SUNW,qlc@0,1/fp/disk SUNW,qlc@0,1/fp@0,0 /SYS/MB/PCIE3 PCIE pci_1 primary OCC [pci@500/pci@2/pci@0/pci@6] network@0 network@0,1 network@0,2 network@0,3 /SYS/MB/PCIE5 PCIE pci_1 primary OCC [pci@500/pci@2/pci@0/pci@0] network@0 network@0,1 /SYS/MB/PCIE7 PCIE pci_1 primary EMP [pci@500/pci@1/pci@0/pci@6] /SYS/MB/PCIE9 PCIE pci_1 primary EMP [pci@500/pci@1/pci@0/pci@0] /SYS/MB/NET2 PCIE pci_1 primary OCC [pci@500/pci@1/pci@0/pci@5] network@0 network@0,1 ethernet@0,80 /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary [pci@400/pci@1/pci@0/pci@4/network@0] maxvfs = 7 /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary [pci@400/pci@1/pci@0/pci@4/network@0,1] maxvfs = 7 /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary [pci@500/pci@2/pci@0/pci@0/network@0] maxvfs = 63 /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary [pci@500/pci@2/pci@0/pci@0/network@0,1] maxvfs = 63 /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary [pci@500/pci@1/pci@0/pci@5/network@0] maxvfs = 7 /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary [pci@500/pci@1/pci@0/pci@5/network@0,1] maxvfs = 7 - Determine the device path of the boot disk that must be retained.
See Step 2 in How to Create a Root Domain by Assigning a PCIe Bus.
- Determine the physical device to which the block device is linked.
See Step 3 in How to Create a Root Domain by Assigning a PCIe Bus.
- Determine the network interface that is used by the system.
See Step 4 in How to Create a Root Domain by Assigning a PCIe Bus.
- Determine the physical device to which the network interface is linked.
The following command uses the igb0 network interface:
primary# ls -l /dev/igb0 lrwxrwxrwx 1 root root 46 Jul 30 17:29 /dev/igb0 -> ../devices/pci@500/pci@0/pci@8/network@0:igb0
In this example, the physical device for the network interface used by the primary domain is connected to the PCIe endpoint device (pci@500/pci@0/pci@8), which corresponds to the listing of MB/NET0 in Step 1. So, you do not want to remove this device from the primary domain. You can safely assign all other PCIe devices to other domains because they are not used by the primary domain.
If the network interface used by the primary domain is on a bus that you want to assign to another domain, the primary domain would need to be reconfigured to use a different network interface.
- Remove the PCIe endpoint devices that you might use in I/O domains.
In this example, you can remove the PCIE2, PCIE3, PCIE4, and PCIE5 endpoint devices because they are not being used by the primary domain.
- Remove the PCIe endpoint devices.
Caution - Do not remove the devices that are used or required by the primary domain. Do not remove a bus that has devices that are used by a domain, such as network ports or usbecm devices. If you mistakenly remove the wrong devices, use the ldm cancel-reconf primary command to cancel the delayed reconfiguration on the primary domain.
You can remove multiple devices at one time to avoid multiple reboots.
primary# ldm start-reconf primary primary# ldm set-io iov=on pci_1 All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. primary# ldm remove-io /SYS/MB/PCIE1 primary ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ primary# ldm remove-io /SYS/MB/PCIE3 primary ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ primary# ldm remove-io /SYS/MB/PCIE5 primary ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------
- Save the new configuration to the service processor (SP).
The following command saves the configuration in a file called dio:
primary# ldm add-config dio
- Reboot the system to reflect the removal of the PCIe endpoint devices.
primary# shutdown -i6 -g0 -y
- Remove the PCIe endpoint devices.
- Log in to the primary domain and verify that the PCIe endpoint devices are no longer assigned to the domain.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary pci_0 BUS pci_0 primary pci_1 BUS pci_1 primary IOV /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 OCC /SYS/MB/PCIE3 PCIE pci_1 OCC /SYS/MB/PCIE5 PCIE pci_1 OCC /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE9 PCIE pci_1 primary EMP /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary
Note - The ldm list-io -l output might show SUNW,assigned-device for the PCIe endpoint devices that were removed. Actual information is no longer available from the primary domain, but the domain to which the device is assigned has this information. - Assign a PCIe endpoint device to a domain.
- Add the PCIE3 device to the ldg1 domain.
primary# ldm add-io /SYS/MB/PCIE3 ldg1
- Bind and start the ldg1 domain.
primary# ldm bind ldg1 primary# ldm start ldg1 LDom ldg1 started
- Add the PCIE3 device to the ldg1 domain.
- Log in to the ldg1 domain and verify that the device is available for use.
Verify that the network device is available and then configure the network device for use in the domain.
primary# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net0 Ethernet unknown 0 unknown nxge0 net1 Ethernet unknown 0 unknown nxge1 net2 Ethernet unknown 0 unknown nxge2 net3 Ethernet unknown 0 unknown nxge3