Plan all DIO deployments ahead of time to minimize downtime.
Caution - The primary domain loses access to the on-board DVD device if you assign the /SYS/MB/SASHBA1 slot on a SPARC T3-1 or a SPARC T4-1 system to a DIO domain. The SPARC T3-1 and SPARC T4-1 systems include two DIO slots for on-board storage, which are represented by the /SYS/MB/SASHBA0 and /SYS/MB/SASHBA1 paths. In addition to hosting multiheaded on-board disks, the /SYS/MB/SASHBA1 slot hosts the on-board DVD device. So, if you assign /SYS/MB/SASHBA1 to a DIO domain, the primary domain loses access to the on-board DVD device. The SPARC T3-2 and SPARC T4-2 systems have a single SASHBA slot that hosts all on-board disks as well as the on-board DVD device. So, if you assign SASHBA to a DIO domain, the on-board disks and the on-board DVD device are loaned to the DIO domain and unavailable to the primary domain. |
For an example of adding a PCIe endpoint device to create an I/O domain, see Planning PCIe Endpoint Device Configuration.
The Oracle Solaris 11 OS includes the following NCPs:
DefaultFixed – Enables you to use the dladm or ipadm command to manage networking
Automatic – Enables you to use the netcfg or netadm command to manage networking
Ensure that the DefaultFixed NCP is enabled by using the netadm list command. See Chapter 7, Using Datalink and Interface Configuration Commands on Profiles in Oracle Solaris Administration: Network Interfaces and Network Virtualization.
The output of the ldm list-io -l command shows how the I/O devices are currently configured. You can obtain more detailed information by using the prtdiag -v command.
primary# ldm list-io -l NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary [niu@480] niu_1 NIU niu_1 primary [niu@580] pci_0 BUS pci_0 primary [pci@400] pci_1 BUS pci_1 primary [pci@500] /SYS/MB/PCIE0 PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@8] SUNW,emlxs@0/fp/disk SUNW,emlxs@0/fp/tape SUNW,emlxs@0/fp@0,0 SUNW,emlxs@0,1/fp/disk SUNW,emlxs@0,1/fp/tape SUNW,emlxs@0,1/fp@0,0 /SYS/MB/PCIE2 PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@4] pci/scsi/disk pci/scsi/tape pci/scsi/disk pci/scsi/tape /SYS/MB/PCIE4 PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@0] ethernet@0 ethernet@0,1 SUNW,qlc@0,2/fp/disk SUNW,qlc@0,2/fp@0,0 SUNW,qlc@0,3/fp/disk SUNW,qlc@0,3/fp@0,0 /SYS/MB/PCIE6 PCIE pci_0 primary EMP [pci@400/pci@1/pci@0/pci@8] /SYS/MB/PCIE8 PCIE pci_0 primary EMP [pci@400/pci@1/pci@0/pci@c] /SYS/MB/SASHBA PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@e] scsi@0/iport@1 scsi@0/iport@2 scsi@0/iport@4 scsi@0/iport@8 scsi@0/iport@80/cdrom@p7,0 scsi@0/iport@v0 /SYS/MB/NET0 PCIE pci_0 primary OCC [pci@400/pci@1/pci@0/pci@4] network@0 network@0,1 /SYS/MB/PCIE1 PCIE pci_1 primary OCC [pci@500/pci@2/pci@0/pci@a] SUNW,qlc@0/fp/disk SUNW,qlc@0/fp@0,0 SUNW,qlc@0,1/fp/disk SUNW,qlc@0,1/fp@0,0 /SYS/MB/PCIE3 PCIE pci_1 primary OCC [pci@500/pci@2/pci@0/pci@6] network@0 network@0,1 network@0,2 network@0,3 /SYS/MB/PCIE5 PCIE pci_1 primary OCC [pci@500/pci@2/pci@0/pci@0] network@0 network@0,1 /SYS/MB/PCIE7 PCIE pci_1 primary EMP [pci@500/pci@1/pci@0/pci@6] /SYS/MB/PCIE9 PCIE pci_1 primary EMP [pci@500/pci@1/pci@0/pci@0] /SYS/MB/NET2 PCIE pci_1 primary OCC [pci@500/pci@1/pci@0/pci@5] network@0 network@0,1 ethernet@0,80 /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary [pci@400/pci@1/pci@0/pci@4/network@0] maxvfs = 7 /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary [pci@400/pci@1/pci@0/pci@4/network@0,1] maxvfs = 7 /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary [pci@500/pci@2/pci@0/pci@0/network@0] maxvfs = 63 /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary [pci@500/pci@2/pci@0/pci@0/network@0,1] maxvfs = 63 /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary [pci@500/pci@1/pci@0/pci@5/network@0] maxvfs = 7 /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary [pci@500/pci@1/pci@0/pci@5/network@0,1] maxvfs = 7
See Step 2 in How to Create a Root Domain by Assigning a PCIe Bus.
See Step 3 in How to Create a Root Domain by Assigning a PCIe Bus.
See Step 4 in How to Create a Root Domain by Assigning a PCIe Bus.
The following command uses the igb0 network interface:
primary# ls -l /dev/igb0 lrwxrwxrwx 1 root root 46 Jul 30 17:29 /dev/igb0 -> ../devices/pci@500/pci@0/pci@8/network@0:igb0
In this example, the physical device for the network interface used by the primary domain is connected to the PCIe endpoint device (pci@500/pci@0/pci@8), which corresponds to the listing of MB/NET0 in Step 1. So, you do not want to remove this device from the primary domain. You can safely assign all other PCIe devices to other domains because they are not used by the primary domain.
If the network interface used by the primary domain is on a bus that you want to assign to another domain, the primary domain would need to be reconfigured to use a different network interface.
In this example, you can remove the PCIE2, PCIE3, PCIE4, and PCIE5 endpoint devices because they are not being used by the primary domain.
Caution - Do not remove the devices that are used or required by the primary domain. Do not remove a bus that has devices that are used by a domain, such as network ports or usbecm devices. If you mistakenly remove the wrong devices, use the ldm cancel-reconf primary command to cancel the delayed reconfiguration on the primary domain. |
You can remove multiple devices at one time to avoid multiple reboots.
primary# ldm start-reconf primary primary# ldm set-io iov=on pci_1 All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. primary# ldm remove-io /SYS/MB/PCIE1 primary ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ primary# ldm remove-io /SYS/MB/PCIE3 primary ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ primary# ldm remove-io /SYS/MB/PCIE5 primary ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------
The following command saves the configuration in a file called dio:
primary# ldm add-config dio
primary# shutdown -i6 -g0 -y
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary pci_0 BUS pci_0 primary pci_1 BUS pci_1 primary IOV /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 OCC /SYS/MB/PCIE3 PCIE pci_1 OCC /SYS/MB/PCIE5 PCIE pci_1 OCC /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE9 PCIE pci_1 primary EMP /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary
primary# ldm add-io /SYS/MB/PCIE3 ldg1
primary# ldm bind ldg1 primary# ldm start ldg1 LDom ldg1 started
Verify that the network device is available and then configure the network device for use in the domain.
primary# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net0 Ethernet unknown 0 unknown nxge0 net1 Ethernet unknown 0 unknown nxge1 net2 Ethernet unknown 0 unknown nxge2 net3 Ethernet unknown 0 unknown nxge3