How to Create a Root Domain by Assigning a PCIe Bus
This example procedure shows how to create a new root domain from an initial configuration where several buses are owned by the primary domain. By default the primary domain owns all buses present on the system. This example is for a SPARC T4-2 server. This procedure can also be used on other servers. The instructions for different servers might vary slightly from these, but you can obtain the basic principles from this example.
Ensure that you do not remove the PCIe buses that host the boot disk and primary network interface from the primary domain.
 | Caution - All internal disks on the supported servers might be connected to a single PCIe bus. If a domain is booted from an internal disk, do not remove that bus from the domain. Ensure that you do not remove a bus that has devices that are used by a domain, such as network ports or usbecm devices. If you remove the wrong bus, a domain might not be able to access the required devices and could become unusable. To remove a bus that has devices that are used by a domain, reconfigure that domain to use devices from other buses. For example, you might have to reconfigure the domain to use a different on-board network port or a PCIe card from a different PCIe slot. On certain SPARC servers, you can remove a PCIe bus that contains USB, graphics controllers, and other devices. However, you cannot add such a PCIe bus to any other domain. Such PCIe buses can be added only to the primary domain. |
In this example, the primary domain uses only a ZFS pool (rpool) and network interface (igb0). If the primary domain uses more devices, repeat Steps 2-4 for each device to ensure that none are located on the bus that will be removed.
You can add a bus to or remove a bus from a domain by using its device path (pci@nnn) or its pseudonym (pci_n). The ldm list-bindings primary or ldm list -l -o physio primary command shows the following:
pci@400 corresponds to pci_0
pci@500 corresponds to pci_1
pci@600 corresponds to pci_2
pci@700 corresponds to pci_3
- Verify that the primary domain owns more than one PCIe bus.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ /SYS/PM0/CMP0/PEX BUS pci_0 primary IOV /SYS/PM0/CMP1/PEX BUS pci_1 /SYS/PM0/CMP2/PEX BUS pci_2 primary /SYS/PM0/CMP3/PEX BUS pci_3 primary /SYS/MB/PCIE1 PCIE pci_0 primary EMP /SYS/MB/SASHBA0 PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary EMP /SYS/MB/PCIE6 PCIE pci_1 primary EMP /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE2 PCIE pci_2 primary EMP /SYS/MB/PCIE3 PCIE pci_2 primary EMP /SYS/MB/PCIE4 PCIE pci_2 primary EMP /SYS/MB/PCIE8 PCIE pci_3 primary EMP /SYS/MB/SASHBA1 PCIE pci_3 primary OCC /SYS/MB/NET2 PCIE pci_3 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_3 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_3 primary
- Determine the device path of the boot disk that must be retained.
- For UFS file systems, run the df / command to determine the device path of the boot disk.
primary# df / / (/dev/dsk/c0t5000CCA03C138904d0s0):22755742 blocks 2225374 files
- For ZFS file systems, first run the df / command to determine the pool name. Then, run the zpool status command to determine the device path of the boot disk.
primary# zpool status rpool pool: rpool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 c0t5000CCA03C138904d0s0 ONLINE 0 0 0
- For UFS file systems, run the df / command to determine the device path of the boot disk.
- Obtain information about the system's boot disk.
- For a disk that is managed with Solaris I/O multipathing, determine the PCIe bus to which the boot disk is connected by using the mpathadm command.
Starting with the SPARC T3 servers, the internal disks are managed by Solaris I/O multipathing.
- Find the initiator port to which the disk is connected.
primary# mpathadm show lu /dev/rdsk/c0t5000CCA03C138904d0s0 Logical Unit: /dev/rdsk/c0t5000CCA03C138904d0s2 mpath-support: libmpscsi_vhci.so Vendor: HITACHI Product: H106030SDSUN300G Revision: A2B0 Name Type: unknown type Name: 5000cca03c138904 Asymmetric: no Current Load Balance: round-robin Logical Unit Group ID: NA Auto Failback: on Auto Probing: NA Paths: Initiator Port Name: w50800200014100c8 Target Port Name: w5000cca03c138905 Override Path: NA Path State: OK Disabled: no Target Ports: Name: w5000cca03c138905 Relative ID: 0 - Determine the PCIe bus on which the initiator port is present.
primary# mpathadm show initiator-port w50800200014100c8 Initiator Port: w50800200014100c8 Transport Type: unknown OS Device File: /devices/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@1
- Find the initiator port to which the disk is connected.
- For a disk that is not managed with Solaris I/O multipathing, determine the physical device to which the block device is linked by using the ls -l command.
The following example uses block device c1t0d0s0:
primary# ls -l /dev/dsk/c0t1d0s0 lrwxrwxrwx 1 root root 49 Oct 1 10:39 /dev/dsk/c0t1d0s0 -> ../../devices/pci@400/pci@0/pci@1/scsi@0/sd@1,0:a
In this example, the physical device for the primary domain's boot disk is connected to the pci@400 bus.
- For a disk that is managed with Solaris I/O multipathing, determine the PCIe bus to which the boot disk is connected by using the mpathadm command.
- Determine the network interface that is used by the system.
Identify the primary network interface that is “plumbed” by using the ifconfig command. A plumbed interface has streams set up so that the IP protocol can use the device.
primary# ifconfig -a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 net0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 3 inet 10.129.241.135 netmask ffffff00 broadcast 10.129.241.255 ether 0:10:e0:e:f1:78 primary# dladm show-phys net0 LINK MEDIA STATE SPEED DUPLEX DEVICE net0 Ethernet up 1000 full igb0 - Determine the physical device to which the network interface is linked.
The following command uses the igb0 network interface:
primary# ls -l /dev/igb0 lrwxrwxrwx 1 root root 46 Oct 1 10:39 /dev/igb0 -> ../devices/pci@500/pci@0/pci@c/network@0:igb0
Perform the ls -l /dev/usbecm command, as well.
In this example, the physical device for the network interface used by the primary domain is under bus pci@500, which corresponds to the earlier listing of pci_1. So, the other two buses, pci_2 (pci@600) and pci_3 (pci@700), can safely be assigned to other domains because they are not used by the primary domain.
If the network interface used by the primary domain is on a bus that you want to assign to another domain, reconfigure the primary domain to use a different network interface.
- Remove a bus that does not contain the boot disk or the network interface from the primary domain.
In this example, the pci_2 bus is being removed from the primary domain.
- Dynamic method:
Ensure that the devices in the pci_2 bus are not in use by the primary domain OS. If they are, this command might fail to remove the bus. Use the static method to forcibly remove the pci_2 bus.
primary# ldm remove-io pci_2 primary
- Static method:
Before you remove the bus, you must initiate a delayed reconfiguration.
primary# ldm start-reconf primary primary# ldm remove-io pci_2 primary primary# shutdown -y -g0 -i6
The bus that the primary domain uses for the boot disk and the network device cannot be assigned to other domains. You can assign any of the other buses to another domain. In this example, the pci@600 is not used by the primary domain, so you can reassign it to another domain.
- Dynamic method:
- Add a bus to a domain.
In this example, you add the pci_2 bus to the ldg1 domain.
- Dynamic method:
primary# ldm add-io pci_2 ldg1
- Static method:
Before you add the bus, you must stop the domain.
primary# ldm stop-domain ldg1 primary# ldm add-io pci_2 ldg1 primary# ldm start-domain ldg1
- Dynamic method:
- Save this SP configuration to the service processor.
In this example, the SP configuration is io-domain.
primary# ldm add-spconfig io-domain
This SP configuration, io-domain, is also set as the next SP configuration to be used after the reboot.
- Confirm that the correct bus is still assigned to the primary domain and that the correct bus is assigned to domain ldg1.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ /SYS/PM0/CMP0/PEX BUS pci_0 primary IOV /SYS/PM0/CMP1/PEX BUS pci_1 primary /SYS/PM0/CMP2/PEX BUS pci_2 ldg1 /SYS/PM0/CMP3/PEX BUS pci_3 primary /SYS/MB/PCIE1 PCIE pci_0 primary EMP /SYS/MB/SASHBA0 PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary EMP /SYS/MB/PCIE6 PCIE pci_1 primary EMP /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE2 PCIE pci_2 ldg1 EMP /SYS/MB/PCIE3 PCIE pci_2 ldg1 EMP /SYS/MB/PCIE4 PCIE pci_2 ldg1 EMP /SYS/MB/PCIE8 PCIE pci_3 primary EMP /SYS/MB/SASHBA1 PCIE pci_3 primary OCC /SYS/MB/NET2 PCIE pci_3 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_3 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_3 primary
This output confirms that PCIe buses pci_0, pci_1, and pci_3 and their devices are assigned to the primary domain. It also confirms that PCIe bus pci_2 and its devices are assigned to the ldg1 domain.