This example procedure shows how to create a new root domain from an initial configuration where several buses are owned by the primary domain. By default the primary domain owns all buses present on the system. This example is for a SPARC T4-2 server. This procedure can also be used on other servers. The instructions for different servers might vary slightly from these, but you can obtain the basic principles from this example.
Single-bus systems. Such systems, like a SPARC T4-1 server, can use only the primary domain as a root domain.
Some smaller multi-bus systems. Systems such as SPARC S7-2 and SPARC S7-2L servers have built-in cards that communicate only with a single bus and cannot be split across multiple buses. To create a second root domain, you must install additional cards to ensure that each root domain has a network interface and a boot disk.
Ensure that you do not remove the PCIe buses that host the boot disk and primary network interface from the primary domain.
Caution - All internal disks on the supported servers might be connected to a single PCIe bus. If a domain is booted from an internal disk, do not remove that bus from the domain. Ensure that you do not remove a bus that has devices that are used by a domain, such as network ports or usbecm devices. If you remove the wrong bus, a domain might not be able to access the required devices and could become unusable. To remove a bus that has devices that are used by a domain, reconfigure that domain to use devices from other buses. For example, you might have to reconfigure the domain to use a different on-board network port or a PCIe card from a different PCIe slot. On certain SPARC servers, you can remove a PCIe bus that contains USB, graphics controllers, and other devices. However, you cannot add such a PCIe bus to any other domain. Such PCIe buses can be added only to the primary domain.
In this example, the primary domain uses only a ZFS pool (rpool) and network interface (igb0). If the primary domain uses more devices, repeat Steps 2-4 for each device to ensure that none are located on the bus that will be removed.
You can add a bus to or remove a bus from a domain by using its device path (pci@nnn) or its pseudonym (pci_n). The ldm list-bindings primary or ldm list -l -o physio primary command shows the following:
pci@400 corresponds to pci_0
pci@500 corresponds to pci_1
pci@600 corresponds to pci_2
pci@700 corresponds to pci_3
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ /SYS/PM0/CMP0/PEX BUS pci_0 primary IOV /SYS/PM0/CMP1/PEX BUS pci_1 /SYS/PM0/CMP2/PEX BUS pci_2 primary /SYS/PM0/CMP3/PEX BUS pci_3 primary /SYS/MB/PCIE1 PCIE pci_0 primary EMP /SYS/MB/SASHBA0 PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary EMP /SYS/MB/PCIE6 PCIE pci_1 primary EMP /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE2 PCIE pci_2 primary EMP /SYS/MB/PCIE3 PCIE pci_2 primary EMP /SYS/MB/PCIE4 PCIE pci_2 primary EMP /SYS/MB/PCIE8 PCIE pci_3 primary EMP /SYS/MB/SASHBA1 PCIE pci_3 primary OCC /SYS/MB/NET2 PCIE pci_3 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_3 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_3 primary
primary# df / / (/dev/dsk/c0t5000CCA03C138904d0s0):22755742 blocks 2225374 files
primary# zpool status rpool pool: rpool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 c0t5000CCA03C138904d0s0 ONLINE 0 0 0
Starting with the SPARC T4 servers, the internal disks are managed by Solaris I/O multipathing.
primary# mpathadm show lu /dev/rdsk/c0t5000CCA03C138904d0s0 Logical Unit: /dev/rdsk/c0t5000CCA03C138904d0s2 mpath-support: libmpscsi_vhci.so Vendor: HITACHI Product: H106030SDSUN300G Revision: A2B0 Name Type: unknown type Name: 5000cca03c138904 Asymmetric: no Current Load Balance: round-robin Logical Unit Group ID: NA Auto Failback: on Auto Probing: NA Paths: Initiator Port Name: w50800200014100c8 Target Port Name: w5000cca03c138905 Override Path: NA Path State: OK Disabled: no Target Ports: Name: w5000cca03c138905 Relative ID: 0
primary# mpathadm show initiator-port w50800200014100c8 Initiator Port: w50800200014100c8 Transport Type: unknown OS Device File: /devices/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@1
The following example uses block device c1t0d0s0:
primary# ls -l /dev/dsk/c0t1d0s0 lrwxrwxrwx 1 root root 49 Oct 1 10:39 /dev/dsk/c0t1d0s0 -> ../../devices/pci@400/pci@0/pci@1/scsi@0/sd@1,0:a
In this example, the physical device for the primary domain's boot disk is connected to the pci@400 bus.
Identify the primary network interface that is “plumbed” by using the ifconfig command. A plumbed interface has streams set up so that the IP protocol can use the device.
primary# ifconfig -a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 net0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 3 inet 10.129.241.135 netmask ffffff00 broadcast 10.129.241.255 ether 0:10:e0:e:f1:78 primary# dladm show-phys net0 LINK MEDIA STATE SPEED DUPLEX DEVICE net0 Ethernet up 1000 full igb0
The following command uses the igb0 network interface:
primary# ls -l /dev/igb0 lrwxrwxrwx 1 root root 46 Oct 1 10:39 /dev/igb0 -> ../devices/pci@500/pci@0/pci@c/network@0:igb0
Perform the ls -l /dev/usbecm command, as well.
In this example, the physical device for the network interface used by the primary domain is under bus pci@500, which corresponds to the earlier listing of pci_1. So, the other two buses, pci_2 (pci@600) and pci_3 (pci@700), can safely be assigned to other domains because they are not used by the primary domain.
If the network interface used by the primary domain is on a bus that you want to assign to another domain, reconfigure the primary domain to use a different network interface.
In this example, the pci_2 bus is being removed from the primary domain.
Ensure that the devices in the pci_2 bus are not in use by the primary domain OS. If they are, this command might fail to remove the bus. Use the static method to forcibly remove the pci_2 bus.
primary# ldm remove-io pci_2 primary
Before you remove the bus, you must initiate a delayed reconfiguration.
primary# ldm start-reconf primary primary# ldm remove-io pci_2 primary primary# shutdown -y -g0 -i6
The bus that the primary domain uses for the boot disk and the network device cannot be assigned to other domains. You can assign any of the other buses to another domain. In this example, the pci@600 is not used by the primary domain, so you can reassign it to another domain.
In this example, you add the pci_2 bus to the ldg1 domain.
primary# ldm add-io pci_2 ldg1
Before you add the bus, you must stop the domain.
primary# ldm stop-domain ldg1 primary# ldm add-io pci_2 ldg1 primary# ldm start-domain ldg1
In this example, the SP configuration is io-domain.
primary# ldm add-spconfig io-domain
This SP configuration, io-domain, is also set as the next SP configuration to be used after the reboot.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ /SYS/PM0/CMP0/PEX BUS pci_0 primary IOV /SYS/PM0/CMP1/PEX BUS pci_1 primary /SYS/PM0/CMP2/PEX BUS pci_2 ldg1 /SYS/PM0/CMP3/PEX BUS pci_3 primary /SYS/MB/PCIE1 PCIE pci_0 primary EMP /SYS/MB/SASHBA0 PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary EMP /SYS/MB/PCIE6 PCIE pci_1 primary EMP /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE2 PCIE pci_2 ldg1 EMP /SYS/MB/PCIE3 PCIE pci_2 ldg1 EMP /SYS/MB/PCIE4 PCIE pci_2 ldg1 EMP /SYS/MB/PCIE8 PCIE pci_3 primary EMP /SYS/MB/SASHBA1 PCIE pci_3 primary OCC /SYS/MB/NET2 PCIE pci_3 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_3 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_3 primary
This output confirms that PCIe buses pci_0, pci_1, and pci_3 and their devices are assigned to the primary domain. It also confirms that PCIe bus pci_2 and its devices are assigned to the ldg1 domain.