How to Create an I/O Domain by Assigning a PCIe Endpoint Device

Plan all DIO deployments ahead of time to minimize downtime.

Caution:

The primary domain loses access to the on-board DVD device if you assign the /SYS/MB/SASHBA1 slot on a SPARC T4-1 system to a DIO domain. The SPARC T4-1 system includes two DIO slots for on-board storage, which are represented by the /SYS/MB/SASHBA0 and /SYS/MB/SASHBA1 paths. In addition to hosting multiheaded on-board disks, the /SYS/MB/SASHBA1 slot hosts the on-board DVD device. So, if you assign /SYS/MB/SASHBA1 to a DIO domain, the primary domain loses access to the on-board DVD device. The SPARC T4-2 system has a single SASHBA slot that hosts all on-board disks as well as the on-board DVD device. So, if you assign SASHBA to a DIO domain, the on-board disks and the on-board DVD device are loaned to the DIO domain and unavailable to the primary domain.

For an example of adding a PCIe endpoint device to create an I/O domain, see Planning PCIe Endpoint Device Configuration.

Note:

For Oracle Solaris 11 releases prior to Oracle Solaris 11.4, use the DefaultFixed NCP to configure datalinks and network interfaces on Oracle Solaris 11 systems. The Oracle Solaris 11 OS includes the following NCPs:

DefaultFixed – Enables you to use the dladm or ipadm command to manage networking
Automatic – Enables you to use the netcfg or netadm command to manage networking

Ensure that the DefaultFixed NCP is enabled by using the netadm list command. See Chapter 7, Using Datalink and Interface Configuration Commands on Profiles in Oracle Solaris Administration: Network Interfaces and Network Virtualization.

Identify and archive the devices that are currently installed on the system.

The output of the ldm list-io -l command shows how the I/O devices are currently configured. You can obtain more detailed information by using the prtdiag -v command.

Note:

After the devices are assigned to I/O domains, the identity of the devices can be determined only in the I/O domains.

primary# ldm list-io -l
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/MB/CMP0/PEX                          BUS    pci_0    primary
[pci@400]
/SYS/MB/CMP0/NIU_CORE                     NIU    niu_0    primary
[niu@480]

/SYS/MB/CMP1/PEX                          BUS    pci_1    primary
[pci@500]
/SYS/MB/CMP1/NIU_CORE                     NIU    niu_1    primary
[niu@580] 

/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
[pci@400/pci@2/pci@0/pci@8]
    SUNW,emlxs@0/fp/disk
    SUNW,emlxs@0/fp/tape
    SUNW,emlxs@0/fp@0,0
    SUNW,emlxs@0,1/fp/disk
    SUNW,emlxs@0,1/fp/tape
    SUNW,emlxs@0,1/fp@0,0
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
[pci@400/pci@2/pci@0/pci@4]
    pci/scsi/disk
    pci/scsi/tape
    pci/scsi/disk
    pci/scsi/tape
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
[pci@400/pci@2/pci@0/pci@0]
    ethernet@0
    ethernet@0,1
    SUNW,qlc@0,2/fp/disk
    SUNW,qlc@0,2/fp@0,0
    SUNW,qlc@0,3/fp/disk
    SUNW,qlc@0,3/fp@0,0
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
[pci@400/pci@1/pci@0/pci@8]
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
[pci@400/pci@1/pci@0/pci@c]
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
[pci@400/pci@2/pci@0/pci@e]
    scsi@0/iport@1
    scsi@0/iport@2
    scsi@0/iport@4
    scsi@0/iport@8
    scsi@0/iport@80/cdrom@p7,0
    scsi@0/iport@v0
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
[pci@400/pci@1/pci@0/pci@4]
    network@0
    network@0,1
/SYS/MB/PCIE1                             PCIE   pci_1    primary  OCC      
[pci@500/pci@2/pci@0/pci@a]
    SUNW,qlc@0/fp/disk
    SUNW,qlc@0/fp@0,0
    SUNW,qlc@0,1/fp/disk
    SUNW,qlc@0,1/fp@0,0
/SYS/MB/PCIE3                             PCIE   pci_1    primary  OCC      
[pci@500/pci@2/pci@0/pci@6]
    network@0
    network@0,1
    network@0,2
    network@0,3
/SYS/MB/PCIE5                             PCIE   pci_1    primary  OCC      
[pci@500/pci@2/pci@0/pci@0]
    network@0
    network@0,1
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP      
[pci@500/pci@1/pci@0/pci@6]
/SYS/MB/PCIE9                             PCIE   pci_1    primary  EMP      
[pci@500/pci@1/pci@0/pci@0]
/SYS/MB/NET2                              PCIE   pci_1    primary  OCC      
[pci@500/pci@1/pci@0/pci@5]
    network@0
    network@0,1
    ethernet@0,80
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
[pci@400/pci@1/pci@0/pci@4/network@0]
    maxvfs = 7
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
[pci@400/pci@1/pci@0/pci@4/network@0,1]
    maxvfs = 7
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    primary           
[pci@500/pci@2/pci@0/pci@0/network@0]
    maxvfs = 63
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    primary           
[pci@500/pci@2/pci@0/pci@0/network@0,1]
    maxvfs = 63
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    primary           
[pci@500/pci@1/pci@0/pci@5/network@0]
    maxvfs = 7
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    primary           
[pci@500/pci@1/pci@0/pci@5/network@0,1]
    maxvfs = 7

Determine the device path of the boot disk that must be retained.

See Step 2 in How to Create a Root Domain by Assigning a PCIe Bus.
Determine the physical device to which the block device is linked.

See Step 3 in How to Create a Root Domain by Assigning a PCIe Bus.
Determine the network interface that is used by the system.

See Step 4 in How to Create a Root Domain by Assigning a PCIe Bus.
Determine the physical device to which the network interface is linked.
The following command uses the igb0 network interface:
```
primary# ls -l /dev/igb0
lrwxrwxrwx   1 root     root          46 Jul 30 17:29 /dev/igb0 ->
../devices/pci@500/pci@0/pci@8/network@0:igb0
```
In this example, the physical device for the network interface used by the primary domain is connected to the PCIe endpoint device (pci@500/pci@0/pci@8), which corresponds to the listing of MB/NET0 in Step 1. So, you do not want to remove this device from the primary domain. You can safely assign all other PCIe devices to other domains because they are not used by the primary domain.

If the network interface used by the primary domain is on a bus that you want to assign to another domain, the primary domain would need to be reconfigured to use a different network interface.

Remove the PCIe endpoint devices that you might use in I/O domains.

In this example, you can remove the PCIE2, PCIE3, PCIE4, and PCIE5 endpoint devices because they are not being used by the primary domain.

Remove the PCIe endpoint devices.

Caution:

Do not remove the devices that are used or required by the primary domain. Do not remove a bus that has devices that are used by a domain, such as network ports or usbecm devices. If you mistakenly remove the wrong devices, use the ldm cancel-reconf primary command to cancel the delayed reconfiguration on the primary domain.

You can remove multiple devices at one time to avoid multiple reboots.

primary# ldm start-reconf primary
primary# ldm set-io iov=on pci_1
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.
primary# ldm remove-io /SYS/MB/PCIE1 primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
primary# ldm remove-io /SYS/MB/PCIE3 primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
primary# ldm remove-io /SYS/MB/PCIE5 primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

Save the new SP configuration to the service processor (SP).
The following command saves the SP configuration in a file called dio:
```
primary# ldm add-spconfig dio
```
Reboot the system to reflect the removal of the PCIe endpoint devices.
```
primary# shutdown -i6 -g0 -y
```

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/PM0/CMP0/NIU_CORE                    NIU    niu_0    primary
/SYS/PM0/CMP1/NIU_CORE                    NIU    niu_1    primary
/SYS/PM0/CMP0/PEX                         BUS    pci_0    primary  IOV
/SYS/PM0/CMP1/PEX                         BUS    pci_1    rootdom1 IOV
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1             OCC      
/SYS/MB/PCIE3                             PCIE   pci_1             OCC      
/SYS/MB/PCIE5                             PCIE   pci_1             OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP      
/SYS/MB/PCIE9                             PCIE   pci_1    primary  EMP      
/SYS/MB/NET2                              PCIE   pci_1    primary  OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    primary

Note:

The ldm list-io -l output might show SUNW,assigned-device for the PCIe endpoint devices that were removed. Actual information is no longer available from the primary domain, but the domain to which the device is assigned has this information.

Assign a PCIe endpoint device to a domain.

Add the PCIE3 device to the ldg1 domain.
```
primary# ldm add-io /SYS/MB/PCIE3 ldg1
```

Bind and start the ldg1 domain.

primary# ldm bind ldg1
primary# ldm start-domain ldg1
LDom ldg1 started

Verify that the network device is available and then configure the network device for use in the domain.

primary# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             unknown    0      unknown   nxge0
net1              Ethernet             unknown    0      unknown   nxge1
net2              Ethernet             unknown    0      unknown   nxge2
net3              Ethernet             unknown    0      unknown   nxge3