How to Create an I/O Domain by Assigning a PCIe Bus

This example procedure shows how to create a new I/O domain from an initial configuration where several buses are owned by the primary domain. By default the primary domain owns all buses present on the system. This example is for a SPARC T4-2 server. This procedure can also be used on other servers. The instructions for different servers might vary slightly from these, but you can obtain the basic principles from this example.

First, you must retain the bus that has the primary domain's boot disk. Then, you remove another bus from the primary domain and assign it to another domain.

---

Caution  - All internal disks on the supported servers might be connected to a single PCIe bus. If a domain is booted from an internal disk, do not remove that bus from the domain. Also, ensure that you are not removing a bus with devices (such as network ports) that are used by a domain. If you remove the wrong bus, a domain might not be able to access the required devices and could become unusable. To remove a bus that has devices that are used by a domain, reconfigure that domain to use devices from other buses. For example, you might have to reconfigure the domain to use a different on-board network port or a PCIe card from a different PCIe slot.

---

In this example, the primary domain uses only a ZFS pool (rpool) and network interface (igb0). If the primary domain uses more devices, repeat Steps 2-4 for each device to ensure that none are located on the bus that will be removed.

You can add a bus to or remove a bus from a domain by using its device path (pci@nnn) or its pseudonym (pci_n). The ldm list-bindings primary or ldm list -l -o physio primary command shows the following:

• pci@400 corresponds to pci_0

• pci@500 corresponds to pci_1

• pci@600 corresponds to pci_2

• pci@700 corresponds to pci_3

1. Verify that the primary domain owns more than one PCIe bus.

   primary# ldm list-io
   NAME                                      TYPE   BUS      DOMAIN   STATUS
   ----                                      ----   ---      ------   ------
   pci_0                                     BUS    pci_0    primary
   pci_1                                     BUS    pci_1    primary
   pci_2                                     BUS    pci_2    primary
   pci_3                                     BUS    pci_3    primary
   /SYS/MB/PCIE1                             PCIE   pci_0    primary  EMP
   /SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
   /SYS/MB/NET0                              PCIE   pci_0    primary  OCC
   /SYS/MB/PCIE5                             PCIE   pci_1    primary  EMP
   /SYS/MB/PCIE6                             PCIE   pci_1    primary  EMP
   /SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP
   /SYS/MB/PCIE2                             PCIE   pci_2    primary  EMP
   /SYS/MB/PCIE3                             PCIE   pci_2    primary  EMP
   /SYS/MB/PCIE4                             PCIE   pci_2    primary  EMP
   /SYS/MB/PCIE8                             PCIE   pci_3    primary  EMP
   /SYS/MB/SASHBA1                           PCIE   pci_3    primary  OCC
   /SYS/MB/NET2                              PCIE   pci_3    primary  OCC
   /SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
   /SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
   /SYS/MB/NET2/IOVNET.PF0                   PF     pci_3    primary
   /SYS/MB/NET2/IOVNET.PF1                   PF     pci_3    primary

2. Determine the device path of the boot disk that must be retained.
   • For UFS file systems, run the df / command to determine the device path of the boot disk.

     primary# df /
     /                  (/dev/dsk/c0t5000CCA03C138904d0s0):22755742 blocks  2225374 files

   • For ZFS file systems, first run the df / command to determine the pool name. Then, run the zpool status command to determine the device path of the boot disk.

     primary# zpool status rpool
       pool: rpool
      state: ONLINE
       scan: none requested
     config:
     
             NAME                       STATE     READ WRITE CKSUM
             rpool                      ONLINE       0     0     0
               c0t5000CCA03C138904d0s0  ONLINE       0     0     0

3. Obtain information about the system's boot disk.
   • For a disk that is managed with Solaris I/O multipathing, determine the PCIe bus under which the boot disk is connected by using the mpathadm command.

     Starting with the SPARC T3 servers, the internal disks are managed by Solaris I/O multipathing.

     1. Find the initiator port to which the disk is connected.

        primary# mpathadm show lu /dev/rdsk/c0t5000CCA03C138904d0s0
        Logical Unit:  /dev/rdsk/c0t5000CCA03C138904d0s2
                mpath-support:  libmpscsi_vhci.so
                Vendor:  HITACHI 
                Product:  H106030SDSUN300G
                Revision:  A2B0
                Name Type:  unknown type
                Name:  5000cca03c138904
                Asymmetric:  no
                Current Load Balance:  round-robin
                Logical Unit Group ID:  NA
                Auto Failback:  on
                Auto Probing:  NA
        
                Paths:  
                        Initiator Port Name:  w50800200014100c8
                        Target Port Name:  w5000cca03c138905
                        Override Path:  NA
                        Path State:  OK
                        Disabled:  no
        
                Target Ports:
                        Name:  w5000cca03c138905
                        Relative ID:  0

     2. Determine the PCIe bus on which the initiator port is present.

        primary# mpathadm show initiator-port w50800200014100c8
        Initiator Port:  w50800200014100c8
                Transport Type:  unknown
                OS Device File:  /devices/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@1

   • For a disk that is not managed with Solaris I/O multipathing, determine the physical device to which the block device is linked by using the ls -l command.

     Use this command for a disk on an UltraSPARC T2 or UltraSPARC T2 Plus system that is not managed with Solaris I/O multipathing.

     The following example uses block device c1t0d0s0:

     primary# ls -l /dev/dsk/c0t1d0s0
     lrwxrwxrwx   1 root     root          49 Oct  1 10:39 /dev/dsk/c0t1d0s0 ->
     ../../devices/pci@400/pci@0/pci@1/scsi@0/sd@1,0:a

     In this example, the physical device for the primary domain's boot disk is connected to the pci@400 bus.

4. Determine the network interface that is used by the system.

   Identify the primary network interface that is “plumbed” by using the ifconfig command. A plumbed interface has streams set up so that the IP protocol can use the device.

   • Oracle Solaris 10:

     primary# ifconfig -a
     lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
             inet 127.0.0.1 netmask ff000000 
     igb0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 3
             inet 10.129.241.135 netmask ffffff00 broadcast 10.129.241.255
             ether 0:10:e0:e:f1:78

   • Oracle Solaris 11:

     primary# ifconfig -a
     lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
             inet 127.0.0.1 netmask ff000000 
     net0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 3
             inet 10.129.241.135 netmask ffffff00 broadcast 10.129.241.255
             ether 0:10:e0:e:f1:78 
     
     primary# dladm show-phys net0
     LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
     net0              Ethernet             up         1000   full      igb0

5. Determine the physical device to which the network interface is linked.

   The following command uses the igb0 network interface:

   primary# ls -l /dev/igb0
   lrwxrwxrwx   1 root     root          46 Oct  1 10:39 /dev/igb0  ->
   ../devices/pci@500/pci@0/pci@c/network@0:igb0 

   In this example, the physical device for the network interface used by the primary domain is under bus pci@500, which corresponds to the earlier listing of pci_1. So, the other two buses, pci_2 (pci@600) and pci_3 (pci@700), can safely be assigned to other domains because they are not used by the primary domain.

   If the network interface used by the primary domain is on a bus that you want to assign to another domain, reconfigure the primary domain to use a different network interface.

6. Remove a bus that does not contain the boot disk or the network interface from the primary domain.

   In this example, the pci_2 bus is being removed from the primary domain. You must also initiate a delayed reconfiguration.

   primary# ldm start-reconf primary
   primary# ldm remove-io pci_2 primary
   

   The bus that the primary domain uses for the boot disk and the network device cannot be assigned to other domains. You can assign any of the other buses to another domain. In this example, the pci@600 is not used by the primary domain, so you can reassign it to another domain.

7. Save this configuration to the service processor.

   In this example, the configuration is io-domain.

   primary# ldm add-config io-domain

   This configuration, io-domain, is also set as the next configuration to be used after the reboot.

8. Reboot the root domain so that the change takes effect.

   primary# shutdown -i6 -g0 -y

9. Stop the domain to which you want to add the PCIe bus.

   The following example stops the ldg1 domain:

   primary# ldm stop ldg1

10. Add the available bus to the domain that needs direct access.

    The available bus is pci_2 and the domain is ldg1.

    primary# ldm add-io pci_2 ldg1

11. Restart the domain so that the change takes affect.

    The following commands restart the ldg1 domain:

    primary# ldm start ldg1

12. Confirm that the correct bus is still assigned to the primary domain and that the correct bus is assigned to domain ldg1.

    primary# ldm list-io
    NAME                                      TYPE   BUS      DOMAIN   STATUS
    ----                                      ----   ---      ------   ------
    pci_0                                     BUS    pci_0    primary
    pci_1                                     BUS    pci_1    primary
    pci_2                                     BUS    pci_2    ldg1
    pci_3                                     BUS    pci_3    primary
    /SYS/MB/PCIE1                             PCIE   pci_0    primary  EMP
    /SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
    /SYS/MB/NET0                              PCIE   pci_0    primary  OCC
    /SYS/MB/PCIE5                             PCIE   pci_1    primary  EMP
    /SYS/MB/PCIE6                             PCIE   pci_1    primary  EMP
    /SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP
    /SYS/MB/PCIE2                             PCIE   pci_2    ldg1     EMP
    /SYS/MB/PCIE3                             PCIE   pci_2    ldg1     EMP
    /SYS/MB/PCIE4                             PCIE   pci_2    ldg1     EMP
    /SYS/MB/PCIE8                             PCIE   pci_3    primary  EMP
    /SYS/MB/SASHBA1                           PCIE   pci_3    primary  OCC
    /SYS/MB/NET2                              PCIE   pci_3    primary  OCC
    /SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
    /SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
    /SYS/MB/NET2/IOVNET.PF0                   PF     pci_3    primary
    /SYS/MB/NET2/IOVNET.PF1                   PF     pci_3    primary

    This output confirms that PCIe buses pci_0, pci_1, and pci_3 and their devices are assigned to the primary domain. It also confirms that PCIe bus pci_2 and its devices are assigned to the ldg1 domain.