Go to main content

Oracle® VM Server for SPARC 3.6 Administration Guide

Exit Print View

Updated: September 2019
 
 

How to Create a Root Domain by Assigning a PCIe Bus

This example procedure shows how to create a new root domain from an initial configuration where several buses are owned by the primary domain. By default the primary domain owns all buses present on the system. This example is for a SPARC T4-2 server. This procedure can also be used on other servers. The instructions for different servers might vary slightly from these, but you can obtain the basic principles from this example.


Note - Do not add independent root domains to the following system types:
  • Single-bus systems. Such systems, like a SPARC T4-1 server, can use only the primary domain as a root domain.

  • Some smaller multi-bus systems. Systems such as SPARC S7-2 and SPARC S7-2L servers have built-in cards that communicate only with a single bus and cannot be split across multiple buses. To create a second root domain, you must install additional cards to ensure that each root domain has a network interface and a boot disk.


Ensure that you do not remove the PCIe buses that host the boot disk and primary network interface from the primary domain.


Caution

Caution  - All internal disks on the supported servers might be connected to a single PCIe bus. If a domain is booted from an internal disk, do not remove that bus from the domain. Ensure that you do not remove a bus that has devices that are used by a domain, such as network ports or usbecm devices. If you remove the wrong bus, a domain might not be able to access the required devices and could become unusable. To remove a bus that has devices that are used by a domain, reconfigure that domain to use devices from other buses. For example, you might have to reconfigure the domain to use a different on-board network port or a PCIe card from a different PCIe slot. On certain SPARC servers, you can remove a PCIe bus that contains USB, graphics controllers, and other devices. However, you cannot add such a PCIe bus to any other domain. Such PCIe buses can be added only to the primary domain.


In this example, the primary domain uses only a ZFS pool (rpool) and network interface (igb0). If the primary domain uses more devices, repeat Steps 2-4 for each device to ensure that none are located on the bus that will be removed.

You can add a bus to or remove a bus from a domain by using its device path (pci@nnn) or its pseudonym (pci_n). The ldm list-bindings primary or ldm list -l -o physio primary command shows the following:

  • pci@400 corresponds to pci_0

  • pci@500 corresponds to pci_1

  • pci@600 corresponds to pci_2

  • pci@700 corresponds to pci_3

  1. Verify that the primary domain owns more than one PCIe bus.
    primary# ldm list-io
    NAME                                      TYPE   BUS      DOMAIN   STATUS
    ----                                      ----   ---      ------   ------
    /SYS/PM0/CMP0/PEX                         BUS    pci_0    primary  IOV
    /SYS/PM0/CMP1/PEX                         BUS    pci_1
    /SYS/PM0/CMP2/PEX                         BUS    pci_2    primary
    /SYS/PM0/CMP3/PEX                         BUS    pci_3    primary
    /SYS/MB/PCIE1                             PCIE   pci_0    primary  EMP
    /SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
    /SYS/MB/NET0                              PCIE   pci_0    primary  OCC
    /SYS/MB/PCIE5                             PCIE   pci_1    primary  EMP
    /SYS/MB/PCIE6                             PCIE   pci_1    primary  EMP
    /SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP
    /SYS/MB/PCIE2                             PCIE   pci_2    primary  EMP
    /SYS/MB/PCIE3                             PCIE   pci_2    primary  EMP
    /SYS/MB/PCIE4                             PCIE   pci_2    primary  EMP
    /SYS/MB/PCIE8                             PCIE   pci_3    primary  EMP
    /SYS/MB/SASHBA1                           PCIE   pci_3    primary  OCC
    /SYS/MB/NET2                              PCIE   pci_3    primary  OCC
    /SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
    /SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
    /SYS/MB/NET2/IOVNET.PF0                   PF     pci_3    primary
    /SYS/MB/NET2/IOVNET.PF1                   PF     pci_3    primary
  2. Determine the device path of the boot disk that must be retained.
    • For UFS file systems, run the df / command to determine the device path of the boot disk.
      primary# df /
      /                  (/dev/dsk/c0t5000CCA03C138904d0s0):22755742 blocks  2225374 files
    • For ZFS file systems, first run the df / command to determine the pool name. Then, run the zpool status command to determine the device path of the boot disk.
      primary# zpool status rpool
        pool: rpool
       state: ONLINE
        scan: none requested
      config:
      
              NAME                       STATE     READ WRITE CKSUM
              rpool                      ONLINE       0     0     0
                c0t5000CCA03C138904d0s0  ONLINE       0     0     0
  3. Obtain information about the system's boot disk.
    • For a disk that is managed with Solaris I/O multipathing, determine the PCIe bus to which the boot disk is connected by using the mpathadm command.

      Starting with the SPARC T4 servers, the internal disks are managed by Solaris I/O multipathing.

      1. Find the initiator port to which the disk is connected.
        primary# mpathadm show lu /dev/rdsk/c0t5000CCA03C138904d0s0
        Logical Unit:  /dev/rdsk/c0t5000CCA03C138904d0s2
                mpath-support:  libmpscsi_vhci.so
                Vendor:  HITACHI 
                Product:  H106030SDSUN300G
                Revision:  A2B0
                Name Type:  unknown type
                Name:  5000cca03c138904
                Asymmetric:  no
                Current Load Balance:  round-robin
                Logical Unit Group ID:  NA
                Auto Failback:  on
                Auto Probing:  NA
        
                Paths:  
                        Initiator Port Name:  w50800200014100c8
                        Target Port Name:  w5000cca03c138905
                        Override Path:  NA
                        Path State:  OK
                        Disabled:  no
        
                Target Ports:
                        Name:  w5000cca03c138905
                        Relative ID:  0
      2. Determine the PCIe bus on which the initiator port is present.
        primary# mpathadm show initiator-port w50800200014100c8
        Initiator Port:  w50800200014100c8
                Transport Type:  unknown
                OS Device File:  /devices/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@1
    • For a disk that is not managed with Solaris I/O multipathing, determine the physical device to which the block device is linked by using the ls -l command.

      The following example uses block device c1t0d0s0:

      primary# ls -l /dev/dsk/c0t1d0s0
      lrwxrwxrwx   1 root     root          49 Oct  1 10:39 /dev/dsk/c0t1d0s0 ->
      ../../devices/pci@400/pci@0/pci@1/scsi@0/sd@1,0:a

      In this example, the physical device for the primary domain's boot disk is connected to the pci@400 bus.

  4. Determine the network interface that is used by the system.

    Identify the primary network interface that is “plumbed” by using the ifconfig command. A plumbed interface has streams set up so that the IP protocol can use the device.

    primary# ifconfig -a
    lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000 
    net0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 3
            inet 10.129.241.135 netmask ffffff00 broadcast 10.129.241.255
            ether 0:10:e0:e:f1:78 
    
    primary# dladm show-phys net0
    LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
    net0              Ethernet             up         1000   full      igb0
  5. Determine the physical device to which the network interface is linked.

    The following command uses the igb0 network interface:

    primary# ls -l /dev/igb0
    lrwxrwxrwx   1 root     root          46 Oct  1 10:39 /dev/igb0  ->
    ../devices/pci@500/pci@0/pci@c/network@0:igb0

    Perform the ls -l /dev/usbecm command, as well.

    In this example, the physical device for the network interface used by the primary domain is under bus pci@500, which corresponds to the earlier listing of pci_1. So, the other two buses, pci_2 (pci@600) and pci_3 (pci@700), can safely be assigned to other domains because they are not used by the primary domain.

    If the network interface used by the primary domain is on a bus that you want to assign to another domain, reconfigure the primary domain to use a different network interface.

  6. Remove a bus that does not contain the boot disk or the network interface from the primary domain.

    In this example, the pci_2 bus is being removed from the primary domain.

    • Dynamic method:

      Ensure that the devices in the pci_2 bus are not in use by the primary domain OS. If they are, this command might fail to remove the bus. Use the static method to forcibly remove the pci_2 bus.

      primary# ldm remove-io pci_2 primary
    • Static method:

      Before you remove the bus, you must initiate a delayed reconfiguration.

      primary# ldm start-reconf primary
      primary# ldm remove-io pci_2 primary
      primary# shutdown -y -g0 -i6

    The bus that the primary domain uses for the boot disk and the network device cannot be assigned to other domains. You can assign any of the other buses to another domain. In this example, the pci@600 is not used by the primary domain, so you can reassign it to another domain.

  7. Add a bus to a domain.

    In this example, you add the pci_2 bus to the ldg1 domain.

    • Dynamic method:
      primary# ldm add-io pci_2 ldg1
    • Static method:

      Before you add the bus, you must stop the domain.

      primary# ldm stop-domain ldg1
      primary# ldm add-io pci_2 ldg1
      primary# ldm start-domain ldg1
  8. Save this SP configuration to the service processor.

    In this example, the SP configuration is io-domain.

    primary# ldm add-spconfig io-domain

    This SP configuration, io-domain, is also set as the next SP configuration to be used after the reboot.

  9. Confirm that the correct bus is still assigned to the primary domain and that the correct bus is assigned to domain ldg1.
    primary# ldm list-io
    NAME                                      TYPE   BUS      DOMAIN   STATUS
    ----                                      ----   ---      ------   ------
    /SYS/PM0/CMP0/PEX                         BUS    pci_0    primary  IOV
    /SYS/PM0/CMP1/PEX                         BUS    pci_1    primary
    /SYS/PM0/CMP2/PEX                         BUS    pci_2    ldg1
    /SYS/PM0/CMP3/PEX                         BUS    pci_3    primary
    /SYS/MB/PCIE1                             PCIE   pci_0    primary  EMP
    /SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
    /SYS/MB/NET0                              PCIE   pci_0    primary  OCC
    /SYS/MB/PCIE5                             PCIE   pci_1    primary  EMP
    /SYS/MB/PCIE6                             PCIE   pci_1    primary  EMP
    /SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP
    /SYS/MB/PCIE2                             PCIE   pci_2    ldg1     EMP
    /SYS/MB/PCIE3                             PCIE   pci_2    ldg1     EMP
    /SYS/MB/PCIE4                             PCIE   pci_2    ldg1     EMP
    /SYS/MB/PCIE8                             PCIE   pci_3    primary  EMP
    /SYS/MB/SASHBA1                           PCIE   pci_3    primary  OCC
    /SYS/MB/NET2                              PCIE   pci_3    primary  OCC
    /SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
    /SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
    /SYS/MB/NET2/IOVNET.PF0                   PF     pci_3    primary
    /SYS/MB/NET2/IOVNET.PF1                   PF     pci_3    primary

    This output confirms that PCIe buses pci_0, pci_1, and pci_3 and their devices are assigned to the primary domain. It also confirms that PCIe bus pci_2 and its devices are assigned to the ldg1 domain.