Assigning PCIe Buses

You can use the Oracle VM Server for SPARC software to assign an entire PCIe bus (also known as a root complex) to a domain. An entire PCIe bus consists of the PCIe bus itself, and all of its PCI switches and devices. PCIe buses that are present on a server are identified with names such as pci@400 (pci_0). An I/O domain that is configured with an entire PCIe bus is also known as a root domain.

The following diagram shows a system that has two PCIe buses (pci_0 and pci_1). Each bus is assigned to a different domain. Thus, the system is configured with two I/O domains.

Figure 6-1 Assigning a PCIe Bus to an I/O Domain

image:Diagram shows how to assign a PCIe bus to an I/O domain.

The maximum number of I/O domains that you can create with PCIe buses depends on the number of PCIe buses that are available on the server. For example, if you are using an Oracle Sun SPARC Enterprise T5440 server, you can have up to four I/O domains.

Note - Some UltraSPARC servers have only one PCIe bus. In such cases, you can create an I/O domain by assigning a PCIe endpoint (or direct I/O-assignable) device to a domain. See Assigning PCIe Endpoint Devices. If the system has a Network Interface Unit (NIU), you can also assign an NIU to a domain to create an I/O domain.

When you assign a PCIe bus to an I/O domain, all devices on that bus are owned by that I/O domain. You are not permitted to assign any of the PCIe endpoint devices on that bus to other domains. Only the PCIe endpoint devices on the PCIe buses that are assigned to the primary domain can be assigned to other domains.

When a server is initially configured in a Logical Domains environment or is using the factory-default configuration, the primary domain has access to all the physical device resources. This means that the primary domain is the only I/O domain configured on the system and that it owns all the PCIe buses.

How to Create an I/O Domain by Assigning a PCIe Bus

This example procedure shows how to create a new I/O domain from an initial configuration where several buses are owned by the primary domain. By default the primary domain owns all buses present on the system. This example is for a SPARC T4-2 server. This procedure can also be used on other servers. The instructions for different servers might vary slightly from these, but you can obtain the basic principles from this example.

First, you must retain the bus that has the primary domain's boot disk. Then, remove another bus from the primary domain and assign it to another domain.

Caution - All internal disks on the supported servers might be connected to a single PCIe bus. If a domain is booted from an internal disk, do not remove that bus from the domain. Also, ensure that you are not removing a bus with devices (such as network ports) that are used by a domain. If you remove the wrong bus, a domain might not be able to access the required devices and could become unusable. To remove a bus that has devices that are used by a domain, reconfigure that domain to use devices from other buses. For example, you might have to reconfigure the domain to use a different on-board network port or a PCIe card from a different PCIe slot.

In this example, the primary domain only uses a ZFS pool (rpool (c0t1d0s0)) and network interface (igb0). If the primary domain uses more devices, repeat Steps 2-4 for each device to ensure that none are located on the bus that will be removed.

Verify that the primary domain owns more than one PCIe bus.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
niu_0                                     NIU    niu_0    primary           
niu_1                                     NIU    niu_1    primary           
pci_0                                     BUS    pci_0    primary           
pci_1                                     BUS    pci_1    primary           
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE3                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE5                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP      
/SYS/MB/PCIE9                             PCIE   pci_1    primary  EMP      
/SYS/MB/NET2                              PCIE   pci_1    primary  OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    primary           
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    primary

Determine the device path of the boot disk that must be retained.

For UFS file systems, run the df / command to determine the device path of the boot disk.

primary# df /
/                  (/dev/dsk/c0t5000CCA03C138904d0s0):22755742 blocks  2225374 files

For ZFS file systems, first run the df / command to determine the pool name. Then, run the zpool status command to determine the device path of the boot disk.

primary# zpool status rpool
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        rpool                      ONLINE       0     0     0
          c0t5000CCA03C138904d0s0  ONLINE       0     0     0

Determine the PCIe bus under which the boot disk is connected.

Find the initiator port to which the disk is connected.

primary# mpathadm show lu /dev/rdsk/c0t5000CCA03C138904d0s0
Logical Unit:  /dev/rdsk/c0t5000CCA03C138904d0s2
        mpath-support:  libmpscsi_vhci.so
        Vendor:  HITACHI 
        Product:  H106030SDSUN300G
        Revision:  A2B0
        Name Type:  unknown type
        Name:  5000cca03c138904
        Asymmetric:  no
        Current Load Balance:  round-robin
        Logical Unit Group ID:  NA
        Auto Failback:  on
        Auto Probing:  NA

        Paths:  
                Initiator Port Name:  w50800200014100c8
                Target Port Name:  w5000cca03c138905
                Override Path:  NA
                Path State:  OK
                Disabled:  no

        Target Ports:
                Name:  w5000cca03c138905
                Relative ID:  0

Determine on which PCIe bus the initiator port is present.

primary# mpathadm show initiator-port w50800200014100c8
Initiator Port:  w50800200014100c8
        Transport Type:  unknown
        OS Device File:  /devices/pci@400/pci@2/pci@0/pci@e/scsi@0/iport@1

Determine the network interface that is used by the system.

Identify the primary network interface that is “plumbed” by using the ifconfig command.

Oracle Solaris 10.

primary# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
igb0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 3
        inet 10.129.241.135 netmask ffffff00 broadcast 10.129.241.255
        ether 0:10:e0:e:f1:78

Oracle Solaris 11.

primary# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
net0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 3
        inet 10.129.241.135 netmask ffffff00 broadcast 10.129.241.255
        ether 0:10:e0:e:f1:78 

primary# dladm show-phys net0
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      igb0

Determine the physical device to which the network interface is linked.
The following command uses the igb0 network interface:
```
primary# ls -l /dev/igb0
lrwxrwxrwx   1 root     root          46 Oct  1 10:39 /dev/igb0  ->
../devices/pci@500/pci@0/pci@c/network@0:igb0 
```
In this example, the physical device for the network interface used by the primary domain is under bus pci@500, which corresponds to the earlier listing of pci_1. So, the other two buses, pci_2 (pci@600) and pci_3 (pci@700), can safely be assigned to other domains because they are not used by the primary domain.
If the network interface used by the primary domain was on a bus that you want to assign to another domain, the primary domain would need to be reconfigured to use a different network interface.
Remove a bus that does not contain the boot disk or the network interface from the primary domain.
In this example, bus pci_1 is being removed from the primary domain. You must also initiate a delayed reconfiguration.
```
primary# ldm start-reconf primary
primary# ldm remove-io pci_1 primary
```
Save this configuration to the service processor.
In this example, the configuration is io-domain.
```
primary# ldm add-config io-domain
```
This configuration, io-domain, is also set as the next configuration to be used after the reboot.
Reboot the root domain so that the change takes effect.
```
primary# shutdown -i6 -g0 -y
```
Stop the domain to which you want to add the PCIe bus.
The following example stops the ldg1 domain:
```
primary# ldm stop ldg1
```
Add the available bus to the domain that needs direct access.
The available bus is pci_1 and the domain is ldg1.
```
primary# ldm add-io pci_1 ldg1
```
Restart the domain so that the change takes affect.
The following commands restart the ldg1 domain:
```
primary# ldm start ldg1
```

Confirm that the correct bus is still assigned to the primary domain and that the correct bus is assigned to domain ldg1.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
niu_0                                     NIU    niu_0    primary           
niu_1                                     NIU    niu_1    primary           
pci_0                                     BUS    pci_0    primary  
pci_1                                     BUS    pci_1    ldg1    
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE3                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE5                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    ldg1     EMP      
/SYS/MB/PCIE9                             PCIE   pci_1    ldg1     EMP      
/SYS/MB/NET2                              PCIE   pci_1    ldg1     OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    ldg1              
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    ldg1              
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    ldg1              
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    ldg1

This output confirms that PCIe bus pci_0 and its devices are assigned to the primary domain. It also confirms that PCIe bus pci_1 and its devices are assigned to the ldg1 domain.

Skip Navigation Links
Exit Print View
	Oracle VM Server for SPARC 3.0 Administration Guide Oracle VM Server for SPARC