| Skip Navigation Links | |
| Exit Print View | |
|
Oracle VM Server for SPARC 2.0 Administration Guide |
1. Overview of the Oracle VM Server for SPARC Software
2. Installing and Enabling Software
4. Setting Up Services and the Control Domain
Assigning PCIe Endpoint Devices
Direct I/O Hardware and Software Requirements
Planning PCIe Endpoint Device Configuration
Create an I/O Domain by Assigning a PCIe Endpoint Device
12. Performing Other Administration Tasks
A. Oracle VM Server for SPARC Physical-to-Virtual Conversion Tool
B. Oracle VM Server for SPARC Configuration Assistant
C. Logical Domains Manager Discovery
D. Using the XML Interface With the Logical Domains Manager
You can use the Oracle VM Server for SPARC software to assign an entire PCIe bus (also known as a root complex) to a domain. An entire PCIe bus consists of the PCIe bus itself, and all of its PCI switches and devices. PCIe buses that are present on a server are identified with names such as pci@400 (pci_0). An I/O domain that is configured with an entire PCIe bus is also known as a root domain.
The following diagram shows a system that has two PCIe buses (pci_0 and pci_1). Each bus is assigned to a different domain. Thus, the system is configured with two I/O domains.
Figure 6-1 Assigning a PCIe Bus to an I/O Domain
The maximum number of I/O domains that you can create with PCIe buses depends on the number of PCIe buses that are available on the server. For example, if you are using a Sun SPARC Enterprise T5440 server, you can have up to four I/O domains.
Note - Some Sun UltraSPARC servers have only one PCIe bus. In such cases, you can create an I/O domain by assigning a PCIe endpoint (or direct I/O-assignable) device to a domain. See Assigning PCIe Endpoint Devices. If the system has a Network Interface Unit (NIU), you can also assign an NIU to a domain to create an I/O domain.
When you assign a PCIe bus to an I/O domain, all devices on that bus are owned by that I/O domain. You are not permitted to assign any of the PCIe endpoint devices on that bus to other domains. Only the PCIe endpoint devices on the PCIe buses that are assigned to the primary domain can be assigned to other domains.
When a server is initially configured in a Logical Domains environment or is using the factory-default configuration, the primary domain has access to all the physical device resources. This means that the primary domain is the only I/O domain configured on the system and that it owns all the PCIe buses.
This example procedure shows how to create a new I/O domain from an initial configuration where several buses are owned by the primary domain. By default the primary domain owns all buses present on the system. This example is for a Sun SPARC Enterprise T5440 server. This procedure can also be used on other servers. The instructions for different servers might vary slightly from these, but you can obtain the basic principles from this example.
First, you must retain the bus that has the primary domain's boot disk. Then, remove another bus from the primary domain and assign it to another domain.
 | Caution - All internal disks on the supported servers are connected to a single PCIe bus. If a domain is booted from an internal disk, do not remove that bus from the domain. Also, ensure that you are not removing a bus with devices (such as network ports) that are used by a domain. If you remove the wrong bus, a domain might not be able to access the required devices and could become unusable. To remove a bus that has devices that are used by a domain, reconfigure that domain to use devices from other buses. For example, you might have to reconfigure the domain to use a different onboard network port or a PCIe card from a different PCIe slot. |
In this example, the primary domain only uses a ZFS pool (rpool (c0t1d0s0)) and network interface (nxge0). If the primary domain uses more devices, repeat Steps 2-4 for each device to ensure that none are located on the bus that will be removed.
primary# ldm list-io IO PSEUDONYM DOMAIN -- --------- ------ pci@400 pci_0 primary pci@500 pci_1 primary pci@600 pci_2 primary pci@700 pci_3 primary PCIE PSEUDONYM STATUS DOMAIN ---- --------- ------ ------ pci@400/pci@0/pci@d MB/PCIE0 EMP - pci@400/pci@0/pci@c MB/PCIE1 OCC primary pci@400/pci@0/pci@1 MB/HBA OCC primary pci@500/pci@0/pci@d MB/PCIE4 EMP - pci@500/pci@0/pci@9 MB/PCIE5 EMP - pci@500/pci@0/pci@c MB/NET0 OCC primary pci@600/pci@0/pci@c MB/PCIE2 OCC primary pci@600/pci@0/pci@9 MB/PCIE3 OCC primary pci@700/pci@0/pci@c MB/PCIE6 OCC primary pci@700/pci@0/pci@9 MB/PCIE7 EMP -
primary# df / / (/dev/dsk/c0t1d0s0 ): 1309384 blocks 457028 files
primary# df /
/ (rpool/ROOT/s10s_u8wos_08a):245176332 blocks 245176332 files
primary# zpool status rpool
zpool status rpool
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c0t1d0s0 ONLINE 0 0 0The following example uses block device c1t0d0s0:
primary# ls -l /dev/dsk/c0t1d0s0 lrwxrwxrwx 1 root root 49 Oct 1 10:39 /dev/dsk/c0t1d0s0 -> ../../devices/pci@400/pci@0/pci@1/scsi@0/sd@1,0:a
In this example, the physical device for the primary domain's boot disk is connected to bus pci@400, which corresponds to the earlier listing of pci_0. This means that you cannot assign pci_0 (pci@400) to another domain.
primary# dladm show-dev vsw0 link: up speed: 1000 Mbps duplex: full nxge0 link: up speed: 1000 Mbps duplex: full nxge1 link: unknown speed: 0 Mbps duplex: unknown nxge2 link: unknown speed: 0 Mbps duplex: unknown nxge3 link: unknown speed: 0 Mbps duplex: unknown
Interfaces that are in the unknown state are not configured, so they are not used. In this example, the nxge0 interface is used.
The following command uses the nxge0 network interface:
primary# ls -l /dev/nxge0 lrwxrwxrwx 1 root root 46 Oct 1 10:39 /dev/nxge0 -> ../devices/pci@500/pci@0/pci@c/network@0:nxge0
In this example, the physical device for the network interface used by the primary domain is under bus pci@500, which corresponds to the earlier listing of pci_1. So, the other two buses, pci_2 (pci@600) and pci_3 (pci@700), can safely be assigned to other domains because they are not used by the primary domain.
If the network interface used by the primary domain was on a bus that you want to assign to another domain, the primary domain would need to be reconfigured to use a different network interface.
In this example, bus pci_2 and bus pci_3 are being removed from the primary domain. You might see a message from the ldm command that the primary domain is entering delayed reconfiguration mode.
primary# ldm remove-io pci_2 primary primary# ldm remove-io pci_3 primary
In this example, the configuration is io-domain.
primary# ldm add-config io-domain
This configuration, io-domain, is also set as the next configuration to be used after the reboot.
Note - Currently, there is a limit of 8 configurations that can be saved on the SP, not including the factory-default configuration.
primary# shutdown -i6 -g0 -y
The following example stops the ldg1 domain:
primary# ldm stop ldg1
The available bus is pci_2 and the domain is ldg1.
primary# ldm add-io pci_2 ldg1
The following commands restart the ldg1 domain:
primary# ldm start ldg1
primary# ldm list-io IO PSEUDONYM DOMAIN -- --------- ------ pci@400 pci_0 primary pci@500 pci_1 primary pci@600 pci_2 ldg1 pci@700 pci_3 PCIE PSEUDONYM STATUS DOMAIN ---- --------- ------ ------ pci@400/pci@0/pci@d MB/PCIE0 EMP - pci@400/pci@0/pci@c MB/PCIE1 OCC primary pci@400/pci@0/pci@1 MB/HBA OCC primary pci@500/pci@0/pci@d MB/PCIE4 EMP - pci@500/pci@0/pci@9 MB/PCIE5 EMP - pci@500/pci@0/pci@c MB/NET0 OCC primary pci@600/pci@0/pci@c MB/PCIE2 UNK - pci@600/pci@0/pci@9 MB/PCIE3 UNK - pci@700/pci@0/pci@c MB/PCIE6 UNK - pci@700/pci@0/pci@9 MB/PCIE7 UNK -
This output confirms that the PCIe buses pci_0 and pci_1 and the devices below them are assigned to domain primary, and that pci_2 and its devices are assigned to ldg1.
|