Skip Navigation Links | |
Exit Print View | |
![]() |
Oracle VM Server for SPARC 3.0 Administration Guide Oracle VM Server for SPARC |
Part I Oracle VM Server for SPARC 3.0 Software
1. Overview of the Oracle VM Server for SPARC Software
2. Installing and Enabling Software
3. Oracle VM Server for SPARC Security
4. Setting Up Services and the Control Domain
General Guidelines for Creating an I/O Domain
How to Create an I/O Domain by Assigning a PCIe Bus
Assigning PCIe Endpoint Devices
Direct I/O Hardware and Software Requirements
Current Direct I/O Feature Limitations
Planning PCIe Endpoint Device Configuration
How to Create an I/O Domain by Assigning a PCIe Endpoint Device
Using PCIe SR-IOV Virtual Functions
SR-IOV Hardware and Software Requirements
Current SR-IOV Feature Limitations
Planning for the Use of PCIe SR-IOV Virtual Functions
Creating, Modifying, and Destroying Virtual Functions
How to Create a Virtual Function
How to Modify a Virtual Function
How to Destroy a Virtual Function
Adding and Removing Virtual Functions on I/O Domains
How to Add a Virtual Function to an I/O Domain
How to Remove a Virtual Function From an I/O Domain
SR-IOV: Rebooting the Root Domain
Using an SR-IOV Virtual Function to Create an I/O Domain
How to Create an I/O Domain by Assigning an SR-IOV Virtual Function to It
SR-IOV Device-Specific Properties
Advanced Network Configuration for Virtual Functions
11. Managing Domain Configurations
12. Performing Other Administration Tasks
Part II Optional Oracle VM Server for SPARC Software
13. Oracle VM Server for SPARC Physical-to-Virtual Conversion Tool
14. Oracle VM Server for SPARC Configuration Assistant (Oracle Solaris 10)
16. Using the Oracle VM Server for SPARC Management Information Base Software
17. Logical Domains Manager Discovery
18. Using the XML Interface With the Logical Domains Manager
Starting with the Oracle VM Server for SPARC 2.0 release and the Oracle Solaris 10 9/10 OS, you can assign an individual PCIe endpoint (or direct I/O-assignable) device to a domain. This use of PCIe endpoint devices increases the granularity of the device assignment to I/O domains. This capability is delivered by means of the direct I/O (DIO) feature.
The DIO feature enables you to create more I/O domains than the number of PCIe buses in a system. The possible number of I/O domains is now limited only by the number of PCIe endpoint devices.
A PCIe endpoint device can be one of the following:
A PCIe card in a slot
An on-board PCIe device that is identified by the platform
The following diagram shows that the PCIe endpoint device, PCIE3, is assigned to an I/O domain. Both bus pci_0 and the switch in the I/O domain are virtual. The PCIE3 endpoint device is no longer accessible in the primary domain.
In the I/O domain, the pci_0 block and the switch are a virtual root complex and a virtual PCIe switch, respectively. This block and switch are very similar to the pci_0 block and the switch in the primary domain. In the primary domain, the devices in slot PCIE3 are a shadow form of the original devices and are identified as SUNW,assigned.
![]() | Caution - You cannot use Oracle Solaris hot-plug operations to “hot remove” a PCIe endpoint device after that device is removed from the primary domain by using the ldm rm-io command. For information about replacing or removing a PCIe endpoint device, see Making PCIe Hardware Changes. |
Figure 6-2 Assigning a PCIe Endpoint Device to an I/O Domain
Use the ldm list-io command to list the PCIe endpoint devices.
Though the DIO feature permits any PCIe card in a slot to be assigned to an I/O domain, only certain PCIe cards are supported. See Direct I/O Hardware and Software Requirements in Oracle VM Server for SPARC 3.0 Release Notes.
![]() | Caution - PCIe cards that have a bridge are not supported. PCIe function-level assignment is also not supported. Assigning an unsupported PCIe card to an I/O domain might result in unpredictable behavior. |
The following are a few important details about the DIO feature:
This feature is enabled only when all the software requirements are met. See Direct I/O Hardware and Software Requirements in Oracle VM Server for SPARC 3.0 Release Notes.
Only PCIe endpoints that are connected to a PCIe bus assigned to the primary domain can be assigned to another domain with the DIO feature.
I/O domains using DIO have access to the PCIe endpoint devices only when the primary domain is running.
Rebooting the root domain affects I/O domains that have PCIe endpoint devices. See Rebooting the Root Domain. The root domain also performs the following tasks:
Initializes the PCIe bus and manages the bus.
Handles all bus errors that are triggered by the PCIe endpoint devices that are assigned to I/O domains. Note that only the primary domain receives all PCIe bus-related errors.
To successfully use the DIO feature, you must be running the appropriate software and assign only the PCIe cards that are supported by the DIO feature to I/O domains. For the hardware and software requirements, see Direct I/O Hardware and Software Requirements in Oracle VM Server for SPARC 3.0 Release Notes.
Note - All PCIe cards that are supported on a platform are supported in the primary domain. See the documentation for your platform for the list of supported PCIe cards. However, only direct I/O-supported PCIe cards can be assigned to I/O domains.
To add or remove PCIe end-point devices by using the direct I/O feature, you must first enable I/O virtualization on the PCIe bus itself.
You can use the ldm set-io or ldm add-io command to set the iov property to on. You can also use the ldm add-domain or ldm set-domain command to set the rc-add-policy property to iov. See the ldm(1M) man page.
Rebooting the root domain affects direct I/O, so carefully plan your direct I/O configuration changes to maximize the direct I/O-related changes to the primary domain and to minimize primary domain reboots.
For information about how to work around the following limitations, see Planning PCIe Endpoint Device Configuration.
Assignment or removal of a PCIe endpoint device to any other domain is only permitted when that domain is either stopped or inactive.
Carefully plan ahead when you assign or remove PCIe endpoint devices to avoid primary domain downtime. The reboot of the primary domain not only affects the services that are available on the primary domain itself, but it also affects the I/O domains that have PCIe endpoint devices assigned. Though the changes to each I/O domain do not affect the other domains, planning ahead helps to minimize the consequences on the services that are provided by that domain.
When in a delayed reconfiguration, you can continue to add or remove more devices, and then reboot the primary domain only one time to make all the changes take effect.
For an example, see How to Create an I/O Domain by Assigning a PCIe Endpoint Device.
The following describes the general steps you must take to plan and perform a DIO device configuration:
Understand and record your system hardware configuration.
Specifically, record information about the part numbers and other details of the PCIe cards in the system.
Use the ldm list-io -l and prtdiag -v commands to obtain the information and save it for future reference.
Determine which PCIe endpoint devices are required to be in the primary domain.
For example, determine the PCIe endpoint devices that provide access to the following:
Boot disk device
Network device
Other devices that the primary domain offers as services
Remove all PCIe endpoint devices that you might use in I/O domains.
This step helps you to avoid performing subsequent reboot operations on the primary domain, as reboots affect I/O domains.
Use the ldm rm-io command to remove the PCIe endpoint devices. Use pseudonyms rather than device paths to specify the devices to the rm-io and add-io subcommands.
Note - After you have removed all the devices you want during a delayed reconfiguration, you only need to reboot the primary domain one time to make all the changes take effect.
Save this configuration to the service processor (SP).
Use the ldm add-config command.
Reboot the root domain to release the PCIe endpoint devices that you removed in Step 3.
Confirm that the PCIe endpoint devices you removed are no longer assigned to the primary domain.
Use the ldm list-io -l command to verify that the devices you removed appear as SUNW,assigned-device in the output.
Assign an available PCIe endpoint device to a guest domain to provide direct access to the physical device.
After you make this assignment, you can no longer migrate the guest domain to another physical system by means of the domain migration feature.
Add or remove a PCIe endpoint device to or from a guest domain.
Use the ldm add-io command.
Minimize the changes to I/O domains by reducing the reboot operations and by avoiding downtime of services offered by that domain.
(Optional) Make changes to the PCIe hardware.
The primary domain is the owner of the PCIe bus and is responsible for initializing and managing the bus. The primary domain must be active and running a version of the Oracle Solaris OS that supports the DIO feature. Shutting down, halting, or rebooting the primary domain interrupts access to the PCIe bus. When the PCIe bus is unavailable, the PCIe devices on that bus are affected and might become unavailable.
The behavior of I/O domains with PCIe endpoint devices is unpredictable when the primary domain is rebooted while those I/O domains are running. For instance, I/O domains with PCIe endpoint devices might panic during or after the reboot. Upon reboot of the primary domain, you would need to manually stop and start each domain.
To work around these issues, perform one of the following steps:
Manually shut down any domains on the system that have PCIe endpoint devices assigned to them before you shut down the primary domain.
This step ensures that these domains are cleanly shut down before you shut down, halt, or reboot the primary domain.
To find all the domains that have PCIe endpoint devices assigned to them, run the ldm list-io command. This command enables you to list the PCIe endpoint devices that have been assigned to domains on the system. So, use this information to help you plan. For a detailed description of this command output, see the ldm(1M) man page.
For each domain found, stop the domain by running the ldm stop command.
Configure a domain dependency relationship between the primary domain and the domains that have PCIe endpoint devices assigned to them.
This dependency relationship ensures that domains with PCIe endpoint devices are automatically restarted when the primary domain reboots for any reason.
Note that this dependency relationship forcibly resets those domains, and they cannot cleanly shut down. However, the dependency relationship does not affect any domains that were manually shut down.
# ldm set-domain failure-policy=reset primary # ldm set-domain master=primary ldom
The following steps help you avoid misconfiguring the PCIe endpoint assignments. For platform-specific information about installing and removing specific hardware, see the documentation for your platform.
No action is required if you are installing a PCIe card into an empty slot. This PCIe card is automatically owned by the domain that owns the PCIe bus.
To assign the new PCIe card to an I/O domain, use the ldm rm-io command to first remove the card from the primary domain. Then, use the ldm add-io command to assign the card to an I/O domain.
No action is required if a PCIe card is removed from the system and assigned to the primary domain.
To remove a PCIe card that is assigned to an I/O domain, first remove the device from the I/O domain. Then, add the device to the primary domain before you physically remove the device from the system.
To replace a PCIe card that is assigned to an I/O domain, verify that the new card is supported by the DIO feature.
If so, no action is required to automatically assign the new card to the current I/O domain.
If not, first remove that PCIe card from the I/O domain by using the ldm rm-io command. Next, use the ldm add-io command to reassign that PCIe card to the primary domain. Then, physically replace the PCIe card you assigned to the primary domain with a different PCIe card. These steps enable you to avoid a configuration that is unsupported by the DIO feature.
Plan all DIO deployments ahead of time to minimize downtime.
![]() | Caution - The primary domain loses access to the on-board DVD device if you assign the /SYS/MB/SASHBA1 slot on a SPARC T3-1 or a SPARC T4-1 system to a DIO domain. The SPARC T3-1 and SPARC T4-1 systems include two DIO slots for on-board storage, which are represented by the /SYS/MB/SASHBA0 and /SYS/MB/SASHBA1 paths. In addition to hosting multi-headed on-board disks, the /SYS/MB/SASHBA1 slot hosts the on-board DVD device. So, if you assign /SYS/MB/SASHBA1 to a DIO domain, the primary domain loses access to the on-board DVD device. The SPARC T3-2 and SPARC T4-2 systems have a single SASHBA slot that hosts all on-board disks as well as the on-board DVD device. So if you assign SASHBA to a DIO domain, the on-board disks and the on-board DVD device are loaned to the DIO domain and unavailable to the primary domain. |
For an example of adding a PCIe endpoint device to create an I/O domain, see Planning PCIe Endpoint Device Configuration.
Note - In this release, it is best use the DefaultFixed NCP to configure datalinks and network interfaces on Oracle Solaris 11 systems.
The Oracle Solaris 11 OS includes the following NCPs:
DefaultFixed. Enables you to use the dladm or ipadm command to manage networking.
Automatic. Enables you to use the netcfg or netadm command to manage networking.
Ensure that the DefaultFixed NCP is enabled by using the netadm list command. See Chapter 7, Using Datalink and Interface Configuration Commands on Profiles, in Oracle Solaris Administration: Network Interfaces and Network Virtualization.
The output of the ldm list-io -l command shows how the I/O devices are currently configured. You can obtain more detailed information by using the prtdiag -v command.
Note - After the devices are assigned to I/O domains, the identity of the devices can only be determined in the I/O domains.
# ldm list-io -l NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary [niu@480] niu_1 NIU niu_1 primary [niu@580] pci_0 BUS pci_0 primary [pci@400] pci_1 BUS pci_1 primary [pci@500] /SYS/MB/PCIE0 PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@8] SUNW,emlxs@0/fp/disk SUNW,emlxs@0/fp/tape SUNW,emlxs@0/fp@0,0 SUNW,emlxs@0,1/fp/disk SUNW,emlxs@0,1/fp/tape SUNW,emlxs@0,1/fp@0,0 /SYS/MB/PCIE2 PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@4] pci/scsi/disk pci/scsi/tape pci/scsi/disk pci/scsi/tape /SYS/MB/PCIE4 PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@0] ethernet@0 ethernet@0,1 SUNW,qlc@0,2/fp/disk SUNW,qlc@0,2/fp@0,0 SUNW,qlc@0,3/fp/disk SUNW,qlc@0,3/fp@0,0 /SYS/MB/PCIE6 PCIE pci_0 primary EMP [pci@400/pci@1/pci@0/pci@8] /SYS/MB/PCIE8 PCIE pci_0 primary EMP [pci@400/pci@1/pci@0/pci@c] /SYS/MB/SASHBA PCIE pci_0 primary OCC [pci@400/pci@2/pci@0/pci@e] scsi@0/iport@1 scsi@0/iport@2 scsi@0/iport@4 scsi@0/iport@8 scsi@0/iport@80/cdrom@p7,0 scsi@0/iport@v0 /SYS/MB/NET0 PCIE pci_0 primary OCC [pci@400/pci@1/pci@0/pci@4] network@0 network@0,1 /SYS/MB/PCIE1 PCIE pci_1 primary OCC [pci@500/pci@2/pci@0/pci@a] SUNW,qlc@0/fp/disk SUNW,qlc@0/fp@0,0 SUNW,qlc@0,1/fp/disk SUNW,qlc@0,1/fp@0,0 /SYS/MB/PCIE3 PCIE pci_1 primary OCC [pci@500/pci@2/pci@0/pci@6] network@0 network@0,1 network@0,2 network@0,3 /SYS/MB/PCIE5 PCIE pci_1 primary OCC [pci@500/pci@2/pci@0/pci@0] network@0 network@0,1 /SYS/MB/PCIE7 PCIE pci_1 primary EMP [pci@500/pci@1/pci@0/pci@6] /SYS/MB/PCIE9 PCIE pci_1 primary EMP [pci@500/pci@1/pci@0/pci@0] /SYS/MB/NET2 PCIE pci_1 primary OCC [pci@500/pci@1/pci@0/pci@5] network@0 network@0,1 ethernet@0,80 /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary [pci@400/pci@1/pci@0/pci@4/network@0] maxvfs = 7 /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary [pci@400/pci@1/pci@0/pci@4/network@0,1] maxvfs = 7 /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary [pci@500/pci@2/pci@0/pci@0/network@0] maxvfs = 63 /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary [pci@500/pci@2/pci@0/pci@0/network@0,1] maxvfs = 63 /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary [pci@500/pci@1/pci@0/pci@5/network@0] maxvfs = 7 /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary [pci@500/pci@1/pci@0/pci@5/network@0,1] maxvfs = 7
See Step 2 in How to Create an I/O Domain by Assigning a PCIe Bus.
See Step 3 in How to Create an I/O Domain by Assigning a PCIe Bus.
See Step 4 in How to Create an I/O Domain by Assigning a PCIe Bus.
The following command uses the igb0 network interface:
primary# ls -l /dev/igb0 lrwxrwxrwx 1 root root 46 Jul 30 17:29 /dev/igb0 -> ../devices/pci@500/pci@0/pci@8/network@0:igb0
In this example, the physical device for the network interface used by the primary domain is connected to the PCIe endpoint device (pci@500/pci@0/pci@8), which corresponds to the listing of MB/NET0 in Step 1. So, you do not want to remove this device from the primary domain. You can safely assign all other PCIe devices to other domains because they are not used by the primary domain.
If the network interface used by the primary domain is on a bus that you want to assign to another domain, the primary domain would need to be reconfigured to use a different network interface.
In this example, you can remove the PCIE2, PCIE3, PCIE4, and PCIE5 endpoint devices because they are not being used by the primary domain.
![]() | Caution - Do not remove the devices that are used or required by the primary domain. If you mistakenly remove the wrong devices, use the ldm cancel-reconf primary command to cancel the delayed reconfiguration on the primary domain. |
You can remove multiple devices at one time to avoid multiple reboots.
# ldm start-reconf primary primary# ldm set-io iov=on pci_1 All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. primary# ldm remove-io /SYS/MB/PCIE1 primary ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ primary# ldm remove-io /SYS/MB/PCIE3 primary ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ primary# ldm remove-io /SYS/MB/PCIE5 primary ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------
The following command saves the configuration in a file called dio:
# ldm add-config dio
# reboot -- -r
# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary pci_0 BUS pci_0 primary pci_1 BUS pci_1 primary IOV /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 OCC /SYS/MB/PCIE3 PCIE pci_1 OCC /SYS/MB/PCIE5 PCIE pci_1 OCC /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE9 PCIE pci_1 primary EMP /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary
Note - The ldm list-io -l output might show SUNW,assigned-device for the PCIe endpoint devices that were removed. Actual information is no longer available from the primary domain, but the domain to which the device is assigned has this information.
# ldm add-io /SYS/MB/PCIE3 ldg1
# ldm bind ldg1 # ldm start ldg1 LDom ldg1 started
Verify that the network device is available and then configure the network device for use in the domain.
# dladm show-dev nxge0 link: unknown speed: 0 Mbps duplex: unknown nxge1 link: unknown speed: 0 Mbps duplex: unknown nxge2 link: unknown speed: 0 Mbps duplex: unknown nxge3 link: unknown speed: 0 Mbps duplex: unknown
# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net0 Ethernet unknown 0 unknown nxge0 net1 Ethernet unknown 0 unknown nxge1 net2 Ethernet unknown 0 unknown nxge2 net3 Ethernet unknown 0 unknown nxge3