Skip Navigation Links | |
Exit Print View | |
Oracle VM Server for SPARC 3.0 Administration Guide Oracle VM Server for SPARC |
Part I Oracle VM Server for SPARC 3.0 Software
1. Overview of the Oracle VM Server for SPARC Software
2. Installing and Enabling Software
3. Oracle VM Server for SPARC Security
4. Setting Up Services and the Control Domain
General Guidelines for Creating an I/O Domain
How to Create an I/O Domain by Assigning a PCIe Bus
Assigning PCIe Endpoint Devices
Direct I/O Hardware and Software Requirements
Current Direct I/O Feature Limitations
Planning PCIe Endpoint Device Configuration
How to Create an I/O Domain by Assigning a PCIe Endpoint Device
Using PCIe SR-IOV Virtual Functions
SR-IOV Hardware and Software Requirements
Current SR-IOV Feature Limitations
Planning for the Use of PCIe SR-IOV Virtual Functions
Creating, Modifying, and Destroying Virtual Functions
How to Create a Virtual Function
How to Modify a Virtual Function
How to Destroy a Virtual Function
Adding and Removing Virtual Functions on I/O Domains
How to Add a Virtual Function to an I/O Domain
How to Remove a Virtual Function From an I/O Domain
SR-IOV: Rebooting the Root Domain
Using an SR-IOV Virtual Function to Create an I/O Domain
How to Create an I/O Domain by Assigning an SR-IOV Virtual Function to It
11. Managing Domain Configurations
12. Performing Other Administration Tasks
Part II Optional Oracle VM Server for SPARC Software
13. Oracle VM Server for SPARC Physical-to-Virtual Conversion Tool
14. Oracle VM Server for SPARC Configuration Assistant (Oracle Solaris 10)
16. Using the Oracle VM Server for SPARC Management Information Base Software
17. Logical Domains Manager Discovery
18. Using the XML Interface With the Logical Domains Manager
Support for the Peripheral Component Interconnect Express (PCIe) single root I/O virtualization (SR-IOV) feature has been added starting with the Oracle VM Server for SPARC 2.2 release.
The SR-IOV implementation is based on version 1.1 of the standard as defined by the PCI-SIG. The SR-IOV standard enables the efficient sharing of PCIe devices among virtual machines and is implemented in the hardware to achieve I/O performance that is comparable to native performance. The SR-IOV specification defines a new standard wherein new devices that are created enable the virtual machine to be directly connected to the I/O device.
A single I/O resource, which is known as a physical function, can be shared by many virtual machines. The shared devices provide dedicated resources and also use shared common resources. In this way, each virtual machine has access to unique resources. Therefore, a PCIe device, such as an Ethernet port, that is SR-IOV-enabled with appropriate hardware and OS support can appear as multiple, separate physical devices, each with its own PCIe configuration space.
For more information about SR-IOV, see the PCI-SIG web site.
The following figure shows the relationship between virtual functions and a physical function in an I/O domain.
Figure 6-3 Using Virtual Functions and a Physical Function in an I/O Domain
SR-IOV has the following function types:
Physical function – A PCI function that supports the SR-IOV capabilities as defined by the SR-IOV specification. A physical function contains the SR-IOV capability structure and manages the SR-IOV functionality. Physical functions are fully featured PCIe functions that can be discovered, managed, and manipulated like any other PCIe device. Physical functions can be used to configure and control a PCIe device.
Virtual function – A PCI function that is associated with a physical function. A virtual function is a lightweight PCIe function that shares one or more physical resources with the physical function and with virtual functions that are associated with that physical function. Unlike a physical function, a virtual function can only configure its own behavior.
Each SR-IOV device can have a physical function and each physical function can have up to 64,000 virtual functions associated with it. This number is dependent on the particular SR-IOV device. The virtual functions are created by the physical function.
After SR-IOV is enabled in the physical function, the PCI configuration space of each virtual function can be accessed by the bus, device, and function number of the physical function. Each virtual function has a PCI memory space, which is used to map its register set. The virtual function device drivers operate on the register set to enable its functionality and the virtual function appears as an actual PCI device. After creation, you can directly assign a virtual function to an I/O domain. This capability enables the virtual function to share the physical device and to perform I/O without CPU and hypervisor software overhead.
The following benefits are associated with devices that have SR-IOV capabilities:
Higher performance and reduced latency – Direct access to hardware from a virtual machines environment
Cost reduction – Capital and operational expenditure savings, which include:
Power savings
Reduced adapter count
Less cabling
Fewer switch ports
Support for the PCIe SR-IOV feature has been added starting with the Oracle VM Server for SPARC 2.2 release. For information about the required hardware, software, and firmware versions, see PCIe SR-IOV Hardware and Software Requirements in Oracle VM Server for SPARC 3.0 Release Notes.
To create and destroy virtual functions in the SR-IOV physical function devices, you must first enable I/O virtualization on that PCIe bus.
You can use the ldm set-io or ldm add-io command to set the iov property to on. You can also use the ldm add-domain or ldm set-domain command to set the rc-add-policy property to iov. See the ldm(1M) man page.
Rebooting the root domain affects SR-IOV, so carefully plan your direct I/O configuration changes to maximize the SR-IOV related changes to the root domain and to minimize root domain reboots.
Note - To minimize reboots, perform multiple operations within the same delayed reconfiguration.
The SR-IOV feature has the following limitations in this release:
Migration is disabled for any domain that has one or more virtual functions assigned to it.
You can only destroy the last virtual function that was created for a physical function. So, if you create three virtual functions, the first virtual function that you can destroy must be the third one.
Only Ethernet SR-IOV cards are supported.
The SR-IOV feature is enabled only for the SR-IOV cards that are installed on the primary domain. If an SR-IOV card is assigned to a domain, by using either PCIe bus assignment or the Direct I/O (DIO) feature, the SR-IOV feature is not enabled for that card.
You can enable VLAN configurations of virtual functions by setting either the pvid or the vid property. You cannot set both of these virtual function properties simultaneously.
Before you create and destroy virtual functions, plan ahead to determine the virtual functions that you want to use in your configuration. When you create and destroy virtual functions, the primary domain must be rebooted. Such a reboot negatively affects all of the I/O domains that have either PCIe endpoints or the SR-IOV virtual functions configured. So, it is important to minimize the number of primary domain reboots. Determine the number of virtual functions that you require from the various SR-IOV devices to satisfy your current and future configuration needs.
For information about I/O domains, see General Guidelines for Creating an I/O Domain.
Use the following general steps to plan and perform SR-IOV virtual function configuration and assignment:
Determine the PCIe SR-IOV physical functions that are available on your system and which ones are best suited to your needs.
Use the following commands to identify the required information:
Identifies the available SR-IOV physical function devices.
Identifies which PCIe SR-IOV cards and on-board devices are available.
Identifies additional information about a specified physical function, such as the maximum number of virtual functions that are supported by the device.
Identifies the device-specific properties that are supported by the device. See Advanced SR-IOV Topics.
Initiate a delayed reconfiguration on the root domain.
primary# ldm start-reconf primary
Enable I/O virtualization operations for a PCIe bus.
primary# ldm set-io iov=on bus
Create the required number of virtual functions on the specified SR-IOV physical function.
Use the following command to create the virtual function:
primary# ldm create-vf pf-name
Use the ldm create-vf command to set device-specific and network-specific properties of a virtual function. The unicast-slots property is device-specific. The mac-addr, alt-mac-addrs, mtu, pvid, and vid properties are network-specific.
Note that the mac-addr, alt-mac-addrs, and mtu network-specific properties can be changed as follows:
When the virtual function is assigned to the primary domain: You must initiate a delayed reconfiguration.
When the virtual function is assigned to an active I/O domain: A property change request is rejected because the change must be made when the owning domain is in the inactive or bound state.
When the virtual function is assigned to a non-primary domain and a delayed reconfiguration is already in effect: A property change request fails with an error message.
The pvid and vid network-specific properties can be changed without restriction.
During a delayed reconfiguration, you can create more virtual functions and perform just one primary domain reboot to make the changes take effect. You do not need to reboot the primary domain after you create each virtual function.
A particular SR-IOV physical function might support many virtual functions. Create only the virtual functions that you require. For the recommended maximum number of configurations, see Advanced SR-IOV Topics.
Use the ldm add-config command to save the configuration to the SP.
Reboot the root domain to create the virtual function.
An active domain must be stopped before you use the ldm add-io command to assign a virtual function to it. Minimize I/O domain downtime by collectively making all I/O domain changes together. This method enables you to reduce the number of primary domain reboots that is required to set up such configurations.
Boot the I/O domains and configure the virtual functions as if they were any other network devices.
For information about virtual function limitations, see Advanced SR-IOV Topics.
This section describes how to create, modify, and destroy virtual functions.
primary# ldm list-io
Note that the name of the physical function includes the location information for the PCIe SR-IOV card or on-board device.
Initiate a delayed reconfiguration on the control domain.
primary# ldm start-reconf primary
primary# ldm set-io iov=on bus
primary# ldm create-vf [mac-addr=num] [alt-mac-addrs=[auto|num1,[auto|num2,...]]] [pvid=pvid] [vid=vid1,vid2,...] [mtu=size] [name=value...] pf-name
Note - The MAC address is automatically allocated for network devices.
You can use either the path name or the pseudonym name to specify virtual functions. However, it is best to use the pseudonym name.
Example 6-1 Creating a Virtual Function
The following example shows information about the the physical function, /SYS/MB/NET0/IOVNET.PF0:
This physical function is from an on-board NET0 network device.
The IOVNET string indicates that the physical function is a network SR-IOV device.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary pci_0 BUS pci_0 primary pci_1 BUS pci_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 primary OCC /SYS/MB/PCIE3 PCIE pci_1 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary OCC /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE9 PCIE pci_1 primary EMP /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary
The following command shows more details about the specified physical function. The maxvfs value indicates the maximum number of virtual functions that is supported by the device.
primary# ldm list-io -l /SYS/MB/NET0/IOVNET.PF0 NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary [pci@400/pci@1/pci@0/pci@4/network@0] maxvfs = 7
The following examples show how to create a virtual function:
This example creates a virtual function without setting any optional properties. In this case, the MAC address for a network class virtual function is automatically allocated.
primary# ldm start-reconf primary Initiating a delayed reconfiguration operation on the primary domain. All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. primary# ldm set-io iov=on pci_0 primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0 ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0
This example creates a virtual function while setting the mac-addr property to 00:14:2f:f9:14:c0 and the vid property to VLAN IDs 2 and 3.
primary# ldm create-vf mac-addr=00:14:2f:f9:14:c0 vid=2,3 /SYS/MB/NET0/IOVNET.PF0
This example creates a virtual function that has two alternate MAC addresses. One MAC address is automatically allocated, and the other is explicitly specified as 00:14:2f:f9:14:c2.
primary# ldm create-vf alt-mac-addrs=auto,00:14:2f:f9:14:c2 /SYS/MB/NET0/IOVNET.PF0
The ldm set-io bus command modifies the current configuration of a virtual function by changing the property values or by setting new properties. This command can modify both the network-specific properties and the device-specific properties. For information about device-specific properties, see Advanced SR-IOV Topics.
You can use the ldm set-io command to modify the following properties:
mac-addr, alt-mac-addrs, and mtu
To change these virtual function properties, first stop the domain that owns the virtual function and initiate a delayed reconfiguration. You must reboot the domain to make the changes take effect.
pvid and vid
You can dynamically change these properties while the virtual functions are assigned to a domain. Note that doing so might result in a change to the network traffic of an active virtual function. Namely, setting the pvid property enables a transparent VLAN. Setting the vid property to specify VLAN IDs permits VLAN traffic to those specified VLANs.
Device-specific properties
Use the ldm list-io -d pf-name command to view the list of valid device-specific properties. You can modify these properties for both the physical function and the virtual function. You must initiate a delayed reconfiguration and reboot the primary domain to make the changes take effect. For more information about device-specific properties, see Advanced SR-IOV Topics.
primary# ldm set-io name=value [name=value...] vf-name
Example 6-2 Modifying a Virtual Function
The following examples show how to use the ldm set-io command to set properties on a virtual function:
The following example modifies the specified virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, to be part of VLAN IDs 2, 3, and 4:
primary# ldm set-io vid=2,3,4 /SYS/MB/NET0/IOVNET.PF0.VF0
Note that this command dynamically changes the VLAN association for a virtual function. To use these VLANs, the VLAN interfaces in the I/O domains must be configured by using the appropriate Oracle Solaris OS networking commands.
The following example sets the pvid property value to 2 for the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function, which transparently makes the virtual function part of VLAN 2. Namely, the virtual function will not view any tagged VLAN traffic.
primary# ldm set-io pvid=2 /SYS/MB/NET0/IOVNET.PF0.VF0
The following example assigns three automatically allocated alternate MAC addresses to a virtual function. The alternate addresses enable the creation of Oracle Solaris 11 virtual network interface cards (VNICs) on top of a virtual function. Note that to use VNICs, you must run the Oracle Solaris 11 OS in the domain.
Note - Before you run this command, stop the domain that owns the virtual function.
primary# ldm set-io alt-mac-addrs=auto,auto,auto /SYS/MB/NET0/IOVNET.PF0.VF0
The following example sets the device-specific unicast-slots property to 12 for the specified virtual function. To find the device-specific properties that are valid for a physical function, use the ldm list-io -d pf-name command.
primary# ldm set-io unicast-slots=12 /SYS/MB/NET0/IOVNET.PF0.VF0 All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect.
A virtual function can be destroyed if it is not currently assigned to a domain. A virtual function can only be destroyed in the reverse sequential order of creation, so only the last virtual function that was created can be destroyed. The resulting configuration is validated by the physical function driver. You must initiate a delayed reconfiguration and reboot the domain to make the change take effect.
primary# ldm start-reconf primary
primary# ldm destroy-vf vf-name
Example 6-3 Destroying a Virtual Function
The following example shows how to destroy the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function:
primary# ldm destroy-vf /SYS/MB/NET0/IOVNET.PF0.VF0 All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect.
The following command adds a virtual function to a logical domain:
ldm add-io vf-name ldom
vf-name is the pseudonym name or the path name of the virtual function. It is best to use the pseudonym name. ldom specifies the name of the domain to which you add the virtual function. The specified guest must be in the inactive or bound state. If you specify the primary domain, you must initiate a delayed reconfiguration.
primary# ldm add-io vf-name ldom
The device path name for the virtual function in the domain is the path shown in the list-io -l output.
Example 6-4 Adding a Virtual Function
The following example shows how to add the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function to the ldg1 domain. To succeed, the specified domain must be in the inactive or bound state. If the domain is the primary domain, you must initiate a delayed reconfiguration.
primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
If the command succeeds, the virtual function is added to the ldg1 domain. If ldg1 is already bound (or is subsequently bound), the domain can be started and the guest OS can use the added virtual function for I/O operations.
The following command removes an SR-IOV virtual function from a logical domain:
ldm remove-io vf-name ldom
vf-name is the pseudonym name or the path name of the virtual function. It is best to use the device pseudonym. ldom specifies the name of the domain from which you remove the virtual function. The specified guest must be in the inactive or bound state. If you specify the primary domain, you must initiate a delayed reconfiguration.
Note - Before removing the virtual function from the domain, ensure that it is not critical for booting that domain.
primary# ldm rm-io vf-name ldom
Example 6-5 Removing a Virtual Function
The following example shows how to remove the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function from the ldg1 domain:
primary# ldm rm-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
If the command succeeds, the virtual function is removed from the ldg1 domain. When ldg1 is restarted, the specified virtual function no longer appears in that domain.
If the domain that has the virtual function is the primary domain, you must initiate a delayed reconfiguration.
Take care when rebooting the primary domain. See Rebooting the Root Domain. As with PCIe slots in the I/O domain, the concerns that are described in this section also pertain to the virtual functions that are assigned to an I/O domain.
The following procedure explains how to create an I/O domain that includes PCIe SR-IOV virtual functions.
Plan ahead to minimize the number of reboots of the primary domain, which minimizes the downtime.
primary# ldm list-io
primary# ldm list-io -l pf-name
primary# ldm start-reconf primary
primary# ldm set-io iov=on bus
primary# ldm create-vf pf-name
You can run this command for each virtual function that you want to create. If you perform these commands as a batch, you only need to reboot the primary domain one time.
primary# ldm stop ldom
primary# reboot
primary# ldm list-io
primary# ldm add-io vf-name ldom
primary# ldm bind ldom primary# ldm start ldom
The following Oracle Solaris 11 command shows the availability of the virtual function:
guest# dladm show-phys
Example 6-6 Creating an I/O Domain by Assigning an SR-IOV Virtual Function to It
The following example shows how to create a virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, for a physical function, /SYS/MB/NET0/IOVNET.PF0, and assign the virtual function to the ldg1 I/O domain.
The following ldm list-io output shows that the /SYS/MB/NET0/IOVNET.PF0 physical function is available:
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary pci_0 BUS pci_0 primary pci_1 BUS pci_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 primary OCC /SYS/MB/PCIE3 PCIE pci_1 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary OCC /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE9 PCIE pci_1 primary EMP /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary
The following command shows additional details about the /SYS/MB/NET0/IOVNET.PF0 physical function, which includes the maximum number of virtual functions that can be created:
primary# ldm list-io -l /SYS/MB/NET0/IOVNET.PF0 NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary [pci@400/pci@1/pci@0/pci@4/network@0] maxvfs = 7
The following command creates a virtual function called /SYS/MB/NET0/IOVNET.PF0.VF0 for the /SYS/MB/NET0/IOVNET.PF0 physical function:
primary# ldm start-reconf primary primary# ldm set-io iov=on pci_0 primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0 ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0
Because the ldg1 I/O domain has a PCIe endpoint device that was created by using the DIO feature, you must stop the ldg1 domain and reboot the primary domain, as follows:
primary# ldm stop ldg1 primary# reboot
The following command verifies that the new /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function exists:
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary pci_0 BUS pci_0 primary IOV pci_1 BUS pci_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 primary OCC /SYS/MB/PCIE3 PCIE pci_1 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary OCC /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE9 PCIE pci_1 primary EMP /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary /SYS/MB/NET0/IOVNET.PF0.VF0 VF pci_0
The following command assigns the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function to the ldg1 domain:
primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
The following commands bind and restart the ldg1 domain:
primary# ldm bind ldg1 primary# ldm start ldg1
The following command verifies that the virtual function is available for use:
guest# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net0 Ethernet up 0 unknown vnet0 net1 Ethernet up 1000 full igbvf0
This section describes several advanced topics that arise when you use PCIe SR-IOV-capable I/O devices.
Booting an I/O domain by using an SR-IOV virtual function. An SR-IOV virtual function provides similar capabilities as any other type of PCIe device, such as the ability to use a virtual function as a logical domain boot device. For example, a network virtual function can be used to boot over the network to install the Oracle Solaris OS in an I/O domain.
Note - When booting the Oracle Solaris OS from a virtual function device, verify that the Oracle Solaris OS being loaded has virtual function device support. If so, you can continue with the rest of the installation, as planned.
Maximum number of I/O domains and virtual functions supported. The PCIe endpoint devices and SR-IOV virtual functions from a particular PCIe bus can be assigned up to a maximum of 15 domains. The PCIe resources, such as interrupt vectors for each PCIe bus, are divided among the root domain and I/O domains. As a result, the number of devices that you can assign to a particular I/O domain is also limited. So, ensure that you do not assign a large number virtual functions to the same I/O domain. For a description of the problems related to SR-IOV, see Oracle VM Server for SPARC 3.0 Release Notes.
SR-IOV physical function device drivers can export device-specific properties. These properties can be used to tune the resource allocation of both the physical function and its virtual functions. For information about the properties, see the man page for the physical function driver, such as the igb(7D) and ixgbe(7D) man pages.
The ldm list-io -d command shows device-specific properties that are exported by the specified physical function device driver. Each property has a name, brief description, default value, maximum values, and one or more of the following flags:
Applies to a physical function
Applies to a virtual function
Read-only or informative parameter only
primary# ldm list-io -d pf-name
Use the ldm create-vf or ldm set-io command to set the read-write properties for a physical function or a virtual function. Note that to set a device-specific property, you must initiate a delayed reconfiguration.
The following example shows the device-specific properties that are exported by the on-board Intel 1-Gbps SR-IOV device:
primary# ldm list-io -d /SYS/MB/NET0/IOVNET.PF0 Device-specific Parameters -------------------------- max-config-vfs Flags = PR Default = 7 Descr = Max number of configurable VFs max-vf-mtu Flags = VR Default = 9216 Descr = Max MTU supported for a VF max-vlans Flags = VR Default = 32 Descr = Max number of VLAN filters supported pvid-exclusive Flags = VR Default = 1 Descr = Exclusive configuration of pvid required unicast-slots Flags = PV Default = 0 Min = 0 Max = 24 Descr = Number of unicast mac-address slots
The following example sets the unicast-slots property to 8:
primary# ldm create-vf unicast-slots=8 /SYS/MB/NET0/IOVNET.PF0
SR-IOV virtual functions can only use the MAC addresses that are assigned by the Logical Domains Manager. If you use other Oracle Solaris OS networking commands to change the MAC address on the I/O domain, the commands might fail or might not function properly.
At this time, link aggregation of SR-IOV network virtual functions in the I/O domain is not supported. If you attempt to create a link aggregation, it might not function as expected.
You can create virtual I/O services and assign them to I/O domains. These virtual I/O services can be created on the same physical function from which virtual functions are also created. For example, you can use an on-board 1-Gbps network device (net0 or igb0) as a network back-end device for a virtual switch and also create virtual functions from the same physical function device.
The creation of Oracle Solaris 11 VNICs is supported on SR-IOV virtual functions. However, the number of VNICs that is supported is limited to the number of alternate MAC addresses (alt-mac-addrs property) assigned to the virtual function. So, ensure that you assign a sufficient number of alternate MAC addresses when you use VNICs on the virtual function. Use the ldm create-vf or ldm set-io command to set the alt-mac-addrs property with the alternate MAC addresses.
The following example shows the creation of four VNICs on an SR-IOV virtual function. The first command assigns alternate MAC addresses to the virtual function device. This command uses the automatic allocation method to allocate four alternate MAC addresses to the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function device:
primary# ldm set-io alt-mac-addrs=auto,auto,auto,auto /SYS/MB/NET0/IOVNET.PF0.VF0
The next command starts and boots the Oracle Solaris 11 OS in the I/O domain. In this example, ldg1 is the I/O domain:
primary# ldm start ldg1
The following command uses the Oracle Solaris 11 dladm command in the In the guest domain to create four VNICs. Note that attempts to create more VNICs than are specified by using alternate MAC addresses will fail.
guest# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net0 Ethernet up 0 unknown vnet0 net1 Ethernet up 1000 full igbvf0 guest# dladm create-vnic -l net1 vnic0 guest# dladm create-vnic -l net1 vnic1 guest# dladm create-vnic -l net1 vnic2 guest# dladm create-vnic -l net1 vnic3 guest# dladm show-link LINK CLASS MTU STATE OVER net0 phys 1500 up -- net1 phys 1500 up -- vnic0 vnic 1500 up net1 vnic1 vnic 1500 up net1 vnic2 vnic 1500 up net1 vnic3 vnic 1500 up net1