Part I Oracle VM Server for SPARC 3.1 Software
Chapter 1 Overview of the Oracle VM Server for SPARC Software
About Oracle VM Server for SPARC and Oracle Solaris OS Versions
Hypervisor and Logical Domains
Oracle VM Server for SPARC Physical-to-Virtual Conversion Tool
Oracle VM Server for SPARC Configuration Assistant
Oracle VM Server for SPARC Management Information Base
Chapter 2 Installing and Enabling Software
Required Oracle VM Server for SPARC Software Components
Installing Oracle VM Server for SPARC Software on a New System
Updating the Oracle Solaris OS
Downloading the Logical Domains Manager
How to Download the Logical Domains Manager Software (Oracle Solaris 10)
Installing the Logical Domains Manager
Automatically Installing the Logical Domains Manager Software (Oracle Solaris 10)
Manually Installing the Logical Domains Manager Software
How to Manually Install the Oracle VM Server for SPARC 3.1 Software (Oracle Solaris 10)
Enabling the Logical Domains Manager Daemon
How to Enable the Logical Domains Manager Daemon
Upgrading a System That Is Already Using Oracle VM Server for SPARC
Upgrading the Oracle Solaris OS
Saving and Restoring Autosave Configuration Directories
How to Save and Restore Autosave Directories
Saving and Restoring the Logical Domains Constraints Database File
Upgrading the Logical Domains Manager and the System Firmware
How to Stop All Domains Running on the Platform, Except the Control Domain
Upgrading to Oracle VM Server for SPARC 3.1 Software
How to Upgrade to the Oracle VM Server for SPARC 3.1 Software (Oracle Solaris 10)
How to Upgrade to the Oracle VM Server for SPARC 3.1 Software (Oracle Solaris 11)
Factory Default Configuration and Disabling Domains
How to Remove All Guest Domains
How to Remove All Domain Configurations
How to Restore the Factory Default Configuration
How to Disable the Logical Domains Manager
How to Remove the Logical Domains Manager
How to Restore the Factory Default Configuration From the Service Processor
Chapter 3 Oracle VM Server for SPARC Security
Delegating the Management of Logical Domains by Using Rights
Using Rights Profiles and Roles
How to Assign a Rights Profile to a User
How to Create a Role and Assign the Role to a User
Logical Domains Manager Rights Profile Contents
Controlling Access to a Domain Console by Using Rights
How to Control Access to All Domain Consoles by Using Roles
How to Control Access to All Domain Consoles by Using Rights Profiles
How to Control Access to a Single Console by Using Roles
How to Control Access to a Single Console by Using Rights Profiles
How to Enable or Disable Console Logging
Service Domain Requirements for Domain Console Logging
Chapter 4 Setting Up Services and the Control Domain
How to Create Default Services
Initial Configuration of the Control Domain
How to Configure the Control Domain
Enabling Networking Between the Control/Service Domain and Other Domains
How to Configure the Virtual Switch as the Primary Interface
Enabling the Virtual Network Terminal Server Daemon
How to Enable the Virtual Network Terminal Server Daemon
Chapter 5 Setting Up Guest Domains
Creating and Starting a Guest Domain
How to Create and Start a Guest Domain
Installing Oracle Solaris OS on a Guest Domain
How to Install the Oracle Solaris OS on a Guest Domain From a DVD
How to Install the Oracle Solaris OS on a Guest Domain From an Oracle Solaris ISO File
How to Use the Oracle Solaris JumpStart Feature on an Oracle Solaris 10 Guest Domain
Chapter 6 Setting Up I/O Domains
General Guidelines for Creating an I/O Domain
Creating a Root Domain by Assigning PCIe Buses
How to Create an I/O Domain by Assigning a PCIe Bus
Creating an I/O Domain by Assigning PCIe Endpoint Devices
Direct I/O Hardware and Software Requirements
Current Direct I/O Feature Limitations
Planning PCIe Endpoint Device Configuration
How to Create an I/O Domain by Assigning a PCIe Endpoint Device
Creating an I/O Domain by Assigning PCIe SR-IOV Virtual Functions
SR-IOV Hardware and Software Requirements
Current SR-IOV Feature Limitations
Static SR-IOV Software Requirements
Dynamic SR-IOV Software Requirements
Dynamic SR-IOV Configuration Requirements
How to Enable I/O Virtualization for a PCIe Bus
Planning for the Use of PCIe SR-IOV Virtual Functions
Using Ethernet SR-IOV Virtual Functions
Ethernet SR-IOV Hardware Requirements
Planning for the Use of Ethernet SR-IOV Virtual Functions
Ethernet Device-Specific and Network-Specific Properties
Creating Ethernet Virtual Functions
How to Create an Ethernet SR-IOV Virtual Function
Destroying Ethernet Virtual Functions
How to Destroy an Ethernet SR-IOV Virtual Function
Modifying Ethernet SR-IOV Virtual Functions
How to Modify an Ethernet SR-IOV Virtual Function
Adding and Removing Ethernet SR-IOV Virtual Functions on I/O Domains
How to Add an Ethernet SR-IOV Virtual Function to an I/O Domain
How to Remove an Ethernet Virtual SR-IOV Function From an I/O Domain
Advanced SR-IOV Topics: Ethernet SR-IOV
Advanced Network Configuration for Virtual Functions
Booting an I/O Domain by Using an SR-IOV Virtual Function
SR-IOV Device-Specific Properties
Creating VNICs on SR-IOV Virtual Functions
Using an SR-IOV Virtual Function to Create an I/O Domain
How to Create an I/O Domain by Assigning an SR-IOV Virtual Function to It
Using InfiniBand SR-IOV Virtual Functions
InfiniBand SR-IOV Hardware Requirements
Creating and Destroying InfiniBand Virtual Functions
How to Create an InfiniBand Virtual Function
How to Destroy an InfiniBand Virtual Function
Adding and Removing InfiniBand Virtual Functions on I/O Domains
How to Add an InfiniBand Virtual Function to an I/O Domain
How to Remove an InfiniBand Virtual Function From an I/O Domain
Adding and Removing InfiniBand Virtual Functions to Root Domains
How to Add an InfiniBand Virtual Function to a Root Domain
How to Remove an InfiniBand Virtual Function From a Root Domain
Advanced SR-IOV Topics: InfiniBand SR-IOV
Listing InfiniBand SR-IOV Virtual Functions
Identifying InfiniBand SR-IOV Functions
Using Fibre Channel SR-IOV Virtual Functions
Fibre Channel SR-IOV Hardware Requirements
Fibre Channel SR-IOV Requirements and Limitations
Fibre Channel Device Class-Specific Properties
World-Wide Name Allocation for Fibre Channel Virtual Functions
Creating Fibre Channel SR-IOV Virtual Functions
How to Create a Fibre Channel SR-IOV Virtual Function
Destroying Fibre Channel SR-IOV Virtual Functions
How to Destroy a Fibre Channel SR-IOV Virtual Function
Modifying Fibre Channel SR-IOV Virtual Functions
How to Modify a Fibre Channel SR-IOV Virtual Function
Adding and Removing Fibre Channel SR-IOV Virtual Functions on I/O Domains
How to Add a Fibre Channel SR-IOV Virtual Function to an I/O Domain
How to Remove a Fibre Channel SR-IOV Virtual Function From an I/O Domain
Advanced SR-IOV Topics: Fibre Channel SR-IOV
Accessing a Fibre Channel Virtual Function in a Guest Domain
SR-IOV: Rebooting the Root Domain
Using Non-primary Root Domains
Non-primary Root Domain Requirements
Non-primary Root Domain Limitations
Enabling I/O Virtualization for a PCIe Bus
Managing Direct I/O Devices on Non-primary Root Domains
Managing SR-IOV Virtual Functions on Non-primary Root Domains
Virtual Disk Identifier and Device Name
How to Export a Virtual Disk Back End Multiple Times
How to Change Virtual Disk Options
How to Change the Timeout Option
How to Export a Physical Disk as a Virtual Disk
How to Export a Physical Disk Slice as a Virtual Disk
File or Volume Exported as a Full Disk
How to Export a File as a Full Disk
How to Export a ZFS Volume as a Full Disk
File or Volume Exported as a Single-Slice Disk
How to Export a ZFS Volume as a Single-Slice Disk
Exporting Volumes and Backward Compatibility
Summary of How Different Types of Back Ends Are Exported
Guidelines for Exporting Files and Disk Slices as Virtual Disks
Using the Loopback File (lofi) Driver
Directly or Indirectly Exporting a Disk Slice
Configuring Virtual Disk Multipathing
Virtual Disk Multipathing and Virtual Disk Timeout
How to Configure Virtual Disk Multipathing
How to Export a CD or DVD From the Service Domain to the Guest Domain
How to Export an ISO Image From the Control Domain to Install a Guest Domain
Virtual Disk and the format Command
Configuring a ZFS Pool in a Service Domain
Examples of Storing Disk Images With ZFS
Creating a Snapshot of a Disk Image
Using Clone to Provision a New Domain
How to Create a Snapshot of a Disk Image of an Unconfigured System
Using Volume Managers in an Oracle VM Server for SPARC Environment
Using Virtual Disks With Volume Managers
Using Virtual Disks With Solaris Volume Manager
Using Virtual Disks When VxVM Is Installed
Using Volume Managers With Virtual Disks
Using Solaris Volume Manager With Virtual Disks
Chapter 8 Using Virtual Networks
Introduction to a Virtual Network
Oracle Solaris 10 Networking Overview
Oracle Solaris 11 Networking Overview
Maximizing Virtual Network Performance
Hardware and Software Requirements
Configuring Your Domains to Maximize the Performance of Your Virtual Network
Controlling the Amount of Physical Network Bandwidth That Is Consumed by a Virtual Network Device
Setting the Network Bandwidth Limit
Virtual Device Identifier and Network Interface Name
How to Find Oracle Solaris OS Network Interface Name
Assigning MAC Addresses Automatically or Manually
Range of MAC Addresses Assigned to Domains
Automatic Assignment Algorithm
Duplicate MAC Address Detection
Using Network Adapters With Domains
How to Determine Whether a Network Adapter Is GLDv3-Compliant (Oracle Solaris 10)
Configuring a Virtual Switch and the Service Domain for NAT and Routing
Configuring NAT on an Oracle Solaris 10 System
How to Set Up a Virtual Switch to Provide External Connectivity to Domains (Oracle Solaris 10)
Configuring NAT on an Oracle Solaris 11 System
How to Set Up a Virtual Switch to Provide External Connectivity to Domains (Oracle Solaris 11)
Configuring IPMP in an Oracle VM Server for SPARC Environment
Configuring Virtual Network Devices Into an IPMP Group in a Domain
Configuring and Using IPMP in the Service Domain
Using Link-Based IPMP in Oracle VM Server for SPARC Virtual Networking
How to Configure Physical Link Status Updates
Configuring and Using IPMP in Releases Prior to Logical Domains 1.3
Configuring IPMP in the Guest Domain
Configuring IPMP in the Service Domain
How to Configure a Host Route for Probe-Based IPMP
How to Assign VLANs to a Virtual Switch and Virtual Network Device
How to Install a Guest Domain When the Install Server Is in a VLAN
PVLAN Configuration Information
How to Configure a Virtual Switch With an NIU Network Device
How to Enable or Disable Hybrid Mode
Using Link Aggregation With a Virtual Switch
How to Configure Virtual Network and Virtual Switch Devices to Use Jumbo Frames
Compatibility With Older (Jumbo-Unaware) Versions of the vnet and vsw Drivers (Oracle Solaris 10)
Oracle Solaris 11 Networking-Specific Feature Differences
Introduction to Domain Migration
Overview of a Migration Operation
Security for Migration Operations
Performing Non-Interactive Migrations
Domain Migration Requirements for CPUs
Migration Requirements for Memory
Migration Requirements for Physical I/O Devices
Migration Requirements for Virtual I/O Devices
Migration Requirements for PCIe Endpoint Devices
Migration Requirements for PCIe SR-IOV Virtual Functions
Migration Requirements for NIU Hybrid I/O
Migration Requirements for Cryptographic Units
Delayed Reconfiguration in an Active Domain
Migrating While an Active Domain Has the Power Management Elastic Policy in Effect
Migrating a Domain From the OpenBoot PROM or a Domain That Is Running in the Kernel Debugger
Migrating Bound or Inactive Domains
Migration Requirements for Virtual I/O Devices
Migration Requirements for PCIe Endpoint Devices
Migration Requirements for PCIe SR-IOV Virtual Functions
Monitoring a Migration in Progress
Canceling a Migration in Progress
Recovering From a Failed Migration
How to Apply the Whole-Core Constraint
How to Apply the Max-Cores Constraint
Interactions Between the Whole-Core Constraint and Other Domain Features
Configuring the System With Hard Partitions
Checking the Configuration of a Domain
Configuring a Domain With CPU Whole Cores
How to Create a New Domain With CPU Whole Cores
How to Configure an Existing Domain With CPU Whole Cores
How to Configure the Primary Domain With CPU Whole Cores
Interaction of Hard Partitioned Systems With Other Oracle VM Server for SPARC Features
CPU Dynamic Resource Management
Assigning Physical Resources to Domains
How to Remove the physical-bindings Constraint
How to Remove All Non-Physically Bound Resources
Managing Physical Resources on the Control Domain
Restrictions for Managing Physical Resources on Domains
Using Memory Dynamic Reconfiguration
Memory Reconfiguration of the Control Domain
Decrease the Control Domain's Memory
Dynamic and Delayed Reconfiguration
Memory Alignment for Active Domains
Memory Alignment for Bound Domains
Memory Alignment for Inactive Domains
Using Dynamic Resource Management
Utilization Statistic Definition
Chapter 11 Managing Domain Configurations
Managing Domain Configurations
Available Configuration Recovery Methods
Restoring Configurations By Using Autosave
How to Modify the Autorecovery Policy
Restoring Domain Configurations
How to Restore a Domain Configuration From an XML File (ldm add-domain)
How to Restore a Domain Configuration From an XML File (ldm init-system)
Chapter 12 Handling Hardware Errors
Hardware Error-Handling Overview
Using FMA to Blacklist or Unconfigure Faulty Resources
Recovering Domains After Detecting Faulty or Missing Resources
Marking I/O Resources as Evacuated
Chapter 13 Performing Other Administration Tasks
Connecting to a Guest Console Over the Network
How to Combine Multiple Consoles Into One Group
Stopping a Heavily Loaded Domain Can Time Out
Operating the Oracle Solaris OS With Oracle VM Server for SPARC
OpenBoot Firmware Not Available After the Oracle Solaris OS Has Started
Performing a Power Cycle of a Server
Result of Oracle Solaris OS Breaks
Results From Halting or Rebooting the Control Domain
Using Oracle VM Server for SPARC With the Service Processor
Configuring Domain Dependencies
Determining Where Errors Occur by Mapping CPU and Memory Addresses
Example of CPU and Memory Mapping
Using Universally Unique Identifiers
Virtual Domain Information Command and API
You can use both the static and dynamic SR-IOV methods to manage Ethernet SR-IOV devices.
For information about the required PCIe Ethernet SR-IOV hardware, see PCIe SR-IOV Hardware and Software Requirements in Oracle VM Server for SPARC 3.1.1.1, 3.1.1, and 3.1 Release Notes .
You can enable VLAN configurations of virtual functions by setting either the pvid or the vid property. You cannot set both of these virtual function properties simultaneously.
When dynamically creating virtual functions, ensure that the physical functions use multipathing or that they are not plumbed.
If you cannot use multipathing or must plumb the physical function, use the static method to create the virtual functions. See Static SR-IOV.
Use the ldm create-vf command to set device-specific and network-specific properties of a virtual function. The unicast-slots property is device-specific. The mac-addr, alt-mac-addrs, mtu, pvid, and vid properties are network-specific.
Note that the mac-addr, alt-mac-addrs, and mtu network-specific properties can be changed only when the virtual function is assigned to the primary domain while in a delayed reconfiguration.
Attempts to change these properties fail when the virtual function is assigned as follows:
When the virtual function is assigned to an active I/O domain: A property change request is rejected because the change must be made when the owning domain is in the inactive or bound state.
When the virtual function is assigned to a non-primary domain and a delayed reconfiguration is already in effect: A property change request fails with an error message.
The pvid and vid network-specific properties can be changed without restriction.
This section describes how to dynamically create and destroy virtual functions. If you cannot use the dynamic methods to perform these actions, initiate a delayed reconfiguration on the root domain before you create or destroy virtual functions.
If you cannot use this dynamic method, use the static method instead. See Static SR-IOV.
primary# ldm list-io
Note that the name of the physical function includes the location information for the PCIe SR-IOV card or on-board device.
Perform this step only if I/O virtualization is not enabled already for the bus that has the physical function.
After you create one or more virtual functions, you can assign them to a guest domain.
primary# ldm create-vf -n number | max pf-name
Use the ldm create-vf -n max command to create all the virtual functions for that physical function at one time.
Caution - When your system uses an Intel 10-Gbit Ethernet card, maximize performance by creating no more than 31 virtual functions from each physical function. |
You can use either the path name or the pseudonym name to specify virtual functions. However, the recommended practice is to use the pseudonym name.
primary# ldm create-vf [mac-addr=num] [alt-mac-addrs=[auto|num1,[auto|num2,...]]] [pvid=pvid] [vid=vid1,vid2,...] [mtu=size] [name=value...] pf-name
Use this command to create one virtual function for that physical function. You can also manually specify Fibre Channel class-specific property values.
primary# ldm start-reconf root-domain-name
Use the same commands as shown previously to dynamically create the virtual functions.
primary# ldm stop-domain -r root-domain
primary# shutdown -i6 -g0 -y
This example shows information about the /SYS/MB/NET0/IOVNET.PF0 physical function:
This physical function is from an on-board NET0 network device.
The IOVNET string indicates that the physical function is a network SR-IOV device.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary pci_0 BUS pci_0 primary pci_1 BUS pci_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 primary OCC /SYS/MB/PCIE3 PCIE pci_1 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary OCC /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE9 PCIE pci_1 primary EMP /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary
The following command shows more details about the specified physical function. The maxvfs value indicates the maximum number of virtual functions that is supported by the device.
primary# ldm list-io -l /SYS/MB/NET0/IOVNET.PF0 NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary [pci@400/pci@1/pci@0/pci@4/network@0] maxvfs = 7Example 6-3 Dynamically Creating an Ethernet Virtual Function Without Setting Optional Properties
This example dynamically creates a virtual function without setting any optional properties. In this case, the MAC address for a network class virtual function is automatically allocated.
Ensure that I/O virtualization is enabled on the pci_0 PCIe bus. See How to Enable I/O Virtualization for a PCIe Bus.
Now, you can use the ldm create-vf command to create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.
primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0 Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0Example 6-4 Dynamically Creating an Ethernet Virtual Function and Setting Properties
This example dynamically creates a virtual function while setting the mac-addr property to 00:14:2f:f9:14:c0 and the vid property to VLAN IDs 2 and 3.
primary# ldm create-vf mac-addr=00:14:2f:f9:14:c0 vid=2,3 /SYS/MB/NET0/IOVNET.PF0Example 6-5 Dynamically Creating an Ethernet Virtual Function With Two Alternate MAC Addresses
This example dynamically creates a virtual function that has two alternate MAC addresses. One MAC address is automatically allocated, and the other is explicitly specified as 00:14:2f:f9:14:c2.
primary# ldm create-vf alt-mac-addrs=auto,00:14:2f:f9:14:c2 /SYS/MB/NET0/IOVNET.PF0Example 6-6 Statically Creating a Virtual Function Without Setting Optional Properties
This example statically creates a virtual function without setting any optional properties. In this case, the MAC address for a network class virtual function is automatically allocated.
First you initiate a delayed reconfiguration on the primary domain and then enable I/O virtualization on the pci_0 PCIe bus. Because the pci_0 bus has already been assigned to the primary root domain, use the ldm set-io command to enable I/O virtualization.
primary# ldm start-reconf primary Initiating a delayed reconfiguration operation on the primary domain. All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. primary# ldm set-io iov=on pci_0
Now, you can use the ldm create-vf command to create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.
primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0 ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0
Finally, reboot the primary root domain to make the changes take effect.
primary# shutdown -i6 -g0 -yExample 6-7 Creating Multiple SR-IOV Ethernet Virtual Functions
The following command shows how you can create four virtual functions from the /SYS/MB/NET2/IOVNET.PF1 physical function:
primary# ldm create-vf -n 31 /SYS/MB/NET2/IOVNET.PF1 Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF0 Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF1 Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF2 ... Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF30
Note that the ldm create-vf -n command creates multiple virtual functions that are set with default property values, if appropriate. You can later specify non-default property values by using the ldm set-io command.
A virtual function can be destroyed if it is not currently assigned to a domain. A virtual function can be destroyed only in the reverse sequential order of creation, so only the last virtual function that was created can be destroyed. The resulting configuration is validated by the physical function driver.
If you cannot use this dynamic method, use the static method instead. See Static SR-IOV.
primary# ldm list-io
primary# ldm destroy-vf -n number | max pf-name
Use the ldm destroy-vf -n max command to destroy all the virtual functions for that physical function at one time.
If you specify number as an argument to the –n option, the last number of virtual functions are destroyed. Use this method as it performs this operation with only one physical function device driver state transition.
You can use either the path name or the pseudonym name to specify virtual functions. However, the recommended practice is to use the pseudonym name.
primary# ldm destroy-vf vf-name
primary# ldm start-reconf root-domain-name
primary# ldm destroy-vf -n number | max pf-name
You can use either the path name or the pseudonym name to specify virtual functions. However, the recommended practice is to use the pseudonym name.
primary# ldm destroy-vf vf-name
primary# ldm stop-domain -r root-domain
primary# shutdown -i6 -g0 -y
This example shows how to dynamically destroy the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function.
primary# ldm destroy-vf /SYS/MB/NET0/IOVNET.PF0.VF0
The following example shows how to statically destroy the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function.
primary# ldm start-reconf primary Initiating a delayed reconfiguration operation on the primary domain. All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. primary# ldm destroy-vf /SYS/MB/NET0/IOVNET.PF0.VF0 primary# shutdown -i6 -g0 -yExample 6-9 Destroying Multiple Ethernet SR-IOV Virtual Functions
This example shows the results of destroying all the virtual functions from the /SYS/MB/NET2/IOVNET.PF1 physical function. The ldm list-io output shows that the physical function has seven virtual functions. The ldm destroy-vf command destroys all virtual functions, and the final ldm list-io output shows that none of the virtual functions remain.
primary# ldm list-io ... /SYS/MB/NET2/IOVNET.PF1 PF pci_1 /SYS/MB/NET2/IOVNET.PF1.VF0 VF pci_1 /SYS/MB/NET2/IOVNET.PF1.VF1 VF pci_1 /SYS/MB/NET2/IOVNET.PF1.VF2 VF pci_1 /SYS/MB/NET2/IOVNET.PF1.VF3 VF pci_1 /SYS/MB/NET2/IOVNET.PF1.VF4 VF pci_1 /SYS/MB/NET2/IOVNET.PF1.VF5 VF pci_1 /SYS/MB/NET2/IOVNET.PF1.VF6 VF pci_1 primary# ldm destroy-vf -n max /SYS/MB/NET2/IOVNET.PF1 primary# ldm list-io ... /SYS/MB/NET2/IOVNET.PF1 PF pci_1 ldg1
The ldm set-io vf-name command modifies the current configuration of a virtual function by changing the property values or by setting new properties. This command can modify both the network-specific properties and the device-specific properties. For information about device-specific properties, see Advanced SR-IOV Topics: Ethernet SR-IOV.
If you cannot use this dynamic method, use the static method instead. See Static SR-IOV.
You can use the ldm set-io command to modify the following properties:
mac-addr, alt-mac-addrs, and mtu
To change these virtual function properties, stop the domain that owns the virtual function, use the ldm set-io command to change the property values, and start the domain.
pvid and vid
You can dynamically change these properties while the virtual functions are assigned to a domain. Note that doing so might result in a change to the network traffic of an active virtual function; setting the pvid property enables a transparent VLAN. Setting the vid property to specify VLAN IDs permits VLAN traffic to those specified VLANs.
Device-specific properties
Use the ldm list-io -d pf-name command to view the list of valid device-specific properties. You can modify these properties for both the physical function and the virtual function. You must use the static method to modify device-specific properties. See Static SR-IOV. For more information about device-specific properties, see Advanced SR-IOV Topics: Ethernet SR-IOV.
primary# ldm list-io
Note that the name of the physical function includes the location information for the PCIe SR-IOV card or on-board device.
primary# ldm set-io name=value [name=value...] vf-name
These examples describe how to use the ldm set-io command to set properties on a virtual function.
The following example modifies the specified virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, to be part of VLAN IDs 2, 3, and 4.
primary# ldm set-io vid=2,3,4 /SYS/MB/NET0/IOVNET.PF0.VF0
Note that this command dynamically changes the VLAN association for a virtual function. To use these VLANs, the VLAN interfaces in the I/O domains must be configured by using the appropriate Oracle Solaris OS networking commands.
The following example sets the pvid property value to 2 for the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function, which transparently makes the virtual function part of VLAN 2. Namely, the virtual function will not view any tagged VLAN traffic.
primary# ldm set-io pvid=2 /SYS/MB/NET0/IOVNET.PF0.VF0
The following example assigns three automatically allocated alternate MAC addresses to a virtual function. The alternate addresses enable the creation of Oracle Solaris 11 virtual network interface cards (VNICs) on top of a virtual function. Note that to use VNICs, you must run the Oracle Solaris 11 OS in the domain.
primary# ldm set-io alt-mac-addrs=auto,auto,auto /SYS/MB/NET0/IOVNET.PF0.VF0
The following example sets the device-specific unicast-slots property to 12 for the specified virtual function. To find the device-specific properties that are valid for a physical function, use the ldm list-io -d pf-name command.
primary# ldm set-io unicast-slots=12 /SYS/MB/NET0/IOVNET.PF0.VF0 All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect.
If you cannot dynamically remove the virtual function, use the static method. See Static SR-IOV.
primary# ldm list-io
primary# ldm add-io vf-name domain-name
vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the pseudonym name. domain-name specifies the name of the domain to which you add the virtual function.
The device path name for the virtual function in the domain is the path shown in the list-io -l output.
primary# ldm start-reconf root-domain-name primary# ldm add-io vf-name domain-name
vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the pseudonym name. domain-name specifies the name of the domain to which you add the virtual function. The specified guest must be in the inactive or bound state.
The device path name for the virtual function in the domain is the path shown in the list-io -l output.
primary# ldm stop-domain -r root-domain
primary# shutdown -i6 -g0 -y
This example shows how to dynamically add the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function to the ldg1 domain.
primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
If you cannot add the virtual function dynamically, use the static method:
primary# ldm stop-domain ldg1 primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1 primary# ldm start-domain ldg1
If you cannot dynamically remove the virtual function, use the static method. See Static SR-IOV.
Caution - Before removing the virtual function from the domain, ensure that it is not critical for booting that domain. |
primary# ldm list-io
primary# ldm rm-io vf-name domain-name
vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the device pseudonym. domain-name specifies the name of the domain from which you remove the virtual function.
primary# ldm stop-domain domain-name
primary# ldm rm-io vf-name domain-name
vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the device pseudonym. domain-name specifies the name of the domain from which you remove the virtual function. The specified guest must be in the inactive or bound state.
primary# ldm start-domain domain-name
This example shows how to dynamically remove the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function from the ldg1 domain.
primary# ldm remove-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
If the command succeeds, the virtual function is removed from the ldg1 domain. When ldg1 is restarted, the specified virtual function no longer appears in that domain.
If you cannot remove the virtual function dynamically, use the static method:
primary# ldm stop-domain ldg1 primary# ldm remove-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1 primary# ldm start-domain ldg1
This section describes some advanced topics related to using SR-IOV virtual functions.
When you use SR-IOV virtual functions, note the following issues:
SR-IOV virtual functions can only use the MAC addresses that are assigned by the Logical Domains Manager. If you use other Oracle Solaris OS networking commands to change the MAC address on the I/O domain, the commands might fail or might not function properly.
At this time, link aggregation of SR-IOV network virtual functions in the I/O domain is not supported. If you attempt to create a link aggregation, it might not function as expected.
You can create virtual I/O services and assign them to I/O domains. These virtual I/O services can be created on the same physical function from which virtual functions are also created. For example, you can use an on-board 1-Gbps network device (net0 or igb0) as a network back-end device for a virtual switch and also create virtual functions from the same physical function device.
An SR-IOV virtual function provides similar capabilities to any other type of PCIe device, such as the ability to use a virtual function as a logical domain boot device. For example, a network virtual function can be used to boot over the network to install the Oracle Solaris OS in an I/O domain.
SR-IOV physical function device drivers can export device-specific properties. These properties can be used to tune the resource allocation of both the physical function and its virtual functions. For information about the properties, see the man page for the physical function driver, such as the igb(7D) and ixgbe(7D) man pages.
The ldm list-io -d command shows device-specific properties that are exported by the specified physical function device driver. The information for each property includes its name, brief description, default value, maximum values, and one or more of the following flags:
Applies to a physical function
Applies to a virtual function
Read-only or informative parameter only
primary# ldm list-io -d pf-name
Use the ldm create-vf or ldm set-io command to set the read-write properties for a physical function or a virtual function. Note that to set a device-specific property, you must use the static method. See Static SR-IOV.
The following example shows the device-specific properties that are exported by the on-board Intel 1-Gbps SR-IOV device:
primary# ldm list-io -d /SYS/MB/NET0/IOVNET.PF0 Device-specific Parameters -------------------------- max-config-vfs Flags = PR Default = 7 Descr = Max number of configurable VFs max-vf-mtu Flags = VR Default = 9216 Descr = Max MTU supported for a VF max-vlans Flags = VR Default = 32 Descr = Max number of VLAN filters supported pvid-exclusive Flags = VR Default = 1 Descr = Exclusive configuration of pvid required unicast-slots Flags = PV Default = 0 Min = 0 Max = 24 Descr = Number of unicast mac-address slots
The following example sets the unicast-slots property to 8:
primary# ldm create-vf unicast-slots=8 /SYS/MB/NET0/IOVNET.PF0
The creation of Oracle Solaris 11 VNICs is supported on SR-IOV virtual functions. However, the number of VNICs that is supported is limited to the number of alternate MAC addresses (alt-mac-addrs property) assigned to the virtual function. Make sure that you assign a sufficient number of alternate MAC addresses when you use VNICs on the virtual function. Use the ldm create-vf or ldm set-io command to set the alt-mac-addrs property with the alternate MAC addresses.
The following example shows the creation of four VNICs on an SR-IOV virtual function. The first command assigns alternate MAC addresses to the virtual function device. This command uses the automatic allocation method to allocate four alternate MAC addresses to the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function device:
primary# ldm set-io alt-mac-addrs=auto,auto,auto,auto /SYS/MB/NET0/IOVNET.PF0.VF0
The next command starts the ldg1 I/O domain. Because the auto-boot? property is set to true in this example, the Oracle Solaris 11 OS is also booted in the I/O domain.
primary# ldm start ldg1
The following command uses the Oracle Solaris 11 dladm command in the guest domain to show virtual function that has alternate MAC addresses. This output shows that the net30 virtual function has four alternate MAC addresses.
guest# dladm show-phys -m LINK SLOT ADDRESS INUSE CLIENT net0 primary 0:14:4f:fa:b4:d1 yes net0 net25 primary 0:14:4f:fa:c9:eb no -- net30 primary 0:14:4f:fb:de:4c no -- 1 0:14:4f:f9:e8:73 no -- 2 0:14:4f:f8:21:58 no -- 3 0:14:4f:fa:9d:92 no -- 4 0:14:4f:f9:8f:1d no --
The following commands create four VNICs. Note that attempts to create more VNICs than are specified by using alternate MAC addresses will fail.
guest# dladm create-vnic -l net30 vnic0 guest# dladm create-vnic -l net30 vnic1 guest# dladm create-vnic -l net30 vnic2 guest# dladm create-vnic -l net30 vnic3 guest# dladm show-link LINK CLASS MTU STATE OVER net0 phys 1500 up -- net25 phys 1500 up -- net30 phys 1500 up -- vnic0 vnic 1500 up net30 vnic1 vnic 1500 up net30 vnic2 vnic 1500 up net30 vnic3 vnic 1500 up net30
The following procedure explains how to create an I/O domain that includes PCIe SR-IOV virtual functions.
Plan ahead to minimize the number of reboots of the root domain, which minimizes downtime.
Before You Begin
Before you begin, ensure that you have enabled I/O virtualization for the PCIe bus that is the parent of the physical function from which you create virtual functions. See How to Enable I/O Virtualization for a PCIe Bus.
primary# ldm list-io
primary# ldm create-vf pf-name
You can run this command for each virtual function that you want to create. You can also use the –n option to create more than one virtual function from the same physical function in a single command. See Example 6–7 and the ldm(1M) man page.
primary# ldm list-io
primary# ldm add-io vf-name domain-name
The following Oracle Solaris 11 command shows the availability of the virtual function:
guest# dladm show-phys
The following dynamic example shows how to create a virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, for a physical function, /SYS/MB/NET0/IOVNET.PF0, and assign the virtual function to the ldg1 I/O domain.
This example assumes that the following circumstances are true:
The OS on the primary domain supports dynamic SR-IOV operations
The pci_0 bus is assigned to the primary domain and has been initialized for I/O virtualization operations
The /SYS/MB/NET0/IOVNET.PF0 physical function belongs to the pci_0 bus
The /SYS/MB/NET0/IOVNET.PF0 physical function does not have any existing virtual functions assigned to domains
The ldg1 domain is active and booted and its OS supports dynamic SR-IOV operations
Create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.
primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0 Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0
Add the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function to the ldg1 domain.
primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
The following command shows that the virtual function has been added to the ldg1 domain.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary pci_0 BUS pci_0 primary IOV pci_1 BUS pci_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 primary OCC /SYS/MB/PCIE3 PCIE pci_1 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary OCC /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE9 PCIE pci_1 primary EMP /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary /SYS/MB/NET0/IOVNET.PF0.VF0 VF pci_0 ldg1Example 6-14 Statically Creating an I/O Domain by Assigning an SR-IOV Virtual Function to It
The following static example shows how to create a virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, for a physical function, /SYS/MB/NET0/IOVNET.PF0, and assign the virtual function to the ldg1 I/O domain.
This example assumes that the following circumstances are true:
The OS on the primary domain does not support dynamic SR-IOV operations
The pci_0 bus is assigned to the primary domain and has not been initialized for I/O virtualization operations
The /SYS/MB/NET0/IOVNET.PF0 physical function belongs to the pci_0 bus
The /SYS/MB/NET0/IOVNET.PF0 physical function does not have any existing virtual functions assigned to domains
The ldg1 domain is active and booted and its OS does not support dynamic SR-IOV operations
The ldg1 domain has the auto-boot? property set to true so that the domain boots automatically when the domain is started
First, initiate a delayed reconfiguration on the primary domain, enable I/O virtualization, and create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.
primary# ldm start-reconf primary Initiating a delayed reconfiguration operation on the primary domain. All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. primary# ldm set-io iov=on pci_0 primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0 ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0
Next, shut down the primary domain.
primary# shutdown -i6 -g0 -y
Stop the ldg1 domain, add the virtual function, and start the domain.
primary# ldm stop ldg1 primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1 primary# ldm start ldg1
The following command shows that the virtual function has been added to the ldg1 domain.
primary# ldm list-io NAME TYPE BUS DOMAIN STATUS ---- ---- --- ------ ------ niu_0 NIU niu_0 primary niu_1 NIU niu_1 primary pci_0 BUS pci_0 primary IOV pci_1 BUS pci_1 primary /SYS/MB/PCIE0 PCIE pci_0 primary OCC /SYS/MB/PCIE2 PCIE pci_0 primary OCC /SYS/MB/PCIE4 PCIE pci_0 primary OCC /SYS/MB/PCIE6 PCIE pci_0 primary EMP /SYS/MB/PCIE8 PCIE pci_0 primary EMP /SYS/MB/SASHBA PCIE pci_0 primary OCC /SYS/MB/NET0 PCIE pci_0 primary OCC /SYS/MB/PCIE1 PCIE pci_1 primary OCC /SYS/MB/PCIE3 PCIE pci_1 primary OCC /SYS/MB/PCIE5 PCIE pci_1 primary OCC /SYS/MB/PCIE7 PCIE pci_1 primary EMP /SYS/MB/PCIE9 PCIE pci_1 primary EMP /SYS/MB/NET2 PCIE pci_1 primary OCC /SYS/MB/NET0/IOVNET.PF0 PF pci_0 primary /SYS/MB/NET0/IOVNET.PF1 PF pci_0 primary /SYS/MB/PCIE5/IOVNET.PF0 PF pci_1 primary /SYS/MB/PCIE5/IOVNET.PF1 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF0 PF pci_1 primary /SYS/MB/NET2/IOVNET.PF1 PF pci_1 primary /SYS/MB/NET0/IOVNET.PF0.VF0 VF pci_0 ldg1