JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Oracle VM Server for SPARC 3.1 Administration Guide
Oracle Technology Network
Library
PDF
Print View
Feedback
search filter icon
search icon

Document Information

Using This Documentation

Part I Oracle VM Server for SPARC 3.1 Software

Chapter 1 Overview of the Oracle VM Server for SPARC Software

About Oracle VM Server for SPARC and Oracle Solaris OS Versions

Hypervisor and Logical Domains

Logical Domains Manager

Roles for Domains

Command-Line Interface

Virtual Input/Output

Virtual Network

Virtual Storage

Virtual Console

Resource Configuration

Persistent Configurations

Oracle VM Server for SPARC Physical-to-Virtual Conversion Tool

Oracle VM Server for SPARC Configuration Assistant

Oracle VM Server for SPARC Management Information Base

Chapter 2 Installing and Enabling Software

Required Oracle VM Server for SPARC Software Components

Installing Oracle VM Server for SPARC Software on a New System

Updating the Oracle Solaris OS

Upgrading the System Firmware

Downloading the Logical Domains Manager

How to Download the Logical Domains Manager Software (Oracle Solaris 10)

Installing the Logical Domains Manager

Automatically Installing the Logical Domains Manager Software (Oracle Solaris 10)

Manually Installing the Logical Domains Manager Software

How to Manually Install the Oracle VM Server for SPARC 3.1 Software (Oracle Solaris 10)

Enabling the Logical Domains Manager Daemon

How to Enable the Logical Domains Manager Daemon

Upgrading a System That Is Already Using Oracle VM Server for SPARC

Upgrading the Oracle Solaris OS

Saving and Restoring Autosave Configuration Directories

How to Save and Restore Autosave Directories

Saving and Restoring the Logical Domains Constraints Database File

Preserving the Logical Domains Constraints Database File When Using the Oracle Solaris 10 Live Upgrade Feature

Upgrading the Logical Domains Manager and the System Firmware

How to Stop All Domains Running on the Platform, Except the Control Domain

Upgrading to Oracle VM Server for SPARC 3.1 Software

How to Upgrade to the Oracle VM Server for SPARC 3.1 Software (Oracle Solaris 10)

How to Upgrade to the Oracle VM Server for SPARC 3.1 Software (Oracle Solaris 11)

Factory Default Configuration and Disabling Domains

How to Remove All Guest Domains

How to Remove All Domain Configurations

How to Restore the Factory Default Configuration

How to Disable the Logical Domains Manager

How to Remove the Logical Domains Manager

How to Restore the Factory Default Configuration From the Service Processor

Chapter 3 Oracle VM Server for SPARC Security

Delegating the Management of Logical Domains by Using Rights

Using Rights Profiles and Roles

Managing User Rights Profiles

How to Assign a Rights Profile to a User

Assigning Roles to Users

How to Create a Role and Assign the Role to a User

Logical Domains Manager Rights Profile Contents

Controlling Access to a Domain Console by Using Rights

How to Control Access to All Domain Consoles by Using Roles

How to Control Access to All Domain Consoles by Using Rights Profiles

How to Control Access to a Single Console by Using Roles

How to Control Access to a Single Console by Using Rights Profiles

Enabling and Using Auditing

How to Enable Auditing

How to Disable Auditing

How to Review Audit Records

How to Rotate Audit Logs

Using Domain Console Logging

How to Enable or Disable Console Logging

Service Domain Requirements for Domain Console Logging

Chapter 4 Setting Up Services and the Control Domain

Output Messages

Creating Default Services

How to Create Default Services

Initial Configuration of the Control Domain

How to Configure the Control Domain

Rebooting to Use Domains

How to Reboot

Enabling Networking Between the Control/Service Domain and Other Domains

How to Configure the Virtual Switch as the Primary Interface

Enabling the Virtual Network Terminal Server Daemon

How to Enable the Virtual Network Terminal Server Daemon

Chapter 5 Setting Up Guest Domains

Creating and Starting a Guest Domain

How to Create and Start a Guest Domain

Installing Oracle Solaris OS on a Guest Domain

How to Install the Oracle Solaris OS on a Guest Domain From a DVD

How to Install the Oracle Solaris OS on a Guest Domain From an Oracle Solaris ISO File

How to Use the Oracle Solaris JumpStart Feature on an Oracle Solaris 10 Guest Domain

Chapter 6 Setting Up I/O Domains

I/O Domain Overview

General Guidelines for Creating an I/O Domain

Creating a Root Domain by Assigning PCIe Buses

How to Create an I/O Domain by Assigning a PCIe Bus

Creating an I/O Domain by Assigning PCIe Endpoint Devices

Direct I/O Hardware and Software Requirements

Current Direct I/O Feature Limitations

Planning PCIe Endpoint Device Configuration

Rebooting the Root Domain

Making PCIe Hardware Changes

How to Create an I/O Domain by Assigning a PCIe Endpoint Device

Creating an I/O Domain by Assigning PCIe SR-IOV Virtual Functions

SR-IOV Overview

SR-IOV Hardware and Software Requirements

Current SR-IOV Feature Limitations

Static SR-IOV

Static SR-IOV Software Requirements

Dynamic SR-IOV

Dynamic SR-IOV Software Requirements

Dynamic SR-IOV Configuration Requirements

Enabling I/O Virtualization

How to Enable I/O Virtualization for a PCIe Bus

Planning for the Use of PCIe SR-IOV Virtual Functions

Using Ethernet SR-IOV Virtual Functions

Ethernet SR-IOV Hardware Requirements

Ethernet SR-IOV Limitations

Planning for the Use of Ethernet SR-IOV Virtual Functions

Ethernet Device-Specific and Network-Specific Properties

Creating Ethernet Virtual Functions

How to Create an Ethernet SR-IOV Virtual Function

Destroying Ethernet Virtual Functions

How to Destroy an Ethernet SR-IOV Virtual Function

Modifying Ethernet SR-IOV Virtual Functions

How to Modify an Ethernet SR-IOV Virtual Function

Adding and Removing Ethernet SR-IOV Virtual Functions on I/O Domains

How to Add an Ethernet SR-IOV Virtual Function to an I/O Domain

How to Remove an Ethernet Virtual SR-IOV Function From an I/O Domain

Advanced SR-IOV Topics: Ethernet SR-IOV

Advanced Network Configuration for Virtual Functions

Booting an I/O Domain by Using an SR-IOV Virtual Function

SR-IOV Device-Specific Properties

Creating VNICs on SR-IOV Virtual Functions

Using an SR-IOV Virtual Function to Create an I/O Domain

How to Create an I/O Domain by Assigning an SR-IOV Virtual Function to It

Using InfiniBand SR-IOV Virtual Functions

InfiniBand SR-IOV Hardware Requirements

Creating and Destroying InfiniBand Virtual Functions

How to Create an InfiniBand Virtual Function

How to Destroy an InfiniBand Virtual Function

Adding and Removing InfiniBand Virtual Functions on I/O Domains

How to Add an InfiniBand Virtual Function to an I/O Domain

How to Remove an InfiniBand Virtual Function From an I/O Domain

Adding and Removing InfiniBand Virtual Functions to Root Domains

How to Add an InfiniBand Virtual Function to a Root Domain

How to Remove an InfiniBand Virtual Function From a Root Domain

Advanced SR-IOV Topics: InfiniBand SR-IOV

Listing InfiniBand SR-IOV Virtual Functions

Identifying InfiniBand SR-IOV Functions

Using Fibre Channel SR-IOV Virtual Functions

Fibre Channel SR-IOV Hardware Requirements

Fibre Channel SR-IOV Requirements and Limitations

Fibre Channel Device Class-Specific Properties

World-Wide Name Allocation for Fibre Channel Virtual Functions

Creating Fibre Channel SR-IOV Virtual Functions

How to Create a Fibre Channel SR-IOV Virtual Function

Destroying Fibre Channel SR-IOV Virtual Functions

How to Destroy a Fibre Channel SR-IOV Virtual Function

Modifying Fibre Channel SR-IOV Virtual Functions

How to Modify a Fibre Channel SR-IOV Virtual Function

Adding and Removing Fibre Channel SR-IOV Virtual Functions on I/O Domains

How to Add a Fibre Channel SR-IOV Virtual Function to an I/O Domain

How to Remove a Fibre Channel SR-IOV Virtual Function From an I/O Domain

Advanced SR-IOV Topics: Fibre Channel SR-IOV

Accessing a Fibre Channel Virtual Function in a Guest Domain

SR-IOV: Rebooting the Root Domain

Using Non-primary Root Domains

Non-primary Root Domain Requirements

Non-primary Root Domain Limitations

Enabling I/O Virtualization for a PCIe Bus

Managing Direct I/O Devices on Non-primary Root Domains

Managing SR-IOV Virtual Functions on Non-primary Root Domains

Chapter 7 Using Virtual Disks

Introduction to Virtual Disks

Virtual Disk Identifier and Device Name

Managing Virtual Disks

How to Add a Virtual Disk

How to Export a Virtual Disk Back End Multiple Times

How to Change Virtual Disk Options

How to Change the Timeout Option

How to Remove a Virtual Disk

Virtual Disk Appearance

Full Disk

Single-Slice Disk

Virtual Disk Back End Options

Read-only (ro) Option

Exclusive (excl) Option

Slice (slice) Option

Virtual Disk Back End

Physical Disk or Disk LUN

How to Export a Physical Disk as a Virtual Disk

Physical Disk Slice

How to Export a Physical Disk Slice as a Virtual Disk

How to Export Slice 2

File and Volume Exporting

File or Volume Exported as a Full Disk

How to Export a File as a Full Disk

How to Export a ZFS Volume as a Full Disk

File or Volume Exported as a Single-Slice Disk

How to Export a ZFS Volume as a Single-Slice Disk

Exporting Volumes and Backward Compatibility

Summary of How Different Types of Back Ends Are Exported

Guidelines for Exporting Files and Disk Slices as Virtual Disks

Using the Loopback File (lofi) Driver

Directly or Indirectly Exporting a Disk Slice

Configuring Virtual Disk Multipathing

Virtual Disk Multipathing and Virtual Disk Timeout

How to Configure Virtual Disk Multipathing

CD, DVD and ISO Images

How to Export a CD or DVD From the Service Domain to the Guest Domain

How to Export an ISO Image From the Control Domain to Install a Guest Domain

Virtual Disk Timeout

Virtual Disk and SCSI

Virtual Disk and the format Command

Using ZFS With Virtual Disks

Configuring a ZFS Pool in a Service Domain

Storing Disk Images With ZFS

Examples of Storing Disk Images With ZFS

Creating a Snapshot of a Disk Image

Using Clone to Provision a New Domain

Cloning a Boot Disk Image

How to Create a Snapshot of a Disk Image of an Unconfigured System

Using Volume Managers in an Oracle VM Server for SPARC Environment

Using Virtual Disks With Volume Managers

Using Virtual Disks With Solaris Volume Manager

Using Virtual Disks When VxVM Is Installed

Using Volume Managers With Virtual Disks

Using ZFS With Virtual Disks

Using Solaris Volume Manager With Virtual Disks

Using VxVM With Virtual Disks

Chapter 8 Using Virtual Networks

Introduction to a Virtual Network

Oracle Solaris 10 Networking Overview

Oracle Solaris 11 Networking Overview

Maximizing Virtual Network Performance

Hardware and Software Requirements

Configuring Your Domains to Maximize the Performance of Your Virtual Network

Virtual Switch

Virtual Network Device

Inter-Vnet LDC Channels

Controlling the Amount of Physical Network Bandwidth That Is Consumed by a Virtual Network Device

Network Bandwidth Limitations

Setting the Network Bandwidth Limit

Virtual Device Identifier and Network Interface Name

How to Find Oracle Solaris OS Network Interface Name

Assigning MAC Addresses Automatically or Manually

Range of MAC Addresses Assigned to Domains

Automatic Assignment Algorithm

Duplicate MAC Address Detection

Freed MAC Addresses

Using Network Adapters With Domains

How to Determine Whether a Network Adapter Is GLDv3-Compliant (Oracle Solaris 10)

Configuring a Virtual Switch and the Service Domain for NAT and Routing

Configuring NAT on an Oracle Solaris 10 System

How to Set Up a Virtual Switch to Provide External Connectivity to Domains (Oracle Solaris 10)

Configuring NAT on an Oracle Solaris 11 System

How to Set Up a Virtual Switch to Provide External Connectivity to Domains (Oracle Solaris 11)

Configuring IPMP in an Oracle VM Server for SPARC Environment

Configuring Virtual Network Devices Into an IPMP Group in a Domain

Configuring and Using IPMP in the Service Domain

Using Link-Based IPMP in Oracle VM Server for SPARC Virtual Networking

How to Configure Physical Link Status Updates

Configuring and Using IPMP in Releases Prior to Logical Domains 1.3

Configuring IPMP in the Guest Domain

Configuring IPMP in the Service Domain

How to Configure a Host Route for Probe-Based IPMP

Using VLAN Tagging

Port VLAN ID

VLAN ID

How to Assign VLANs to a Virtual Switch and Virtual Network Device

How to Install a Guest Domain When the Install Server Is in a VLAN

Using NIU Hybrid I/O

How to Configure a Virtual Switch With an NIU Network Device

How to Enable or Disable Hybrid Mode

Using Link Aggregation With a Virtual Switch

Configuring Jumbo Frames

How to Configure Virtual Network and Virtual Switch Devices to Use Jumbo Frames

Compatibility With Older (Jumbo-Unaware) Versions of the vnet and vsw Drivers (Oracle Solaris 10)

Oracle Solaris 11 Networking-Specific Feature Differences

Chapter 9 Migrating Domains

Introduction to Domain Migration

Overview of a Migration Operation

Software Compatibility

Security for Migration Operations

Migrating a Domain

Performing a Dry Run

Performing Non-Interactive Migrations

Migrating an Active Domain

Domain Migration Requirements for CPUs

Migration Requirements for Memory

Migration Requirements for Physical I/O Devices

Migration Requirements for Virtual I/O Devices

Migration Requirements for PCIe Endpoint Devices

Migration Requirements for PCIe SR-IOV Virtual Functions

Migration Requirements for NIU Hybrid I/O

Migration Requirements for Cryptographic Units

Delayed Reconfiguration in an Active Domain

Migrating While an Active Domain Has the Power Management Elastic Policy in Effect

Operations on Other Domains

Migrating a Domain From the OpenBoot PROM or a Domain That Is Running in the Kernel Debugger

Migrating Bound or Inactive Domains

Migration Requirements for Virtual I/O Devices

Migration Requirements for PCIe Endpoint Devices

Migration Requirements for PCIe SR-IOV Virtual Functions

Monitoring a Migration in Progress

Canceling a Migration in Progress

Recovering From a Failed Migration

Migration Examples

Chapter 10 Managing Resources

Resource Reconfiguration

Dynamic Reconfiguration

Delayed Reconfiguration

Resource Allocation

CPU Allocation

How to Apply the Whole-Core Constraint

How to Apply the Max-Cores Constraint

Interactions Between the Whole-Core Constraint and Other Domain Features

CPU Dynamic Reconfiguration

Dynamic Resource Management

Configuring the System With Hard Partitions

Checking the Configuration of a Domain

Configuring a Domain With CPU Whole Cores

How to Create a New Domain With CPU Whole Cores

How to Configure an Existing Domain With CPU Whole Cores

How to Configure the Primary Domain With CPU Whole Cores

Interaction of Hard Partitioned Systems With Other Oracle VM Server for SPARC Features

CPU Dynamic Reconfiguration

CPU Dynamic Resource Management

Power Management

Domain Reboot or Rebind

Assigning Physical Resources to Domains

How to Remove the physical-bindings Constraint

How to Remove All Non-Physically Bound Resources

Managing Physical Resources on the Control Domain

Restrictions for Managing Physical Resources on Domains

Using Memory Dynamic Reconfiguration

Adding Memory

Removing Memory

Partial Memory DR Requests

Memory Reconfiguration of the Control Domain

Decrease the Control Domain's Memory

Dynamic and Delayed Reconfiguration

Memory Alignment

Memory Alignment for Active Domains

Memory Alignment for Bound Domains

Memory Alignment for Inactive Domains

Adding Unaligned Memory

Memory DR Examples

Using Power Management

Using Dynamic Resource Management

Listing Domain Resources

Machine-Readable Output

Flag Definitions

Utilization Statistic Definition

Viewing Various Lists

Listing Constraints

Chapter 11 Managing Domain Configurations

Managing Domain Configurations

Available Configuration Recovery Methods

Restoring Configurations By Using Autosave

Autorecovery Policy

How to Modify the Autorecovery Policy

Saving Domain Configurations

Restoring Domain Configurations

How to Restore a Domain Configuration From an XML File (ldm add-domain)

How to Restore a Domain Configuration From an XML File (ldm init-system)

Chapter 12 Handling Hardware Errors

Hardware Error-Handling Overview

Using FMA to Blacklist or Unconfigure Faulty Resources

Recovering Domains After Detecting Faulty or Missing Resources

Degraded Configuration

Enabling Recovery Mode

Marking Domains as Degraded

Marking I/O Resources as Evacuated

Chapter 13 Performing Other Administration Tasks

Entering Names in the CLI

Connecting to a Guest Console Over the Network

Using Console Groups

How to Combine Multiple Consoles Into One Group

Stopping a Heavily Loaded Domain Can Time Out

Operating the Oracle Solaris OS With Oracle VM Server for SPARC

OpenBoot Firmware Not Available After the Oracle Solaris OS Has Started

Performing a Power Cycle of a Server

Result of Oracle Solaris OS Breaks

Results From Halting or Rebooting the Control Domain

Using Oracle VM Server for SPARC With the Service Processor

Configuring Domain Dependencies

Domain Dependency Examples

Dependency Cycles

Determining Where Errors Occur by Mapping CPU and Memory Addresses

CPU Mapping

Memory Mapping

Example of CPU and Memory Mapping

Using Universally Unique Identifiers

Virtual Domain Information Command and API

Part II Optional Oracle VM Server for SPARC Software

Glossary

Index

Using Ethernet SR-IOV Virtual Functions

You can use both the static and dynamic SR-IOV methods to manage Ethernet SR-IOV devices.

Ethernet SR-IOV Hardware Requirements

For information about the required PCIe Ethernet SR-IOV hardware, see PCIe SR-IOV Hardware and Software Requirements in Oracle VM Server for SPARC 3.1.1 and 3.1 Release Notes .

Ethernet SR-IOV Limitations

You can enable VLAN configurations of virtual functions by setting either the pvid or the vid property. You cannot set both of these virtual function properties simultaneously.

Planning for the Use of Ethernet SR-IOV Virtual Functions

When dynamically creating virtual functions, ensure that the physical functions use multipathing or that they are not plumbed.

If you cannot use multipathing or must plumb the physical function, use the static method to create the virtual functions. See Static SR-IOV.

Ethernet Device-Specific and Network-Specific Properties

Use the ldm create-vf command to set device-specific and network-specific properties of a virtual function. The unicast-slots property is device-specific. The mac-addr, alt-mac-addrs, mtu, pvid, and vid properties are network-specific.

Note that the mac-addr, alt-mac-addrs, and mtu network-specific properties can be changed only when the virtual function is assigned to the primary domain while in a delayed reconfiguration.

The pvid and vid network-specific properties can be changed without restriction.

Creating Ethernet Virtual Functions

This section describes how to dynamically create and destroy virtual functions. If you cannot use the dynamic methods to perform these actions, initiate a delayed reconfiguration on the root domain before you create or destroy virtual functions.

How to Create an Ethernet SR-IOV Virtual Function

If you cannot use this dynamic method, use the static method instead. See Static SR-IOV.

  1. Identify the physical function device.
    primary# ldm list-io

    Note that the name of the physical function includes the location information for the PCIe SR-IOV card or on-board device.

  2. If I/O virtualization for the bus that has the physical function is not enabled already, enable it.

    Perform this step only if I/O virtualization is not enabled already for the bus that has the physical function.

    See How to Enable I/O Virtualization for a PCIe Bus.

  3. Create a single virtual function or multiple virtual functions from an Ethernet physical function either dynamically or statically.

    After you create one or more virtual functions, you can assign them to a guest domain.

    • Dynamic method:
      • To create multiple virtual functions from a physical function all at the same time, use the following command:
        primary# ldm create-vf -n number | max pf-name

        Use the ldm create-vf -n max command to create all the virtual functions for that physical function at one time.


        Caution - When your system uses an Intel 10-Gbit Ethernet card, maximize performance by creating no more than 31 virtual functions from each physical function.


        You can use either the path name or the pseudonym name to specify virtual functions. However, the recommended practice is to use the pseudonym name.

      • To create one virtual function from a physical function, use the following command:
        primary# ldm create-vf [mac-addr=num] [alt-mac-addrs=[auto|num1,[auto|num2,...]]]
          [pvid=pvid] [vid=vid1,vid2,...] [mtu=size] [name=value...] pf-name

        Note - If not explicitly assigned, the MAC address is automatically allocated for network devices.

        Use this command to create one virtual function for that physical function. You can also manually specify Fibre Channel class-specific property values.

    • Static method:
      1. Initiate a delayed reconfiguration.
        primary# ldm start-reconf root-domain-name
      2. Create a single virtual function or multiple virtual functions from an Ethernet physical function.

        Use the same commands as shown previously to dynamically create the virtual functions.

      3. Reboot the root domain.
        • To reboot the non-primary root domain:
          primary# ldm stop-domain -r root-domain
        • To reboot the primary root domain:
          primary# shutdown -i6 -g0 -y
Example 6-2 Displaying Information About the Ethernet Physical Function

This example shows information about the /SYS/MB/NET0/IOVNET.PF0 physical function:

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
niu_0                                     NIU    niu_0    primary           
niu_1                                     NIU    niu_1    primary           
pci_0                                     BUS    pci_0    primary  
pci_1                                     BUS    pci_1    primary  
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE3                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE5                             PCIE   pci_1    primary  OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP      
/SYS/MB/PCIE9                             PCIE   pci_1    primary  EMP      
/SYS/MB/NET2                              PCIE   pci_1    primary  OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    primary
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    primary
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    primary
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    primary

The following command shows more details about the specified physical function. The maxvfs value indicates the maximum number of virtual functions that is supported by the device.

primary# ldm list-io -l /SYS/MB/NET0/IOVNET.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
[pci@400/pci@1/pci@0/pci@4/network@0]
    maxvfs = 7
Example 6-3 Dynamically Creating an Ethernet Virtual Function Without Setting Optional Properties

This example dynamically creates a virtual function without setting any optional properties. In this case, the MAC address for a network class virtual function is automatically allocated.

Ensure that I/O virtualization is enabled on the pci_0 PCIe bus. See How to Enable I/O Virtualization for a PCIe Bus.

Now, you can use the ldm create-vf command to create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.

primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0
Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0
Example 6-4 Dynamically Creating an Ethernet Virtual Function and Setting Properties

This example dynamically creates a virtual function while setting the mac-addr property to 00:14:2f:f9:14:c0 and the vid property to VLAN IDs 2 and 3.

primary# ldm create-vf mac-addr=00:14:2f:f9:14:c0 vid=2,3 /SYS/MB/NET0/IOVNET.PF0
Example 6-5 Dynamically Creating an Ethernet Virtual Function With Two Alternate MAC Addresses

This example dynamically creates a virtual function that has two alternate MAC addresses. One MAC address is automatically allocated, and the other is explicitly specified as 00:14:2f:f9:14:c2.

primary# ldm create-vf alt-mac-addrs=auto,00:14:2f:f9:14:c2 /SYS/MB/NET0/IOVNET.PF0
Example 6-6 Statically Creating a Virtual Function Without Setting Optional Properties

This example statically creates a virtual function without setting any optional properties. In this case, the MAC address for a network class virtual function is automatically allocated.

First you initiate a delayed reconfiguration on the primary domain and then enable I/O virtualization on the pci_0 PCIe bus. Because the pci_0 bus has already been assigned to the primary root domain, use the ldm set-io command to enable I/O virtualization.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm set-io iov=on pci_0

Now, you can use the ldm create-vf command to create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.

primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0

------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0

Finally, reboot the primary root domain to make the changes take effect.

primary# shutdown -i6 -g0 -y
Example 6-7 Creating Multiple SR-IOV Ethernet Virtual Functions

The following command shows how you can create four virtual functions from the /SYS/MB/NET2/IOVNET.PF1 physical function:

primary# ldm create-vf -n 31 /SYS/MB/NET2/IOVNET.PF1
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF0
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF1
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF2
...
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF30

Note that the ldm create-vf -n command creates multiple virtual functions that are set with default property values, if appropriate. You can later specify non-default property values by using the ldm set-io command.

Destroying Ethernet Virtual Functions

A virtual function can be destroyed if it is not currently assigned to a domain. A virtual function can be destroyed only in the reverse sequential order of creation, so only the last virtual function that was created can be destroyed. The resulting configuration is validated by the physical function driver.

How to Destroy an Ethernet SR-IOV Virtual Function

If you cannot use this dynamic method, use the static method instead. See Static SR-IOV.

  1. Identify the physical function device.
    primary# ldm list-io
  2. Destroy single a virtual function or multiple virtual functions either dynamically or statically.
    • Dynamic method:
      • To destroy all of the virtual functions from a physical function at one time, use the following command:
        primary# ldm destroy-vf -n number | max pf-name

        Use the ldm destroy-vf -n max command to destroy all the virtual functions for that physical function at one time.

        If you specify number as an argument to the –n option, the last number of virtual functions are destroyed. Use this method as it performs this operation with only one physical function device driver state transition.

        You can use either the path name or the pseudonym name to specify virtual functions. However, the recommended practice is to use the pseudonym name.

      • To destroy a specified virtual function:
        primary# ldm destroy-vf vf-name
    • Static method:
      1. Initiate a delayed reconfiguration.
        primary# ldm start-reconf root-domain-name
      2. Destroy either a single virtual function or multiple virtual functions.
        • To destroy all of the virtual functions from the specified physical function at the same time, use the following command:
          primary# ldm destroy-vf -n number | max pf-name

          You can use either the path name or the pseudonym name to specify virtual functions. However, the recommended practice is to use the pseudonym name.

        • To destroy a specified virtual function:
          primary# ldm destroy-vf vf-name
      3. Reboot the root domain.
        • To reboot the non-primary root domain:
          primary# ldm stop-domain -r root-domain
        • To reboot the primary root domain:
          primary# shutdown -i6 -g0 -y
Example 6-8 Destroying an Ethernet Virtual Function

This example shows how to dynamically destroy the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function.

primary# ldm destroy-vf /SYS/MB/NET0/IOVNET.PF0.VF0

The following example shows how to statically destroy the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm destroy-vf /SYS/MB/NET0/IOVNET.PF0.VF0
primary# shutdown -i6 -g0 -y
Example 6-9 Destroying Multiple Ethernet SR-IOV Virtual Functions

This example shows the results of destroying all the virtual functions from the /SYS/MB/NET2/IOVNET.PF1 physical function. The ldm list-io output shows that the physical function has seven virtual functions. The ldm destroy-vf command destroys all virtual functions, and the final ldm list-io output shows that none of the virtual functions remain.

primary# ldm list-io
...
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1                      
/SYS/MB/NET2/IOVNET.PF1.VF0               VF     pci_1                      
/SYS/MB/NET2/IOVNET.PF1.VF1               VF     pci_1                      
/SYS/MB/NET2/IOVNET.PF1.VF2               VF     pci_1                      
/SYS/MB/NET2/IOVNET.PF1.VF3               VF     pci_1                      
/SYS/MB/NET2/IOVNET.PF1.VF4               VF     pci_1                      
/SYS/MB/NET2/IOVNET.PF1.VF5               VF     pci_1                      
/SYS/MB/NET2/IOVNET.PF1.VF6               VF     pci_1
primary# ldm destroy-vf -n max /SYS/MB/NET2/IOVNET.PF1
primary# ldm list-io
...
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    ldg1

Modifying Ethernet SR-IOV Virtual Functions

The ldm set-io vf-name command modifies the current configuration of a virtual function by changing the property values or by setting new properties. This command can modify both the network-specific properties and the device-specific properties. For information about device-specific properties, see Advanced SR-IOV Topics: Ethernet SR-IOV.

If you cannot use this dynamic method, use the static method instead. See Static SR-IOV.

How to Modify an Ethernet SR-IOV Virtual Function

  1. Identify the physical function device.
    primary# ldm list-io

    Note that the name of the physical function includes the location information for the PCIe SR-IOV card or on-board device.

  2. Modify a virtual function.
    primary# ldm set-io name=value [name=value...] vf-name
Example 6-10 Modifying an Ethernet Virtual Function

Adding and Removing Ethernet SR-IOV Virtual Functions on I/O Domains

How to Add an Ethernet SR-IOV Virtual Function to an I/O Domain

If you cannot dynamically remove the virtual function, use the static method. See Static SR-IOV.

  1. Identify the virtual function that you want to add to an I/O domain.
    primary# ldm list-io
  2. Add a virtual function dynamically or statically.
    • To dynamically add a virtual function:
      primary# ldm add-io vf-name domain-name

      vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the pseudonym name. domain-name specifies the name of the domain to which you add the virtual function.

      The device path name for the virtual function in the domain is the path shown in the list-io -l output.

    • To statically add a virtual function:
      1. Initiate a delayed reconfiguration and then add the virtual function.
        primary# ldm start-reconf root-domain-name
        primary# ldm add-io vf-name domain-name

        vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the pseudonym name. domain-name specifies the name of the domain to which you add the virtual function. The specified guest must be in the inactive or bound state.

        The device path name for the virtual function in the domain is the path shown in the list-io -l output.

      2. Reboot the root domain.
        • To reboot the non-primary root domain:
          primary# ldm stop-domain -r root-domain
        • To reboot the primary root domain:
          primary# shutdown -i6 -g0 -y
Example 6-11 Adding an Ethernet Virtual Function

This example shows how to dynamically add the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function to the ldg1 domain.

primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1

If you cannot add the virtual function dynamically, use the static method:

primary# ldm stop-domain ldg1
primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
primary# ldm start-domain ldg1

How to Remove an Ethernet Virtual SR-IOV Function From an I/O Domain

If you cannot dynamically remove the virtual function, use the static method. See Static SR-IOV.


Caution - Before removing the virtual function from the domain, ensure that it is not critical for booting that domain.


  1. Identify the virtual function that you want to remove from an I/O domain.
    primary# ldm list-io
  2. Remove a virtual function either dynammically or statically.
    • To dynamically remove a virtual function:
      primary# ldm rm-io vf-name domain-name

      vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the device pseudonym. domain-name specifies the name of the domain from which you remove the virtual function.

    • To statically remove a virtual function:
      1. Stop the I/O domain.
        primary# ldm stop-domain domain-name
      2. Remove the virtual function.
        primary# ldm rm-io vf-name domain-name

        vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the device pseudonym. domain-name specifies the name of the domain from which you remove the virtual function. The specified guest must be in the inactive or bound state.

      3. Start the I/O domain.
        primary# ldm start-domain domain-name
Example 6-12 Dynamically Removing an Ethernet Virtual Function

This example shows how to dynamically remove the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function from the ldg1 domain.

primary# ldm remove-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1

If the command succeeds, the virtual function is removed from the ldg1 domain. When ldg1 is restarted, the specified virtual function no longer appears in that domain.

If you cannot remove the virtual function dynamically, use the static method:

primary# ldm stop-domain ldg1
primary# ldm remove-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
primary# ldm start-domain ldg1

Advanced SR-IOV Topics: Ethernet SR-IOV

This section describes some advanced topics related to using SR-IOV virtual functions.

Advanced Network Configuration for Virtual Functions
Booting an I/O Domain by Using an SR-IOV Virtual Function

An SR-IOV virtual function provides similar capabilities to any other type of PCIe device, such as the ability to use a virtual function as a logical domain boot device. For example, a network virtual function can be used to boot over the network to install the Oracle Solaris OS in an I/O domain.


Note - When booting the Oracle Solaris OS from a virtual function device, verify that the Oracle Solaris OS that is being loaded has virtual function device support. If so, you can continue with the rest of the installation as planned.
SR-IOV Device-Specific Properties

SR-IOV physical function device drivers can export device-specific properties. These properties can be used to tune the resource allocation of both the physical function and its virtual functions. For information about the properties, see the man page for the physical function driver, such as the igb(7D) and ixgbe(7D) man pages.

The ldm list-io -d command shows device-specific properties that are exported by the specified physical function device driver. The information for each property includes its name, brief description, default value, maximum values, and one or more of the following flags:

P

Applies to a physical function

V

Applies to a virtual function

R

Read-only or informative parameter only

primary# ldm list-io -d pf-name

Use the ldm create-vf or ldm set-io command to set the read-write properties for a physical function or a virtual function. Note that to set a device-specific property, you must use the static method. See Static SR-IOV.

The following example shows the device-specific properties that are exported by the on-board Intel 1-Gbps SR-IOV device:

primary# ldm list-io -d /SYS/MB/NET0/IOVNET.PF0
Device-specific Parameters
--------------------------
max-config-vfs
    Flags = PR
    Default = 7
    Descr = Max number of configurable VFs
max-vf-mtu
    Flags = VR
    Default = 9216
    Descr = Max MTU supported for a VF
max-vlans
    Flags = VR
    Default = 32
    Descr = Max number of VLAN filters supported
pvid-exclusive
    Flags = VR
    Default = 1
    Descr = Exclusive configuration of pvid required
unicast-slots
    Flags = PV
    Default = 0 Min = 0 Max = 24
    Descr = Number of unicast mac-address slots

The following example sets the unicast-slots property to 8:

primary# ldm create-vf unicast-slots=8 /SYS/MB/NET0/IOVNET.PF0
Creating VNICs on SR-IOV Virtual Functions

The creation of Oracle Solaris 11 VNICs is supported on SR-IOV virtual functions. However, the number of VNICs that is supported is limited to the number of alternate MAC addresses (alt-mac-addrs property) assigned to the virtual function. Make sure that you assign a sufficient number of alternate MAC addresses when you use VNICs on the virtual function. Use the ldm create-vf or ldm set-io command to set the alt-mac-addrs property with the alternate MAC addresses.

The following example shows the creation of four VNICs on an SR-IOV virtual function. The first command assigns alternate MAC addresses to the virtual function device. This command uses the automatic allocation method to allocate four alternate MAC addresses to the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function device:

primary# ldm set-io alt-mac-addrs=auto,auto,auto,auto /SYS/MB/NET0/IOVNET.PF0.VF0

The next command starts the ldg1 I/O domain. Because the auto-boot? property is set to true in this example, the Oracle Solaris 11 OS is also booted in the I/O domain.

primary# ldm start ldg1

The following command uses the Oracle Solaris 11 dladm command in the guest domain to show virtual function that has alternate MAC addresses. This output shows that the net30 virtual function has four alternate MAC addresses.

guest# dladm show-phys -m
LINK              SLOT     ADDRESS            INUSE CLIENT
net0              primary  0:14:4f:fa:b4:d1   yes   net0
net25             primary  0:14:4f:fa:c9:eb   no    --
net30             primary  0:14:4f:fb:de:4c   no    --
                   1       0:14:4f:f9:e8:73   no    --
                   2       0:14:4f:f8:21:58   no    --
                   3       0:14:4f:fa:9d:92   no    --
                   4       0:14:4f:f9:8f:1d   no    --

The following commands create four VNICs. Note that attempts to create more VNICs than are specified by using alternate MAC addresses will fail.

guest# dladm create-vnic -l net30 vnic0
guest# dladm create-vnic -l net30 vnic1
guest# dladm create-vnic -l net30 vnic2
guest# dladm create-vnic -l net30 vnic3
guest# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net0                phys      1500   up       --
net25               phys      1500   up       --
net30               phys      1500   up       --
vnic0               vnic      1500   up       net30
vnic1               vnic      1500   up       net30
vnic2               vnic      1500   up       net30
vnic3               vnic      1500   up       net30

Using an SR-IOV Virtual Function to Create an I/O Domain

The following procedure explains how to create an I/O domain that includes PCIe SR-IOV virtual functions.

How to Create an I/O Domain by Assigning an SR-IOV Virtual Function to It

Plan ahead to minimize the number of reboots of the root domain, which minimizes downtime.

Before You Begin

Before you begin, ensure that you have enabled I/O virtualization for the PCIe bus that is the parent of the physical function from which you create virtual functions. See How to Enable I/O Virtualization for a PCIe Bus.

  1. Identify an SR-IOV physical function to share with an I/O domain that uses the SR-IOV feature.
    primary# ldm list-io
  2. Create one or more virtual functions for the physical function.
    primary# ldm create-vf pf-name

    You can run this command for each virtual function that you want to create. You can also use the –n option to create more than one virtual function from the same physical function in a single command. See Example 6–7 and the ldm(1M) man page.


    Note - This command fails if other virtual functions have already been created from the associated physical function and if any of those virtual functions are bound to another domain.
  3. View the list of available virtual functions on the root domain.
    primary# ldm list-io
  4. Assign the virtual function that you created in Step 2 to its target I/O domain.
    primary# ldm add-io vf-name domain-name

    Note - If the OS in the target I/O domain does not support dynamic SR-IOV, you must use the static method. See Static SR-IOV.
  5. Verify that the virtual function is available on the I/O domain.

    The following Oracle Solaris 11 command shows the availability of the virtual function:

    guest# dladm show-phys
Example 6-13 Dynamically Creating an I/O Domain by Assigning an SR-IOV Virtual Function to It

The following dynamic example shows how to create a virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, for a physical function, /SYS/MB/NET0/IOVNET.PF0, and assign the virtual function to the ldg1 I/O domain.

Create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.

primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0
Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0

Add the /SYS/MB/NET0/IOVNET.PF0.VF0 virtual function to the ldg1 domain.

primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1

The following command shows that the virtual function has been added to the ldg1 domain.

primary# ldm list-io
NAME                          TYPE  BUS      DOMAIN    STATUS
----                          ----  ---      ------    ------
niu_0                         NIU   niu_0    primary
niu_1                         NIU   niu_1    primary
pci_0                         BUS   pci_0    primary   IOV
pci_1                         BUS   pci_1    primary
/SYS/MB/PCIE0                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE2                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE4                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE6                 PCIE  pci_0    primary   EMP
/SYS/MB/PCIE8                 PCIE  pci_0    primary   EMP
/SYS/MB/SASHBA                PCIE  pci_0    primary   OCC
/SYS/MB/NET0                  PCIE  pci_0    primary   OCC
/SYS/MB/PCIE1                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE3                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE5                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE7                 PCIE  pci_1    primary   EMP
/SYS/MB/PCIE9                 PCIE  pci_1    primary   EMP
/SYS/MB/NET2                  PCIE  pci_1    primary   OCC
/SYS/MB/NET0/IOVNET.PF0       PF    pci_0    primary
/SYS/MB/NET0/IOVNET.PF1       PF    pci_0    primary
/SYS/MB/PCIE5/IOVNET.PF0      PF    pci_1    primary
/SYS/MB/PCIE5/IOVNET.PF1      PF    pci_1    primary
/SYS/MB/NET2/IOVNET.PF0       PF    pci_1    primary
/SYS/MB/NET2/IOVNET.PF1       PF    pci_1    primary
/SYS/MB/NET0/IOVNET.PF0.VF0   VF    pci_0    ldg1
Example 6-14 Statically Creating an I/O Domain by Assigning an SR-IOV Virtual Function to It

The following static example shows how to create a virtual function, /SYS/MB/NET0/IOVNET.PF0.VF0, for a physical function, /SYS/MB/NET0/IOVNET.PF0, and assign the virtual function to the ldg1 I/O domain.

First, initiate a delayed reconfiguration on the primary domain, enable I/O virtualization, and create the virtual function from the /SYS/MB/NET0/IOVNET.PF0 physical function.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm set-io iov=on pci_0
primary# ldm create-vf /SYS/MB/NET0/IOVNET.PF0

------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
Created new vf: /SYS/MB/NET0/IOVNET.PF0.VF0

Next, shut down the primary domain.

primary# shutdown -i6 -g0 -y

Stop the ldg1 domain, add the virtual function, and start the domain.

primary# ldm stop ldg1
primary# ldm add-io /SYS/MB/NET0/IOVNET.PF0.VF0 ldg1
primary# ldm start ldg1

The following command shows that the virtual function has been added to the ldg1 domain.

primary# ldm list-io
NAME                          TYPE  BUS      DOMAIN    STATUS
----                          ----  ---      ------    ------
niu_0                         NIU   niu_0    primary
niu_1                         NIU   niu_1    primary
pci_0                         BUS   pci_0    primary   IOV
pci_1                         BUS   pci_1    primary
/SYS/MB/PCIE0                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE2                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE4                 PCIE  pci_0    primary   OCC
/SYS/MB/PCIE6                 PCIE  pci_0    primary   EMP
/SYS/MB/PCIE8                 PCIE  pci_0    primary   EMP
/SYS/MB/SASHBA                PCIE  pci_0    primary   OCC
/SYS/MB/NET0                  PCIE  pci_0    primary   OCC
/SYS/MB/PCIE1                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE3                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE5                 PCIE  pci_1    primary   OCC
/SYS/MB/PCIE7                 PCIE  pci_1    primary   EMP
/SYS/MB/PCIE9                 PCIE  pci_1    primary   EMP
/SYS/MB/NET2                  PCIE  pci_1    primary   OCC
/SYS/MB/NET0/IOVNET.PF0       PF    pci_0    primary
/SYS/MB/NET0/IOVNET.PF1       PF    pci_0    primary
/SYS/MB/PCIE5/IOVNET.PF0      PF    pci_1    primary
/SYS/MB/PCIE5/IOVNET.PF1      PF    pci_1    primary
/SYS/MB/NET2/IOVNET.PF0       PF    pci_1    primary
/SYS/MB/NET2/IOVNET.PF1       PF    pci_1    primary
/SYS/MB/NET0/IOVNET.PF0.VF0   VF    pci_0    ldg1