JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Oracle VM Server for SPARC 3.1 Administration Guide
Oracle Technology Network
Library
PDF
Print View
Feedback
search filter icon
search icon

Document Information

Using This Documentation

Part I Oracle VM Server for SPARC 3.1 Software

Chapter 1 Overview of the Oracle VM Server for SPARC Software

About Oracle VM Server for SPARC and Oracle Solaris OS Versions

Hypervisor and Logical Domains

Logical Domains Manager

Roles for Domains

Command-Line Interface

Virtual Input/Output

Virtual Network

Virtual Storage

Virtual Console

Resource Configuration

Persistent Configurations

Oracle VM Server for SPARC Physical-to-Virtual Conversion Tool

Oracle VM Server for SPARC Configuration Assistant

Oracle VM Server for SPARC Management Information Base

Chapter 2 Installing and Enabling Software

Required Oracle VM Server for SPARC Software Components

Installing Oracle VM Server for SPARC Software on a New System

Updating the Oracle Solaris OS

Upgrading the System Firmware

Downloading the Logical Domains Manager

How to Download the Logical Domains Manager Software (Oracle Solaris 10)

Installing the Logical Domains Manager

Automatically Installing the Logical Domains Manager Software (Oracle Solaris 10)

Manually Installing the Logical Domains Manager Software

How to Manually Install the Oracle VM Server for SPARC 3.1 Software (Oracle Solaris 10)

Enabling the Logical Domains Manager Daemon

How to Enable the Logical Domains Manager Daemon

Upgrading a System That Is Already Using Oracle VM Server for SPARC

Upgrading the Oracle Solaris OS

Saving and Restoring Autosave Configuration Directories

How to Save and Restore Autosave Directories

Saving and Restoring the Logical Domains Constraints Database File

Preserving the Logical Domains Constraints Database File When Using the Oracle Solaris 10 Live Upgrade Feature

Upgrading the Logical Domains Manager and the System Firmware

How to Stop All Domains Running on the Platform, Except the Control Domain

Upgrading to Oracle VM Server for SPARC 3.1 Software

How to Upgrade to the Oracle VM Server for SPARC 3.1 Software (Oracle Solaris 10)

How to Upgrade to the Oracle VM Server for SPARC 3.1 Software (Oracle Solaris 11)

Factory Default Configuration and Disabling Domains

How to Remove All Guest Domains

How to Remove All Domain Configurations

How to Restore the Factory Default Configuration

How to Disable the Logical Domains Manager

How to Remove the Logical Domains Manager

How to Restore the Factory Default Configuration From the Service Processor

Chapter 3 Oracle VM Server for SPARC Security

Delegating the Management of Logical Domains by Using Rights

Using Rights Profiles and Roles

Managing User Rights Profiles

How to Assign a Rights Profile to a User

Assigning Roles to Users

How to Create a Role and Assign the Role to a User

Logical Domains Manager Rights Profile Contents

Controlling Access to a Domain Console by Using Rights

How to Control Access to All Domain Consoles by Using Roles

How to Control Access to All Domain Consoles by Using Rights Profiles

How to Control Access to a Single Console by Using Roles

How to Control Access to a Single Console by Using Rights Profiles

Enabling and Using Auditing

How to Enable Auditing

How to Disable Auditing

How to Review Audit Records

How to Rotate Audit Logs

Using Domain Console Logging

How to Enable or Disable Console Logging

Service Domain Requirements for Domain Console Logging

Chapter 4 Setting Up Services and the Control Domain

Output Messages

Creating Default Services

How to Create Default Services

Initial Configuration of the Control Domain

How to Configure the Control Domain

Rebooting to Use Domains

How to Reboot

Enabling Networking Between the Control/Service Domain and Other Domains

How to Configure the Virtual Switch as the Primary Interface

Enabling the Virtual Network Terminal Server Daemon

How to Enable the Virtual Network Terminal Server Daemon

Chapter 5 Setting Up Guest Domains

Creating and Starting a Guest Domain

How to Create and Start a Guest Domain

Installing Oracle Solaris OS on a Guest Domain

How to Install the Oracle Solaris OS on a Guest Domain From a DVD

How to Install the Oracle Solaris OS on a Guest Domain From an Oracle Solaris ISO File

How to Use the Oracle Solaris JumpStart Feature on an Oracle Solaris 10 Guest Domain

Chapter 6 Setting Up I/O Domains

I/O Domain Overview

General Guidelines for Creating an I/O Domain

Creating a Root Domain by Assigning PCIe Buses

How to Create an I/O Domain by Assigning a PCIe Bus

Creating an I/O Domain by Assigning PCIe Endpoint Devices

Direct I/O Hardware and Software Requirements

Current Direct I/O Feature Limitations

Planning PCIe Endpoint Device Configuration

Rebooting the Root Domain

Making PCIe Hardware Changes

How to Create an I/O Domain by Assigning a PCIe Endpoint Device

Creating an I/O Domain by Assigning PCIe SR-IOV Virtual Functions

SR-IOV Overview

SR-IOV Hardware and Software Requirements

Current SR-IOV Feature Limitations

Static SR-IOV

Static SR-IOV Software Requirements

Dynamic SR-IOV

Dynamic SR-IOV Software Requirements

Dynamic SR-IOV Configuration Requirements

Enabling I/O Virtualization

How to Enable I/O Virtualization for a PCIe Bus

Planning for the Use of PCIe SR-IOV Virtual Functions

Using Ethernet SR-IOV Virtual Functions

Ethernet SR-IOV Hardware Requirements

Ethernet SR-IOV Limitations

Planning for the Use of Ethernet SR-IOV Virtual Functions

Ethernet Device-Specific and Network-Specific Properties

Creating Ethernet Virtual Functions

How to Create an Ethernet SR-IOV Virtual Function

Destroying Ethernet Virtual Functions

How to Destroy an Ethernet SR-IOV Virtual Function

Modifying Ethernet SR-IOV Virtual Functions

How to Modify an Ethernet SR-IOV Virtual Function

Adding and Removing Ethernet SR-IOV Virtual Functions on I/O Domains

How to Add an Ethernet SR-IOV Virtual Function to an I/O Domain

How to Remove an Ethernet Virtual SR-IOV Function From an I/O Domain

Advanced SR-IOV Topics: Ethernet SR-IOV

Advanced Network Configuration for Virtual Functions

Booting an I/O Domain by Using an SR-IOV Virtual Function

SR-IOV Device-Specific Properties

Creating VNICs on SR-IOV Virtual Functions

Using an SR-IOV Virtual Function to Create an I/O Domain

How to Create an I/O Domain by Assigning an SR-IOV Virtual Function to It

Using InfiniBand SR-IOV Virtual Functions

InfiniBand SR-IOV Hardware Requirements

Creating and Destroying InfiniBand Virtual Functions

How to Create an InfiniBand Virtual Function

How to Destroy an InfiniBand Virtual Function

Adding and Removing InfiniBand Virtual Functions on I/O Domains

How to Add an InfiniBand Virtual Function to an I/O Domain

How to Remove an InfiniBand Virtual Function From an I/O Domain

Adding and Removing InfiniBand Virtual Functions to Root Domains

How to Add an InfiniBand Virtual Function to a Root Domain

How to Remove an InfiniBand Virtual Function From a Root Domain

Advanced SR-IOV Topics: InfiniBand SR-IOV

Listing InfiniBand SR-IOV Virtual Functions

Identifying InfiniBand SR-IOV Functions

Using Fibre Channel SR-IOV Virtual Functions

Fibre Channel SR-IOV Hardware Requirements

Fibre Channel SR-IOV Requirements and Limitations

Fibre Channel Device Class-Specific Properties

World-Wide Name Allocation for Fibre Channel Virtual Functions

Creating Fibre Channel SR-IOV Virtual Functions

How to Create a Fibre Channel SR-IOV Virtual Function

Destroying Fibre Channel SR-IOV Virtual Functions

How to Destroy a Fibre Channel SR-IOV Virtual Function

Modifying Fibre Channel SR-IOV Virtual Functions

How to Modify a Fibre Channel SR-IOV Virtual Function

Adding and Removing Fibre Channel SR-IOV Virtual Functions on I/O Domains

How to Add a Fibre Channel SR-IOV Virtual Function to an I/O Domain

How to Remove a Fibre Channel SR-IOV Virtual Function From an I/O Domain

Advanced SR-IOV Topics: Fibre Channel SR-IOV

Accessing a Fibre Channel Virtual Function in a Guest Domain

SR-IOV: Rebooting the Root Domain

Using Non-primary Root Domains

Non-primary Root Domain Requirements

Non-primary Root Domain Limitations

Enabling I/O Virtualization for a PCIe Bus

Managing Direct I/O Devices on Non-primary Root Domains

Managing SR-IOV Virtual Functions on Non-primary Root Domains

Chapter 7 Using Virtual Disks

Introduction to Virtual Disks

Virtual Disk Identifier and Device Name

Managing Virtual Disks

How to Add a Virtual Disk

How to Export a Virtual Disk Back End Multiple Times

How to Change Virtual Disk Options

How to Change the Timeout Option

How to Remove a Virtual Disk

Virtual Disk Appearance

Full Disk

Single-Slice Disk

Virtual Disk Back End Options

Read-only (ro) Option

Exclusive (excl) Option

Slice (slice) Option

Virtual Disk Back End

Physical Disk or Disk LUN

How to Export a Physical Disk as a Virtual Disk

Physical Disk Slice

How to Export a Physical Disk Slice as a Virtual Disk

How to Export Slice 2

File and Volume Exporting

File or Volume Exported as a Full Disk

How to Export a File as a Full Disk

How to Export a ZFS Volume as a Full Disk

File or Volume Exported as a Single-Slice Disk

How to Export a ZFS Volume as a Single-Slice Disk

Exporting Volumes and Backward Compatibility

Summary of How Different Types of Back Ends Are Exported

Guidelines for Exporting Files and Disk Slices as Virtual Disks

Using the Loopback File (lofi) Driver

Directly or Indirectly Exporting a Disk Slice

Configuring Virtual Disk Multipathing

Virtual Disk Multipathing and Virtual Disk Timeout

How to Configure Virtual Disk Multipathing

CD, DVD and ISO Images

How to Export a CD or DVD From the Service Domain to the Guest Domain

How to Export an ISO Image From the Control Domain to Install a Guest Domain

Virtual Disk Timeout

Virtual Disk and SCSI

Virtual Disk and the format Command

Using ZFS With Virtual Disks

Configuring a ZFS Pool in a Service Domain

Storing Disk Images With ZFS

Examples of Storing Disk Images With ZFS

Creating a Snapshot of a Disk Image

Using Clone to Provision a New Domain

Cloning a Boot Disk Image

How to Create a Snapshot of a Disk Image of an Unconfigured System

Using Volume Managers in an Oracle VM Server for SPARC Environment

Using Virtual Disks With Volume Managers

Using Virtual Disks With Solaris Volume Manager

Using Virtual Disks When VxVM Is Installed

Using Volume Managers With Virtual Disks

Using ZFS With Virtual Disks

Using Solaris Volume Manager With Virtual Disks

Using VxVM With Virtual Disks

Chapter 8 Using Virtual Networks

Introduction to a Virtual Network

Oracle Solaris 10 Networking Overview

Oracle Solaris 11 Networking Overview

Maximizing Virtual Network Performance

Hardware and Software Requirements

Configuring Your Domains to Maximize the Performance of Your Virtual Network

Virtual Switch

Virtual Network Device

Inter-Vnet LDC Channels

Controlling the Amount of Physical Network Bandwidth That Is Consumed by a Virtual Network Device

Network Bandwidth Limitations

Setting the Network Bandwidth Limit

Virtual Device Identifier and Network Interface Name

How to Find Oracle Solaris OS Network Interface Name

Assigning MAC Addresses Automatically or Manually

Range of MAC Addresses Assigned to Domains

Automatic Assignment Algorithm

Duplicate MAC Address Detection

Freed MAC Addresses

Using Network Adapters With Domains

How to Determine Whether a Network Adapter Is GLDv3-Compliant (Oracle Solaris 10)

Configuring a Virtual Switch and the Service Domain for NAT and Routing

Configuring NAT on an Oracle Solaris 10 System

How to Set Up a Virtual Switch to Provide External Connectivity to Domains (Oracle Solaris 10)

Configuring NAT on an Oracle Solaris 11 System

How to Set Up a Virtual Switch to Provide External Connectivity to Domains (Oracle Solaris 11)

Configuring IPMP in an Oracle VM Server for SPARC Environment

Configuring Virtual Network Devices Into an IPMP Group in a Domain

Configuring and Using IPMP in the Service Domain

Using Link-Based IPMP in Oracle VM Server for SPARC Virtual Networking

How to Configure Physical Link Status Updates

Configuring and Using IPMP in Releases Prior to Logical Domains 1.3

Configuring IPMP in the Guest Domain

Configuring IPMP in the Service Domain

How to Configure a Host Route for Probe-Based IPMP

Using VLAN Tagging

Port VLAN ID

VLAN ID

How to Assign VLANs to a Virtual Switch and Virtual Network Device

How to Install a Guest Domain When the Install Server Is in a VLAN

Using NIU Hybrid I/O

How to Configure a Virtual Switch With an NIU Network Device

How to Enable or Disable Hybrid Mode

Using Link Aggregation With a Virtual Switch

Configuring Jumbo Frames

How to Configure Virtual Network and Virtual Switch Devices to Use Jumbo Frames

Compatibility With Older (Jumbo-Unaware) Versions of the vnet and vsw Drivers (Oracle Solaris 10)

Oracle Solaris 11 Networking-Specific Feature Differences

Chapter 9 Migrating Domains

Introduction to Domain Migration

Overview of a Migration Operation

Software Compatibility

Security for Migration Operations

Migrating a Domain

Performing a Dry Run

Performing Non-Interactive Migrations

Migrating an Active Domain

Domain Migration Requirements for CPUs

Migration Requirements for Memory

Migration Requirements for Physical I/O Devices

Migration Requirements for Virtual I/O Devices

Migration Requirements for PCIe Endpoint Devices

Migration Requirements for PCIe SR-IOV Virtual Functions

Migration Requirements for NIU Hybrid I/O

Migration Requirements for Cryptographic Units

Delayed Reconfiguration in an Active Domain

Migrating While an Active Domain Has the Power Management Elastic Policy in Effect

Operations on Other Domains

Migrating a Domain From the OpenBoot PROM or a Domain That Is Running in the Kernel Debugger

Migrating Bound or Inactive Domains

Migration Requirements for Virtual I/O Devices

Migration Requirements for PCIe Endpoint Devices

Migration Requirements for PCIe SR-IOV Virtual Functions

Monitoring a Migration in Progress

Canceling a Migration in Progress

Recovering From a Failed Migration

Migration Examples

Chapter 10 Managing Resources

Resource Reconfiguration

Dynamic Reconfiguration

Delayed Reconfiguration

Resource Allocation

CPU Allocation

How to Apply the Whole-Core Constraint

How to Apply the Max-Cores Constraint

Interactions Between the Whole-Core Constraint and Other Domain Features

CPU Dynamic Reconfiguration

Dynamic Resource Management

Configuring the System With Hard Partitions

Checking the Configuration of a Domain

Configuring a Domain With CPU Whole Cores

How to Create a New Domain With CPU Whole Cores

How to Configure an Existing Domain With CPU Whole Cores

How to Configure the Primary Domain With CPU Whole Cores

Interaction of Hard Partitioned Systems With Other Oracle VM Server for SPARC Features

CPU Dynamic Reconfiguration

CPU Dynamic Resource Management

Power Management

Domain Reboot or Rebind

Assigning Physical Resources to Domains

How to Remove the physical-bindings Constraint

How to Remove All Non-Physically Bound Resources

Managing Physical Resources on the Control Domain

Restrictions for Managing Physical Resources on Domains

Using Memory Dynamic Reconfiguration

Adding Memory

Removing Memory

Partial Memory DR Requests

Memory Reconfiguration of the Control Domain

Decrease the Control Domain's Memory

Dynamic and Delayed Reconfiguration

Memory Alignment

Memory Alignment for Active Domains

Memory Alignment for Bound Domains

Memory Alignment for Inactive Domains

Adding Unaligned Memory

Memory DR Examples

Using Power Management

Using Dynamic Resource Management

Listing Domain Resources

Machine-Readable Output

Flag Definitions

Utilization Statistic Definition

Viewing Various Lists

Listing Constraints

Chapter 11 Managing Domain Configurations

Managing Domain Configurations

Available Configuration Recovery Methods

Restoring Configurations By Using Autosave

Autorecovery Policy

How to Modify the Autorecovery Policy

Saving Domain Configurations

Restoring Domain Configurations

How to Restore a Domain Configuration From an XML File (ldm add-domain)

How to Restore a Domain Configuration From an XML File (ldm init-system)

Chapter 12 Handling Hardware Errors

Hardware Error-Handling Overview

Using FMA to Blacklist or Unconfigure Faulty Resources

Recovering Domains After Detecting Faulty or Missing Resources

Degraded Configuration

Enabling Recovery Mode

Marking Domains as Degraded

Marking I/O Resources as Evacuated

Chapter 13 Performing Other Administration Tasks

Entering Names in the CLI

Connecting to a Guest Console Over the Network

Using Console Groups

How to Combine Multiple Consoles Into One Group

Stopping a Heavily Loaded Domain Can Time Out

Operating the Oracle Solaris OS With Oracle VM Server for SPARC

OpenBoot Firmware Not Available After the Oracle Solaris OS Has Started

Performing a Power Cycle of a Server

Result of Oracle Solaris OS Breaks

Results From Halting or Rebooting the Control Domain

Using Oracle VM Server for SPARC With the Service Processor

Configuring Domain Dependencies

Domain Dependency Examples

Dependency Cycles

Determining Where Errors Occur by Mapping CPU and Memory Addresses

CPU Mapping

Memory Mapping

Example of CPU and Memory Mapping

Using Universally Unique Identifiers

Virtual Domain Information Command and API

Part II Optional Oracle VM Server for SPARC Software

Glossary

Index

Using InfiniBand SR-IOV Virtual Functions

Only the static SR-IOV feature is supported for InfiniBand SR-IOV devices.

To minimize downtime, run all of the SR-IOV commands as a group while the root domain is in delayed reconfiguration or a guest domain is stopped. The SR-IOV commands that are limited in this way are the ldm create-vf, ldm destroy-vf, ldm add-io, and ldm remove-io commands.

Typically, virtual functions are assigned to more than one guest domain. A reboot of the root domain affects all of the guest domains that have been assigned the root domain's virtual functions.

Because an unused InfiniBand virtual function has very little overhead, you can avoid downtime by creating the necessary virtual functions ahead of time, even if they are not used immediately.

InfiniBand SR-IOV Hardware Requirements

For information about the required PCIe InfiniBand SR-IOV hardware, see PCIe SR-IOV Hardware and Software Requirements in Oracle VM Server for SPARC 3.1.1 and 3.1 Release Notes .

For InfiniBand SR-IOV support, the root domain must be running at least the Oracle Solaris 11.1.10.5.0 OS. The I/O domains can run the Oracle Solaris 10 1/13 OS plus patch 148888-04 or at least the Oracle Solaris 11.1.10.5.0 OS.

Creating and Destroying InfiniBand Virtual Functions

How to Create an InfiniBand Virtual Function

This procedure describes how to create an InfiniBand SR-IOV virtual function.

  1. Initiate a delayed reconfiguration on the root domain.
    primary# ldm start-reconf root-domain-name
  2. Enable I/O virtualization by setting iov=on.

    Perform this step only if I/O virtualization is not enabled already for the bus that has the physical function.

    primary# ldm set-io iov=on bus
  3. Create one or more virtual functions that are associated with the physical functions from that root domain.
    primary# ldm create-vf pf-name

    You can run this command for each virtual function that you want to create. You can also use the –n option to create more than one virtual function from the same physical function in a single command. See Example 6–7 and the ldm(1M) man page.

  4. Reboot the root domain.

    Run one of the following commands:

    • Reboot the non-primary root domain.
      primary# ldm stop-domain -r root-domain
    • Reboot the primary root domain.
      primary# shutdown -i6 -g0 -y
Example 6-15 Creating an InfiniBand Virtual Function
primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
pci_0                                     BUS    pci_0    primary
niu_0                                     NIU    niu_0    primary
/SYS/MB/RISER0/PCIE0                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER1/PCIE1                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER2/PCIE2                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER0/PCIE3                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER1/PCIE4                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER2/PCIE5                      PCIE   pci_0    primary  EMP
/SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
/SYS/MB/SASHBA1                           PCIE   pci_0    primary  OCC
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC
/SYS/MB/NET2                              PCIE   pci_0    primary  OCC
/SYS/MB/RISER0/PCIE3/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_0    primary

The following command shows more details about the specified physical function. The maxvfs value indicates the maximum number of virtual functions that are supported by the device.

primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
    maxvfs = 64

The following example shows how to create a static virtual function. First, initiate a delayed reconfiguration on the primary domain and enable I/O virtualization on the pci_0 PCIe bus. Because the pci_0 bus has been assigned already to the primary root domain, use the ldm set-io command to enable I/O virtualization.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm set-io iov=on pci_0
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
-----------------------------------------------------------------------

Now, use the ldm create-vf command to create a virtual function from the /SYS/MB/RISER1/PCIE4/IOVIB.PF0 physical function.

primary# ldm create-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
----------------------------------------------------------------------
Created new vf: /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0

Note that you can create more than one virtual function during the same delayed reconfiguration. The following command creates a second virtual function:

primary# ldm create-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
-----------------------------------------------------------------------
Created new vf: /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1

Finally, reboot the primary root domain to make the changes take effect.

primary# shutdown -i6 -g0 -y
Shutdown started.

Changing to init state 6 - please wait
...

How to Destroy an InfiniBand Virtual Function

This procedure describes how to destroy an InfiniBand SR-IOV virtual function.

A virtual function can be destroyed if it is not currently assigned to a domain. A virtual function can be destroyed only in the reverse sequential order of creation, so only the last virtual function that was created can be destroyed. The resulting configuration is validated by the physical function driver.

  1. Initiate a delayed reconfiguration on the root domain.
    primary# ldm start-reconf root-domain-name
  2. Destroy one or more virtual functions that are associated with the physical functions from that root domain.
    primary# ldm destroy-vf vf-name

    You can run this command for each virtual function that you want to destroy. You can also use the –n option to destroy more than one virtual function from the same physical function in a single command. See Example 6–9 and the ldm(1M) man page.

  3. Reboot the root domain.

    Run one of the following commands:

    • Reboot the non-primary root domain.
      primary# ldm stop-domain -r root-domain
    • Reboot the primary root domain.
      primary# shutdown -i6 -g0 -y
Example 6-16 Destroying an InfiniBand Virtual Function

The following example shows how to destroy a static InfiniBand virtual function, /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1.

The ldm list-io command shows information about the buses, physical functions, and virtual functions.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN STATUS
----                                      ----   ---      ------ ------
pci_0                                     BUS    pci_0    primary  IOV
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0

You can obtain more details about the physical function and related virtual functions by using the ldm list-io -l command.

primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME                                      TYPE   BUS      DOMAIN STATUS
----                                      ----   ---      ------ ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
    maxvfs = 64
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,1]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,2]

A virtual function can be destroyed only if it is unassigned to a domain. The DOMAIN column of the ldm list-io -l output shows the name of any domain to which a virtual function is assigned. Also, virtual functions must be destroyed in the reverse order of their creation. Therefore, in this example, you must destroy the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1 virtual function before you can destroy the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0 virtual function.

After you identify the proper virtual function, you can destroy it. First, initiate a delayed reconfiguration.

primary# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

primary# ldm destroy-vf /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1
-----------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
-----------------------------------------------------------------------

You can issue more than one ldm destroy-vf command while in delayed reconfiguration. Thus, you could also destroy the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0.

Finally, reboot the primary root domain to make the changes take effect.

primary# shutdown -i6 -g0 -y
Shutdown started.

Changing to init state 6 - please wait
...

Adding and Removing InfiniBand Virtual Functions on I/O Domains

How to Add an InfiniBand Virtual Function to an I/O Domain

This procedure describes how to add an InfiniBand SR-IOV virtual function to an I/O domain.

  1. Stop the I/O domain.
    primary# ldm stop-domain domain-name
  2. Add one or more virtual functions to the I/O domain.

    vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the pseudonym name. domain-name specifies the name of the domain to which you add the virtual function. The specified I/O domain must be in the inactive or bound state.

    primary# ldm add-io vf-name domain-name
  3. Start the I/O domain.
    primary# ldm start-domain domain-name
Example 6-17 Adding an InfiniBand Virtual Function

The following example shows how to add the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 virtual function to the iodom1 I/O domain.

First, identify the virtual function that you want to assign.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN STATUS
----                                      ----   ---      ------ ------
pci_0                                     BUS    pci_0    primary  IOV
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0

To add a virtual function to an I/O domain, it must be unassigned. The DOMAIN column indicates the name of the domain to which the virtual function is assigned. In this case, the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 is not assigned to a domain.

To add a virtual function to a domain, the domain must be in the inactive or bound state.

primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.2%  0.2%  56m
iodom1           active     -n----  5000    8     8G        33%   33%  25m

The ldm list-domain output shows that the iodom1 I/O domain is active, so it must be stopped.

primary# ldm stop iodom1
LDom iodom1 stopped
primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.0%  0.0%  57m
iodom1           bound      ------  5000    8     8G

Now you can add the virtual function to the I/O domain.

primary# ldm add-io /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 iodom1
primary# ldm list-io
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1

Note that you can add more than one virtual function while an I/O domain is stopped. For example, you might add other unassigned virtual functions such as /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3 to iodom1. After you add the virtual functions, you can restart the I/O domain.

primary# ldm start iodom1
LDom iodom1 started
primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      1.0%  1.0%  1h 18m
iodom1           active     -n----  5000    8     8G        36%   36%  1m

How to Remove an InfiniBand Virtual Function From an I/O Domain

This procedure describes how to remove an InfiniBand SR-IOV virtual function from an I/O domain.

  1. Stop the I/O domain.
    primary# ldm stop-domain domain-name
  2. Remove one or more virtual functions from the I/O domain.

    vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the device pseudonym. domain-name specifies the name of the domain from which you remove the virtual function. The specified I/O domain must be in the inactive or bound state.


    Note - Before removing the virtual function from the I/O domain, ensure that it is not critical for booting that domain.
    primary# ldm rm-io vf-name domain-name
  3. Start the I/O domain.
    primary# ldm start-domain domain-name
Example 6-18 Removing an InfiniBand Virtual Function

The following example shows how to remove the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 virtual function from the iodom1 I/O domain.

First, identify the virtual function that you want to remove.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN STATUS
----                                      ----   ---      ------ ------
pci_0                                     BUS    pci_0    primary  IOV
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0    iodom1

The DOMAIN column shows the name of the domain to which the virtual function is assigned. The /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 virtual function is assigned to iodom1.

To remove a virtual function from an I/O domain, the domain must be inactive or bound state. Use the ldm list-domain command to determine the state of the domain.

primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.3%  0.3%  29m
iodom1           active     -n----  5000    8     8G        17%   17%  11m

In this case, the iodom1 domain is active and so must be stopped.

primary# ldm stop iodom1
LDOM iodom1 stopped
primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.0%  0.0%  31m
iodom1           bound      ------  5000    8     8G

Now you can remove the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 virtual function from iodom1.

primary# ldm rm-io /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 iodom1
primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN STATUS
----                                      ----   ---      ------ ------
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0
...

Note that the DOMAIN column for the virtual function is now empty.

You can remove more than one virtual function while an I/O domain is stopped. In this example, you could also remove the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3 virtual function. After you remove the virtual functions, you can restart the I/O domain.

primary# ldm start iodom1
LDom iodom1 started
primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.3%  0.3%  39m
iodom1           active     -n----  5000    8     8G       9.4%  9.4%  5s

Adding and Removing InfiniBand Virtual Functions to Root Domains

How to Add an InfiniBand Virtual Function to a Root Domain

This procedure describes how to add an InfiniBand SR-IOV virtual function to a root domain.

  1. Initiate a delayed reconfiguration.
    primary# ldm start-reconf root-domain
  2. Add one or more virtual functions to the root domain.

    vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the pseudonym name. root-domain-name specifies the name of the root domain to which you add the virtual function.

    primary# ldm add-io vf-name root-domain-name
  3. Reboot the root domain.

    Run one of the following commands:

    • Reboot the non-primary root domain.
      primary# ldm stop-domain -r root-domain-name
    • Reboot the primary root domain.
      primary# shutdown -i6 -g0 -y

How to Remove an InfiniBand Virtual Function From a Root Domain

This procedure describes how to remove an InfiniBand SR-IOV virtual function from a root domain.

  1. Initiate a delayed reconfiguration.
    primary# ldm start-reconf root-domain
  2. Remove one or more virtual functions from the root domain.

    vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the pseudonym name. root-domain-name specifies the name of the root domain to which you add the virtual function.

    primary# ldm remove-io vf-name root-domain-name
  3. Reboot the root domain.

    Run one of the following commands:

    • Reboot the non-primary root domain.
      primary# ldm stop-domain -r root-domain-name
    • Reboot the primary root domain.
      primary# shutdown -i6 -g0 -y

Advanced SR-IOV Topics: InfiniBand SR-IOV

This section describes how to identify InfiniBand physical and virtual functions as well as to correlate the Logical Domains Manager and the Oracle Solaris view of InfiniBand physical and virtual functions.

Listing InfiniBand SR-IOV Virtual Functions

The following example shows different ways to display information about the /SYS/MB/RISER1/PCIE4/IOVIB.PF0 physical function. A physical function name that includes the IOVIB string indicates that it is an InfiniBand SR-IOV device.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
pci_0                                     BUS    pci_0    primary  IOV
niu_0                                     NIU    niu_0    primary
/SYS/MB/RISER0/PCIE0                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER1/PCIE1                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER2/PCIE2                      PCIE   pci_0    primary  EMP
/SYS/MB/RISER0/PCIE3                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER1/PCIE4                      PCIE   pci_0    primary  OCC
/SYS/MB/RISER2/PCIE5                      PCIE   pci_0    primary  EMP
/SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
/SYS/MB/SASHBA1                           PCIE   pci_0    primary  OCC
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC
/SYS/MB/NET2                              PCIE   pci_0    primary  OCC
/SYS/MB/RISER0/PCIE3/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/RISER0/PCIE3/IOVIB.PF0.VF0        VF     pci_0    primary
/SYS/MB/RISER0/PCIE3/IOVIB.PF0.VF1        VF     pci_0    primary
/SYS/MB/RISER0/PCIE3/IOVIB.PF0.VF2        VF     pci_0    iodom1
/SYS/MB/RISER0/PCIE3/IOVIB.PF0.VF3        VF     pci_0    iodom1
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0    primary
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0    primary
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0    iodom1

The ldm list-io -l command provides more detailed information about the specified physical function device, /SYS/MB/RISER1/PCIE4/IOVIB.PF0. The maxvfs value shows that the maximum number of virtual functions supported by the physical device is 64. For each virtual function that is associated with the physical function, the output shows the following:

This ldm list-io -l output shows that VF0 and VF1 are assigned to the primary domain and that VF2 and VF3 are assigned to the iodom1 I/O domain.

primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
    maxvfs = 64
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,1]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,2]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,3]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0    iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,4]
Identifying InfiniBand SR-IOV Functions

This section describes how to identify the InfiniBand SR-IOV devices on an Oracle Solaris 11 and an Oracle Solaris 10 system.

Use the ldm list-io -l command to show the Oracle Solaris device path name that is associated with each physical function and virtual function.

primary# ldm list-io -l /SYS/MB/RISER1/PCIE4/IOVIB.PF0
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0]
    maxvfs = 64
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,1]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0    primary
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,2]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,3]
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0    iodom1
[pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0,4]

Oracle Solaris 11:

Use the Oracle Solaris 11 dladm show-phys -L command to match each IP over InfiniBand (IPoIB) instance to its physical card. For example, the following command shows which IPoIB instances use the card in slot PCIE4, which is the same card shown in the previous ldm list-io -l example.

primary# dladm show-phys -L | grep PCIE4
net5              ibp0         PCIE4/PORT1
net6              ibp1         PCIE4/PORT2
net19             ibp8         PCIE4/PORT1
net9              ibp9         PCIE4/PORT2
net18             ibp4         PCIE4/PORT1
net11             ibp5         PCIE4/PORT2

Each InfiniBand host channel adapter (HCA) device has a globally unique ID (GUID). There are also GUIDs for each port (typically there are two ports to an HCA). An InfiniBand HCA GUID uniquely identifies the adapter. The port GUID uniquely identifies each HCA port and plays a role similar to a network device's MAC address. These 16-hexadecimal digit GUIDs are used by InfiniBand management tools and diagnostic tools.

Use the Oracle Solaris 11 dladm show-ib command to obtain GUID information about the InfiniBand SR-IOV devices. Physical functions and virtual functions for the same device have related HCA GUID values. The 11th hexadecimal digit of the HCA GUID shows the relationship between a physical function and its virtual functions. Note that leading zeros are suppressed in the HCAGUID and PORTGUID columns.

For example, physical function PF0 has two virtual functions, VF0 and VF1, which are assigned to the primary domain. The 11th hexadecimal digit of each virtual function is incremented by one from the related physical function. So, if the GUID for the PF0 is 8, the GUIDs for VF0 and VF1 will be 9 and A, respectively.

The following dladm show-ib command output shows that the net5 and net6 links belong to the physical function PF0. The net19 and net9 links belong to VF0 of the same device while the net18 and net11 links belong to VF1.

primary# dladm show-ib
LINK         HCAGUID         PORTGUID        PORT STATE  PKEYS
net6         21280001A17F56  21280001A17F58  2    up     FFFF
net5         21280001A17F56  21280001A17F57  1    up     FFFF
net19        21290001A17F56  14050000000001  1    up     FFFF
net9         21290001A17F56  14050000000008  2    up     FFFF
net18        212A0001A17F56  14050000000002  1    up     FFFF
net11        212A0001A17F56  14050000000009  2    up     FFFF

The device in the following Oracle Solaris 11 dladm show-phys output shows the relationship between the links and the underlying InfiniBand port devices (ibpX).

primary# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
...
net6              Infiniband           up         32000  unknown   ibp1
net5              Infiniband           up         32000  unknown   ibp0
net19             Infiniband           up         32000  unknown   ibp8
net9              Infiniband           up         32000  unknown   ibp9
net18             Infiniband           up         32000  unknown   ibp4
net11             Infiniband           up         32000  unknown   ibp5

Use the ls -l command to show the actual InfiniBand port (IB port) device paths. An IB port device is a child of a device path that is shown in the ldm list-io -l output. A physical function has a one-part unit address such as pciex15b3,673c@0 while virtual functions have a two-part unit address, pciex15b3,1002@0,2. The second part of the unit address is one higher than the virtual function number. (In this case, the second component is 2, so this device is virtual function 1.) The following output shows that /dev/ibp0 is a physical function and /dev/ibp5 is a virtual function.

primary# ls -l /dev/ibp0
lrwxrwxrwx   1 root     root          83 Apr 18 12:02 /dev/ibp0 ->
../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0/hermon@0/ibport@1,0,ipib:ibp0
primary# ls -l /dev/ibp5
lrwxrwxrwx   1 root     root          85 Apr 22 23:29 /dev/ibp5 ->
../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,1002@0,2/hermon@3/ibport@2,0,ipib:ibp5

You can use the OpenFabrics ibv_devices command to view the OpenFabrics device name and the node (HCA) GUID. When virtual functions are present, the Type column indicates whether the function is physical or virtual.

primary# ibv_devices
device                 node GUID            type
------              ----------------        ----
mlx4_4              0002c90300a38910         PF
mlx4_5              0021280001a17f56         PF
mlx4_0              0002cb0300a38910         VF
mlx4_1              0002ca0300a38910         VF
mlx4_2              00212a0001a17f56         VF                        
mlx4_3              0021290001a17f56         VF

Oracle Solaris 10:

In an Oracle Solaris 10 guest I/O domain, use the dladm show-dev command to show each IPoIB instance, which has a name in the form ibdxx.

# dladm show-dev
vnet0          link: up        speed: 0     Mbps       duplex: unknown
ibd0           link: up        speed: 32000 Mbps       duplex: unknown
ibd1           link: up        speed: 32000 Mbps       duplex: unknown
ibd2           link: up        speed: 32000 Mbps       duplex: unknown
ibd3           link: up        speed: 32000 Mbps       duplex: unknown

You can use the ls -l command on the HCA path names in the /devices/ directory to extract an HCA and its HCA GUID.

# ls -l /devices/ib\:[0-9]*
crw-r--r--   1 root     sys       67,  0 Jun 12 16:27 /devices/ib:212B0001A17F56
crw-r--r--   1 root     sys       67,  0 Jun 12 16:27 /devices/ib:212C0001A17F56

The GUIDs in the ibv_devices output (note the 11th hexadecimal digit, "B" and "C" in this case) indicate that these are virtual functions assigned to the Oracle Solaris 10 domain. You can obtain more information about the IPoIB instances by using the ls -l command on the /dev IPoIB path names.

# ls -l /dev/ibd*
lrwxrwxrwx   1 root     other         29 May 23 16:26 /dev/ibd ->
../devices/pseudo/clone@0:ibd
lrwxrwxrwx   1 root     root          89 May 31 10:52 /dev/ibd0 ->
../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,1002@0,3/hermon@0/ibport@1,ffff,ipib:ibd0
lrwxrwxrwx   1 root     root          89 May 31 10:52 /dev/ibd1 ->
../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,1002@0,3/hermon@0/ibport@2,ffff,ipib:ibd1
lrwxrwxrwx   1 root     root          89 Jun 12 18:36 /dev/ibd2 ->
../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,1002@0,4/hermon@1/ibport@1,ffff,ipib:ibd2
lrwxrwxrwx   1 root     root          89 Jun 12 18:36 /dev/ibd3 ->
../devices/pci@400/pci@1/pci@0/pci@0/pciex15b3,1002@0,4/hermon@1/ibport@2,ffff,ipib:ibd3

Each path begins with the device path that is shows in the ldm list-io -l output. The virtual functions such as pciex15b3,1002@0,4 have a two-part unit address where the second part of the unit address is one higher than the virtual function number (in this case, VF3).

The ibport device has a three-part unit address followed by a colon and then the IPoIB device instance name. The first part of the unit address is the port number. The second is the partition key (P-key) hexadecimal value. Note that InfiniBand P-key values are similar to VLANs for Ethernet. The third part is a the string ipib.

The ls -l /dev/ibd3 command output shows that the ibd3 IPoIB instance uses port 2 and P-key value ffff.