Go to main content

Oracle® VM Server for SPARC 3.6 Administration Guide

Exit Print View

Updated: September 2019
 
 

Adding and Removing InfiniBand Virtual Functions on I/O Domains

How to Add an InfiniBand Virtual Function to an I/O Domain

This procedure describes how to add an InfiniBand SR-IOV virtual function to an I/O domain.

  1. Stop the I/O domain.
    primary# ldm stop-domain domain-name
  2. Add one or more virtual functions to the I/O domain.

    vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the pseudonym name. domain-name specifies the name of the domain to which you add the virtual function. The specified I/O domain must be in the inactive or bound state.

    primary# ldm add-io vf-name domain-name
  3. Start the I/O domain.
    primary# ldm start-domain domain-name
Example 20  Adding an InfiniBand Virtual Function

The following example shows how to add the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 virtual function to the iodom1 I/O domain.

First, identify the virtual function that you want to assign.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN  STATUS
----                                      ----   ---      ------  ------
/SYS/PM0/CMP0/PEX                         BUS    pci_0    primary IOV
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0

To add a virtual function to an I/O domain, it must be unassigned. The DOMAIN column indicates the name of the domain to which the virtual function is assigned. In this case, the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 is not assigned to a domain.

To add a virtual function to a domain, the domain must be in the inactive or bound state.

primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.2%  0.2%  56m
iodom1           active     -n----  5000    8     8G        33%   33%  25m

The ldm list-domain output shows that the iodom1 I/O domain is active, so it must be stopped.

primary# ldm stop iodom1
LDom iodom1 stopped
primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.0%  0.0%  57m
iodom1           bound      ------  5000    8     8G

Now you can add the virtual function to the I/O domain.

primary# ldm add-io /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 iodom1
primary# ldm list-io
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1

Note that you can add more than one virtual function while an I/O domain is stopped. For example, you might add other unassigned virtual functions such as /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3 to iodom1. After you add the virtual functions, you can restart the I/O domain.

primary# ldm start-domain iodom1
LDom iodom1 started
primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      1.0%  1.0%  1h 18m
iodom1           active     -n----  5000    8     8G        36%   36%  1m

How to Remove an InfiniBand Virtual Function From an I/O Domain

This procedure describes how to remove an InfiniBand SR-IOV virtual function from an I/O domain.

  1. Stop the I/O domain.
    primary# ldm stop-domain domain-name
  2. Remove one or more virtual functions from the I/O domain.

    vf-name is the pseudonym name or the path name of the virtual function. The recommended practice is to use the device pseudonym. domain-name specifies the name of the domain from which you remove the virtual function. The specified I/O domain must be in the inactive or bound state.


    Note - Before removing the virtual function from the I/O domain, ensure that it is not critical for booting that domain.
    primary# ldm remove-io vf-name domain-name
  3. Start the I/O domain.
    primary# ldm start-domain domain-name
Example 21  Removing an InfiniBand Virtual Function

The following example shows how to remove the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 virtual function from the iodom1 I/O domain.

First, identify the virtual function that you want to remove.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN  STATUS
----                                      ----   ---      ------  ------
/SYS/PM0/CMP0/PEX                         BUS    pci_0    primary IOV
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0            PF     pci_0    primary
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF0        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF1        VF     pci_0
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0    iodom1
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3        VF     pci_0    iodom1

The DOMAIN column shows the name of the domain to which the virtual function is assigned. The /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 virtual function is assigned to iodom1.

To remove a virtual function from an I/O domain, the domain must be inactive or bound state. Use the ldm list-domain command to determine the state of the domain.

primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.3%  0.3%  29m
iodom1           active     -n----  5000    8     8G        17%   17%  11m

In this case, the iodom1 domain is active and so must be stopped.

primary# ldm stop iodom1
LDOM iodom1 stopped
primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.0%  0.0%  31m
iodom1           bound      ------  5000    8     8G

Now you can remove the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 virtual function from iodom1.

primary# ldm remove-io /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2 iodom1
primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN STATUS
----                                      ----   ---      ------ ------
...
/SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF2        VF     pci_0
...

Note that the DOMAIN column for the virtual function is now empty.

You can remove more than one virtual function while an I/O domain is stopped. In this example, you could also remove the /SYS/MB/RISER1/PCIE4/IOVIB.PF0.VF3 virtual function. After you remove the virtual functions, you can restart the I/O domain.

primary# ldm start-domain iodom1
LDom iodom1 started
primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    64G      0.3%  0.3%  39m
iodom1           active     -n----  5000    8     8G       9.4%  9.4%  5s