Go to main content

Oracle® VM Server for SPARC 3.6 Administration Guide

Exit Print View

Updated: September 2019
 
 

Non-primary Root Domain Examples

The following examples describe how to enable I/O virtualization for a PCIe bus, manage direct I/O devices on non-primary root domains, and manage SR-IOV virtual functions on non-primary root domains.

Enabling I/O Virtualization for a PCIe Bus

The following example shows how to enable I/O virtualization by using the ldm add-io and ldm set-io commands.

The following SPARC T4-2 I/O configuration shows that bus pci_1 already has been removed from the primary domain.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/PM0/CMP0/NIU_CORE                    NIU    niu_0    primary
/SYS/PM0/CMP1/NIU_CORE                    NIU    niu_1    primary
/SYS/PM0/CMP0/PXE                         BUS    pci_0    primary
/SYS/PM0/CMP1/PXE                         BUS    pci_1    primary
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1             UNK      
/SYS/MB/PCIE3                             PCIE   pci_1             UNK      
/SYS/MB/PCIE5                             PCIE   pci_1             UNK      
/SYS/MB/PCIE7                             PCIE   pci_1             UNK      
/SYS/MB/PCIE9                             PCIE   pci_1             UNK      
/SYS/MB/NET2                              PCIE   pci_1             UNK      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary

The following listing shows that the guest domains are in the bound state:

primary# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    8     8G       0.6%  0.6%  8m
rootdom1         bound      ------  5000    8     4G                   
ldg2             bound      ------  5001    8     4G                   
ldg3             bound      ------  5002    8     4G

The following ldm add-io command adds the pci_1 bus to the rootdom1 domain with I/O virtualization enabled for that bus. The ldm start-domain command starts the rootdom1 domain.

primary# ldm add-io iov=on pci_1 rootdom1
primary# ldm start-domain rootdom1
LDom rootdom1 started

If a specified PCIe bus is assigned already to a root domain, use the ldm set-io command to enable I/O virtualization.

primary# ldm start-reconf rootdom1
primary# ldm set-io iov=on pci_1
primary# ldm stop-domain -r rootdom1

The root domain must be running its OS before you can configure the I/O devices. Connect to the console of the rootdom1 guest domain and then boot the OS of the rootdom1 root domain if your guest domains are not already set to autoboot.

primary# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connecting to console "rootdom1" in group "rootdom1" ....
Press ~? for control options ..
ok> boot
...
primary# 

The following command shows that the pci_1 PCIe bus and its children are now owned by the rootdom1 root domain.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/PM0/CMP0/PXE                         BUS    pci_0    primary
/SYS/PM0/CMP1/PXE                         BUS    pci_1    primary
/SYS/PM0/CMP0/NIU_CORE                    NIU    niu_0    primary
/SYS/PM0/CMP1/NIU_CORE                    NIU    niu_1    primary
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    rootdom1     OCC
/SYS/MB/PCIE3                             PCIE   pci_1    rootdom1     OCC
/SYS/MB/PCIE5                             PCIE   pci_1    rootdom1     OCC
/SYS/MB/PCIE7                             PCIE   pci_1    rootdom1     OCC
/SYS/MB/PCIE9                             PCIE   pci_1    rootdom1     EMP
/SYS/MB/NET2                              PCIE   pci_1    rootdom1     OCC
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    rootdom1
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    rootdom1
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    rootdom1
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    rootdom1

Managing Direct I/O Devices on Non-primary Root Domains

The following example shows how to manage direct I/O devices on non-primary root domains.

The following command produces an error because it attempts to remove a slot from the root domain while it is still active:

primary# ldm remove-io /SYS/MB/PCIE7 ldg1
Dynamic I/O operations on PCIe slots are not supported.
Use start-reconf command to trigger delayed reconfiguration and make I/O
changes statically.

The following command shows the correct method of removing a slot by first initiating a delayed reconfiguration on the root domain.

primary# ldm start-reconf ldg1
Initiating a delayed reconfiguration operation on the ldg1 domain.
All configuration changes for other domains are disabled until the ldg1
domain reboots, at which time the new configuration for the ldg1 domain
will also take effect.
primary# ldm remove-io /SYS/MB/PCIE7 ldg1
------------------------------------------------------------------------------
Notice: The ldg1 domain is in the process of a delayed reconfiguration.
Any changes made to the ldg1 domain will only take effect after it reboots.
------------------------------------------------------------------------------
primary# ldm stop-domain -r ldg1

The following ldm list-io command verifies that the /SYS/MB/PCIE7 slot is no longer on the root domain.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/PM0/CMP0/PXE                         BUS    pci_0    primary
/SYS/PM0/CMP1/PXE                         BUS    pci_1    primary
/SYS/PM0/CMP0/NIU_CORE                    NIU    niu_0    primary
/SYS/PM0/CMP1/NIU_CORE                    NIU    niu_1    primary
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE3                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE5                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE7                             PCIE   pci_1             OCC
/SYS/MB/PCIE9                             PCIE   pci_1    ldg1     EMP      
/SYS/MB/NET2                              PCIE   pci_1    ldg1     OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    ldg1              
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    ldg1              
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    ldg1              
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    ldg1

The following commands assign the /SYS/MB/PCIE7 slot to the ldg2 domain. The ldm start-domain command starts the ldg2 domain.

primary# ldm add-io /SYS/MB/PCIE7 ldg2
primary# ldm start-domain ldg2
LDom ldg2 started

Managing SR-IOV Virtual Functions on Non-primary Root Domains

These commands create two virtual functions from each of the two physical functions that belong to the non-primary root domain.

primary# ldm create-vf /SYS/MB/PCIE5/IOVNET.PF0
Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF0
primary# ldm create-vf /SYS/MB/PCIE5/IOVNET.PF0
Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF1
primary# ldm create-vf /SYS/MB/NET2/IOVNET.PF1
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF0
primary# ldm create-vf /SYS/MB/NET2/IOVNET.PF1
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF1

You can also use the –n option to create the two virtual functions by using the following two commands:

primary# ldm create-vf -n 2 /SYS/MB/PCIE5/IOVNET.PF0
Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF0
Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF1
primary# ldm create-vf -n 2 /SYS/MB/NET2/IOVNET.PF1
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF0
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF1

If you were unable to dynamically create the virtual functions on a given physical function, initiate a delayed reconfiguration to create them statically.

primary# ldm start-reconf ldg1
primary# ldm create-vf /SYS/MB/PCIE5/IOVNET.PF0
Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF0
primary# ldm create-vf /SYS/MB/PCIE5/IOVNET.PF0
Created new vf: /SYS/MB/PCIE5/IOVNET.PF0.VF1
primary# ldm create-vf /SYS/MB/NET2/IOVNET.PF1
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF0
primary# ldm create-vf /SYS/MB/NET2/IOVNET.PF1
Created new vf: /SYS/MB/NET2/IOVNET.PF1.VF1

Then reboot the root domain, ldg1, to effect the changes.

primary# ldm stop-domain -r ldg1

The following command shows the new virtual functions.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/PM0/CMP0/PXE                         BUS    pci_0    primary  IOV
/SYS/PM0/CMP1/PXE                         BUS    pci_1    ldg1     IOV
/SYS/PM0/CMP0/NIU_CORE                    NIU    niu_0    primary
/SYS/PM0/CMP1/NIU_CORE                    NIU    niu_1    primary
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE3                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE5                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    ldg2     OCC      
/SYS/MB/PCIE9                             PCIE   pci_1    ldg1     EMP      
/SYS/MB/NET2                              PCIE   pci_1    ldg1     OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    ldg1              
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    ldg1              
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    ldg1              
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    ldg1              
/SYS/MB/PCIE5/IOVNET.PF0.VF0              VF     pci_1
/SYS/MB/PCIE5/IOVNET.PF0.VF1              VF     pci_1
/SYS/MB/NET2/IOVNET.PF1.VF0               VF     pci_1
/SYS/MB/NET2/IOVNET.PF1.VF1               VF     pci_1

The following command dynamically adds the /SYS/MB/PCIE5/IOVNET.PF0.VF1 virtual function to the ldg1 non-primary root domain:

primary# ldm add-io /SYS/MB/PCIE5/IOVNET.PF0.VF1 ldg1

The following command dynamically adds the /SYS/MB/NET2/IOVNET.PF1.VF0 virtual function to the ldg2 domain:

primary# ldm add-io /SYS/MB/NET2/IOVNET.PF1.VF0 ldg2

The following command adds the /SYS/MB/NET2/IOVNET.PF1.VF1 virtual function to the bound ldg3 domain:

primary# ldm add-io /SYS/MB/NET2/IOVNET.PF1.VF1 ldg3
primary# ldm start-domain ldg3
LDom ldg3 started

Connect to the console of the ldg3 domain and then boot its OS.

The following output shows that all the assignments appear as expected. One virtual function is unassigned so it can be assigned dynamically to the ldg1, ldg2, or ldg3 domain.

primary# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
/SYS/PM0/CMP0/PXE                         BUS    pci_0    primary  IOV
/SYS/PM0/CMP1/PXE                         BUS    pci_1    ldg1     IOV
/SYS/PM0/CMP0/NIU_CORE                    NIU    niu_0    primary
/SYS/PM0/CMP1/NIU_CORE                    NIU    niu_1    primary
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE3                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE5                             PCIE   pci_1    ldg1     OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    ldg2     OCC      
/SYS/MB/PCIE9                             PCIE   pci_1    ldg1     EMP      
/SYS/MB/NET2                              PCIE   pci_1    ldg1     OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    ldg1              
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    ldg1              
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    ldg1              
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    ldg1              
/SYS/MB/PCIE5/IOVNET.PF0.VF0              VF     pci_1                      
/SYS/MB/PCIE5/IOVNET.PF0.VF1              VF     pci_1    ldg1              
/SYS/MB/NET2/IOVNET.PF1.VF0               VF     pci_1    ldg2              
/SYS/MB/NET2/IOVNET.PF1.VF1               VF     pci_1    ldg3