Dynamically Reconfiguring IB Devices (cfgadm)

One can configure or unconfigure an IB device from a running system by using the cfgadm CLI only. This command also provides a way to display the IB fabric, manage communication services, and update P_key table databases. For more information, see cfgadm_ib(1M).

The cfgadm CLI manages dynamic reconfiguration, referred to in this guide as DR, of the entire IB fabric as seen by a host. The cfgadm operations are supported on all the IB devices, such as Port, VPPA, HCA_SVC, IOC, and pseudo devices.

The cfgadm command displays information about attachment points (Ap_Ids), which are locations in the system where DR operations can occur. For details on the Ap_Ids that cfgadm supports, see cfgadm_ib.1M. Note that all IB Ap_Ids are shown as connected.

The cfgadm command provides the following IB device status information.

All of the procedures in this section require administrative privileges that are not generally granted to user accounts. For more information, see How to Use Your Assigned Administrative Rights in Oracle Solaris 11.1 Administration: Security Services.

The following sections describe how to dynamically reconfigure (DR) IB devices with the cfgadm command. All of the sample IB device information in these sections has been truncated to focus on relevant information.

Unconfiguring IB Device Considerations

An actual dynamic reconfiguration (DR) of an HCA is beyond the scope of the IB cfgadm plugin. Although DR of an HCA can be achieved by using the plugin of the underlying bus. For example, a PCI based HCA can use the cfgadm_pci command. For more information, see cfgadm_pci(1M).

# cfgadm -c unconfigure ib::rdsv3,0
This operation will suspend activity on the IB device
Continue (yes/no)? yes
cfgadm: Hardware specific failure: unconfigure operation failed ap_id: /devices/ib:fabric::rdsv3,0
 
# cfgadm -c unconfigure PCI-EM0
cfgadm: Component system is busy, try again: unconfigure failed 

# rem_drv rdsv3
Device busy
Cannot unload module: rdsv3
Will be unloaded upon reboot.
 
# init 6

How to Display IB Device Information

You can use the prtconf command to display general information about IB devices. For example:

$ prtconf
.
.
.
    ib, instance #0
        rpcib, instance #0
        rdsib, instance #0
        daplt, instance #0
        rdsv3, instance #0
        sdpib, instance #0
        eibnx, instance #0
        sol_umad, instance #0
        sol_uverbs, instance #0
        iser, instance #0
.
.
.
    pci15b3,673c, instance #0
                 ibport, instance #0
                 ibport, instance #1 

In the above example, pci15b3,673c refers to an IB HCA.

Use the following steps to display specific IB device information.

1. Become an administrator.
2. Display IB fabric information.

   For example:

   # cfgadm -al  
   Ap_Id                       Type         Receptacle     Occupant     Condition
   ib                          IB-Fabric    connected      configured   ok
   hca:21280001A0A478          IB-HCA       connected      configured   ok
   ib::21280001A0A47A,0,ipib   IB-PORT      connected      configured   ok
   ib::21280001A0A479,0,ipib   IB-PORT      connected      configured   ok
   ib::1730000008070,0,hnfs    IB-HCA_SVC   connected      configured   ok
   ib::daplt,0                 IB-PSEUDO    connected      configured   ok
   ib::eibnx,0                 IB-PSEUDO    connected      configured   ok
   ib::iser,0                  IB-PSEUDO    connected      configured   ok
   ib::rdsib,0                 IB-PSEUDO    connected      configured   ok
   ib::rdsv3,0                 IB-PSEUDO    connected      configured   ok
   ib::rpcib,0                 IB-PSEUDO    connected      configured   ok
   ib::sdpib,0                 IB-PSEUDO    connected      configured   ok
   ib::sol_umad,0              IB-PSEUDO    connected      configured   ok
   ib::sol_uverbs,0            IB-PSEUDO    connected      configured   ok

   In the above example output, the components are described as follows:

   Ap_Id ib::21280001A0A47A,0,ipib

   Identifies an IB port device that is connected to port GUID and is bound to the ipib service.

   Ap_ID ib::sdpib,0

   Identifies a pseudo device.

   Ap_ID hca:21280001A0A478

   Identifies an HCA device.

   Ap_ID ib::1730000008070,0,hnfs

   Identifies an IB HCA_SVC device that is bound to the hnfs service.

   Ap_Id ib::ibgen,0

   Identifies a pseudo device.

3. Display specific IB device information.

   For example, for an IB port device:

   # cfgadm -al -s "cols=ap_id:info" ib::21280001A0A47A,0,ipib
   Ap_Id                          Information
   ib::21280001A0A47A,0,ipib      ipib

   For example, for an IB HCA device:

   # cfgadm -al -s "cols=ap_id:info" hca::1730000008070
   Ap_Id                          Information
   hca::1730000008070             VID: 0x15b3, PID: 0x5a44, #ports: 0x2, 
   port1 GUID: 0x1730000008071, port2 GUID: 0x1730000008072

   The preceding output displays the number of ports and their GUIDs.

How to Unconfigure an IB Port, HCA_SVC, or a VPPA Device

Use the following steps if you want to remove an IB Port, HCA_SVC, or a VPPA device from the system.

The example below illustrates how to unconfigure an IB Port device, but the same procedure applies to VPPA and HCA_SVC devices as well.

1. Become an administrator.
2. Unconfigure virtual IB port devices.

   For example:

   # cfgadm -c unconfigure ib::1730000007F51,*0*,ipib
   Unconfigure the device: /devices/ib:fabric::1730000007F51,*0*,ipib
   This operation will suspend activity on the IB device
   Continue (yes/no)? Y

3. Verify that the device is disconnected.

   For example:

   # cfgadm -a ib::1730000007F51,*0*,ipib
   Ap_Id                       Type    Receptacle Occupant     Condition
   ib::1730000007F51,*0*,ipib IB-VPPA  connected  unconfigured unknown

How to Configure a IB Port, HCA_SVC, or a VPPA Device

Use the following steps if you want to configure an IB Port, HCA_SVC, or a VPPA device on the system.

The example below illustrates how to configure a VPPA device, but similar steps can be used to configure an IB Port and HCA_SVC devices as well.

1. Become an administrator.
2. Configure the virtual IB port devices.

   For example:

   # cfgadm -c configure ib::1730000007F51,*0*,ipib

3. Verify that the device is connected.

   For example:

   # cfgadm -a ib::1730000007F51,*0*,ipib   
   Ap_Id                       Type      Receptacle Occupant   Condition
   ib::1730000007F51,*0*,ipib  IB-VPPA   connected  configured ok

   ---

   Note - A cfgadm based configure or unconfigure operation of IB Port and HCA_SVC devices is similar to the preceding examples for an IB VPPA device.

   ---

How to Unconfigure an IB Pseudo Device

Use the following steps if you want to remove an IB pseudo device from the system.

1. Become an administrator.
2. Unconfigure the IB pseudo device.

   For example:

   # cfgadm -c unconfigure ib::ibgen,0
   Unconfigure the device: /devices/ib:fabric::ibgen,0
   This operation will suspend activity on the IB device
   Continue (yes/no)? Y

3. Verify that the device is disconnected.

   # cfgadm -a ib::ibgen,0
   Ap_Id                  Type      Receptacle Occupant     Condition
   ib::ibgen,0            IB-PSEUDO connected  unconfigured unknown

How to Configure an IB Pseudo Device

Use the following steps to configure an IB pseudo device.

1. Become an administrator.
2. Configure the IB pseudo device.

   For example:

   # cfgadm -yc configure ib::ibgen,0

3. Verify that the device is connected.

   For example:

   # cfgadm -a ib::ibgen,0
   Ap_Id                  Type       Receptacle Occupant   Condition
   ib::ibgen,0            IB-PSEUDO  connected  configured ok

How to Display Kernel IB Clients of an HCA

The following IB cfgadm plugin command can be invoked to list kernel IB clients using this HCA. Note that the last column would show a “yes” if a kernel IB client uses another HCA. IB Managers and kernel clients that do not use the HCA are shown with an Ap_Id of “-”.

• Display kernel IB clients of an HCA.

  For example:

  $ cfgadm -x list_clients hca:173000007F50   
  Ap_Id                       IB Client         Alternate HCA
  ib::1730000007F51D0            ibgen              no
  ib::1730000007F51D1            ibgen              no
  ib::1730000007F51,0,ipib       ibd                no
  ib::ibgen,0                    ibgen              no
  -                              ibdm               no
  -                              ibmf               no
  -                              nfs/ib             no

How to Dynamically Reconfigure an HCA With Active EoIB Devices

When a dynamic reconfiguration (DR) operation is attempted on a system and a Connect-X family IB HCA is in use with active EoIB interfaces, (for example, EoIB datalinks are plumbed or VNICs are created over the datalinks on the HCA), the DR operation succeeds if either of the following conditions is true:

• This is the only Connect-X family HCA on the system

• Multiple Connect-X HCAs exist on the system, but this adapter is the last one to be unconfigured

Under any other condition, a dynamic reconfiguration (DR) operation on an IB HCA with active EoIB interfaces fails with a cfgadm error message.

If the DR operation fails, you must unplumb the active EoIB interfaces and delete any VNICs on the datalink and retry the DR operation. In addition, in such a case, if a Connect-X family HCA is replaced in the same slot after the DR operation and configured again using the cfgadm or hotplug commands, you must replumb the EoIB datalinks that were unplumbed earlier and recreate any VNICs that were previously deleted.

Note that a functioning EoIB interface requires that the HCA port GUID be listed in the appropriate config file on the Sun Network QDR Infiniband Gateway Switch. So, after a hot-removal and before a re-insertion, you will need to update the new HCA port GUID information in the Gateway Switch config file. For information on how to update the configuration, see the Sun Network QDR Infiniband Gateway Switch Administration Guide.

1. Become an administrator on the system with multiple IB HCAs.
2. Attempt to unconfigure the attachment point associated with the EoIB datalink.

   # cfgadm -c unconfigure PCI-EM0
   cfgadm: Component system is busy, try again: unconfigure failed

3. Determine the EoIB interface that is preventing the unconfigure operation.

   # dmesg | tail | grep 'failing HCA detach'
   Aug 23 12:37:20  eoib: [ID 530795 kern.warning] WARNING: eoib0 still in use, 
   failing HCA detach

4. Determine if eoib0 is in use because an IP interface exists on it.

   # ipadm show-if eoib0
   ipadm: cannot get information for interface(s): No such interface

5. If no IP interface over eoib0 exists, check to see if any VNIC is present that is causing eoib0 to be busy.

   # dladm show-vnic
       LINK                OVER         SPEED  MACADDRESS        MACADDRTYPE       VID
       evnic0              eoib0        10000  2:8:20:e5:56:99   random            0

6. Delete the VNIC over eoib0.

   # dladm delete-vnic evnic0

7. Retry the unconfigure operation.

   # cfgadm -c unconfigure PCI-EM0

How to Reconfigure and Restore an EoIB Interface After Hot Removal

1. Become an administrator on the Sun Network QDR Gateway Switch.
2. Edit the BXM config file on the Sun Network QDR Gateway Switch and replace all occurrences of the old-HCA port GUIDs with the port GUIDs of the new HCA replacing it.
3. Restart BXM.
4. Become an administrator on the Oracle Solaris system.
5. Hot insert the new HCA on the Solaris system and use the cfgadm command to reconfigure the attachment point originally associated with the EoIB datalink.
6. Confirm the restoration of the original EoIB datalink.

   # dladm show-link | grep eoib0
   eoib0               phys      1500   unknown       --

7. If any VNICs had to be explicitly deleted during a previous unconfigure operation, re-create them now.

   # dladm create-vnic -l eoib0 evnic0

8. Confirm the presence of the VNIC over eoib0.

   # dladm show-vnic
   LINK         OVER     SPEED  MACADDRESS        MACADDRTYPE       VID
   evnic0       eoib0    10000  2:8:20:e5:56:99   random              0

9. If any IP addresses had to be explicitly deleted on eoib0 during a previous unconfigure operation, recreate them by using the ipadm command.

Configuring an IB HCA

Invoke the bus-specific cfgadm plugin to configure the HCA. The exact details are beyond the scope of this chapter.

How to Update the IB P_key Tables

If the P_key table information of an HCA ports changes, for example, additional P_keys are enabled or disabled, InfiniBand Transport Framework (IBTF) and IBDM need to be notified so that their internal P_key databases are updated. The cfgadm command helps update the P_key databases of IBTF and IBDM. For more information, see ibtl(7D) and ibdm(7D).

1. Become an administrator.
2. Update the P_key tables.

   For example:

   # cfgadm -x update_pkey_tbls -y ib

How to Display IB Communication Services

Use the following steps to display the communication services that are currently in use by the IBTF.

1. Become an administrator.
2. Display IB communication services.

   For example:

   # cfgadm -x list_services ib
   Port communication services:
                   srp
   VPPA communication services:
                   ibd
   HCA_SVC communication services:
                   hnfs

How to Add a VPPA Communication Service

Use the following steps to add a new VPPA communication service.

Similar steps can be used to add a new HCA_SVC or a port communication service.

1. Become an administrator.
2. Add a new VPPA communication service.

   For example:

   # cfgadm -o comm=vppa,service=new -x add_service ib

3. Verify that the new service has been added.

   For example:

   # cfgadm -x list_services ib
   Port communication services:
                   srp
   VPPA communication services:
                   ibd
                   new
   HCA_SVC communication services:
                   nfs_service

How to Remove an Existing IB Port, HCA_SVC, or a VPPA Communication Service

Use the following steps to delete an existing IB Port, HCA_SVC, or a VPPA communication service.

1. Become an administrator.
2. Remove a VPPA communication service.

   For example:

   # cfgadm -o comm=vppa,service=new -x delete_service ib

3. Verify that the communication service has been removed.

   For example:

   # cfgadm -x list_services ib
   Port communication services:
                   srp
   VPPA communication services:
                   ibd
   HCA_SVC communication services:
                   hnfs