Chapter 9 Using InfiniBand Devices (Overview/Tasks)

InfiniBand (IB) is a new I/O technology based on switch fabrics introduced in the Solaris 10 release. It provides high bandwidth, low latency interconnect for attaching I/O devices to hosts and for host-to-host communication.

This is a list of the overview information in this chapter.

• Overview of InfiniBand Devices

• Dynamically Reconfiguring IB Devices (cfgadm)

For information on the procedures associated with using IB devices, see the following:

• Dynamically Reconfiguring IB Devices (Task Map)

• Using the uDAPL Application Interface With InfiniBand Devices

For general information about dynamic reconfiguration and hot-plugging, see Chapter 6, Dynamically Configuring Devices (Tasks).

Overview of InfiniBand Devices

IB devices are managed by the Solaris IB nexus driver. This driver supports 5 types of devices:

• IB Port devices

• IB virtual physical point of attachment (VPPA) devices

• IB HCA service (HCA_SVC) devices

• Pseudo devices

• I/O controller (IOC) devices

The IB nexus driver queries the Solaris IB Device Manager (IBDM) for services, referred in this guide as communication services, to enumerate the IB Port, HCA_SVC, and IB VPPA devices.

The Port devices bind a communication service to a given port# of a Host Channel Adapter (HCA). The VPPA devices bind a communication service to a port#, p_key# combination instead. The HCA_SVC devices bind a communication service to a given HCA. Note that the Port devices and the HCA_SVC devices always use a p_key (partition key) whose value is zero. The Port, HCA_SVC, and VPPA devices are children of the HCA and are enumerated through the ib.conf file. For more information, see ib(7D).

The IOC devices are children of the IB nexus driver and are part of an I/O unit. The pseudo devices are also children of the IB nexus driver and refer to all other devices that provide their own configuration files to enumerate. For more information, see ib(4).

The possible IB device tree path name(s) are listed in the following table.

Note that the IB HCA_SVC devices have zero as the port# and the p_key.

The IB components in the preceding table are described as follows:

<services>

Is a communication service. For example, ipib is the communication service used by the ibd kernel client driver.

<p_key>

Is the partition key value being used.

<port>

Is the port number.

<unit-address>

Refers to IB kernel client driver's property by this name specified in its driver.conf file. For more information, see driver.conf(4).

Dynamically Reconfiguring IB Devices (Task Map)

Dynamically Reconfiguring IB Devices (cfgadm)

One can configure or unconfigure an IB device from a running system by using the cfgadm CLI only. This command also provides a way to display the IB fabric, manage communication services, and update p_key table databases. For more information, see cfgadm_ib(1M).

The cfgadm CLI manages dynamic reconfiguration, referred to in this guide as DR, of the entire IB fabric as seen by a host. The cfgadm operations are supported on all the IB devices, such as Port, VPPA, HCA_SVC, IOC, and pseudo devices.

The cfgadm command displays information about attachment points (Ap_Ids), which are locations in the system where DR operations can occur. For details on the Ap_Ids that cfgadm supports, see cfgadm_ib.1M. Note that all IB Ap_Ids are shown as connected.

The cfgadm command provides the following IB device status information.

The rpcib driver module that provides NFS over RDMA functionality uses only one HCA in the system. If the system has multiple HCAs, the rpcib driver uses only one of the HCAs present on the system. NFS over RDMA is not usable if the HCA that it is using is unconfigured. Use the following steps to enable NFS over RDMA.

1. Become superuser.

2. Unconfigure the rpcib module.

   cfgadm -c unconfigure ib::rpcib,0

   For more information, see cfgadm_ib(1M).

3. Reconfigure the rpcib module.

   cfgadm -c configure ib::rpcib,0

The following sections describe how to dynamically reconfigure (DR) IB devices with the cfgadm command. All of the sample IB device information in these sections has been truncated to focus on relevant information.

How to Display IB Device Information

You can use the prtconf command to display general information about IB devices. For example:

$ prtconf
    pci, instance #0
        pci15b3,5a44, instance #0
             ibport, instance #253
             ibport, instance #254
             ibport, instance #255
    .
    .
    .
    ib, instance #0
        ioc, instance #243
        ioc, instance #244
        ioc, instance #245
        ioc, instance #246
        ioc, instance #247
        ioc, instance #248
        ibgen, instance #249

In the preceding example, pci15b3,5a44 refers to an IB HCA.

Use the following steps to display specific IB device information.

1. Become superuser.

2. Display IB fabric information.

   For example:

   # cfgadm -a Ap_Id Type Receptacle Occupant Condition ib IB-Fabric connected configured ok hca:1730000008070 IB-HCA connected configured ok ib::1730000007F5198 IB-IOC connected configured ok ib::1730000007F5199 IB-IOC connected configured ok ib::1730000008070,0,hnfs IB-HCA_SVC connected configured ok ib::1730000008071,0,sdp IB-PORT connected configured ok ib::1730000008072,0,sdp IB-PORT connected configured ok ib::1730000008071,8001,ipib IB-VPPA connected configured ok ib::1730000008072,8001,ipib IB-VPPA connected configured ok ib::ibgen,0 IB-PSEUDO connected configured ok #

   In the preceding example output, the components are described as follows:

   Ap_Id ib::1730000008072,0,sdp

   Identifies an IB Port device that is connected to port 2 and is bound to the sdp service.

   Ap_Id ib::1730000008072,8001,ipib

   Identifies an IB VPPA device that is connected to port 2, using a p_key value of 0x8001, and is bound to the ibd service.

   Ap_Id ib:: 1730000008070,0,hnfs

   Identifies an IB HCA_SVC device bound to the hnfs service.

   Ap_Id ib::1730000007F5198

   Identifies an IOC device.

   Ap_Id ib::ibgen,0

   Identifies a pseudo device.

3. Display specific IB device information.

   For example, for an IB VPPA device:

   # cfgadm -al -s "cols=ap_id:info" ib::1730000008072,8001,ipib Ap_Id Information ib::1730000008072,8001,ipib ipib #

   For example, for an IB HCA device:

   # cfgadm -al -s "cols=ap_id:info" hca::1730000008070 Ap_Id Information hca::1730000008070 VID: 0x15b3, PID: 0x5a44, #ports: 0x2, port1 GUID: 0x1730000008071, port2 GUID: 0x1730000008072 #

   The preceding output displays the number of ports and their GUIDs.

How to Unconfigure an IOC Device

You can unconfigure an IB device that is still physically connected to the system, but a driver will never attach to it.

1. Become superuser.

2. Unconfigure the IB device.

   For example:

   # cfgadm -c unconfigure ib::1730000007F5198 Unconfigure the device: /devices/ib:fabric::1730000007F5198 This operation will suspend activity on the IB device Continue (yes/no)? y #

3. Verify that the device is unconfigured.

   For example:

   # cfgadm -a ib::1730000007F5198 ib::1730000007F5198 IB-IOC connected unconfigured unknown #

How to Configure an IOC Device

1. Become superuser.

2. Configure a IB device.

   For example:

   # cfgadm -yc configure ib::1730000007F5198

3. Verify that the IB device is configured.

   For example:

   # cfgadm -al ib::1730000007F5198 Ap_Id Type Receptacle Occupant Condition ib::1730000007F5198 IB-IOC connected configured ok

How to Unconfigure an IB Port, HCA_SVC, or a VPPA Device

Use the following steps if you want to remove an IB Port, HCA_SVC, or a VPPA device from the system.

The example below illustrates how to unconfigure a VPPA device, but the same procedure applies to Port and HCA_SVC devices as well.

1. Become superuser.

2. Unconfigure the IB VPPA device.

   For example:

   # cfgadm -c unconfigure ib::1730000007F51,8001,ipib Unconfigure the device: /devices/ib:fabric::1730000007F51,8001,ipib This operation will suspend activity on the IB device Continue (yes/no)? Y #

3. Verify that the device is disconnected.

   For example:

   # cfgadm -a ib::1730000007F51,8001,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,8001,ipib IB-VPPA connected unconfigured unknown #

How to Configure a IB Port, HCA_SVC, or a VPPA Device

Use the following steps if you want to configure an IB Port, HCA_SVC, or a VPPA device on the system.

The example below illustrates how to configure a VPPA device, but similar steps can be used to configure Port and HCA_SVC devices as well.

1. Become superuser.

2. Configure the IB VPPA device.

   For example:

   # cfgadm -c configure ib::1730000007F51,8001,ipib

3. Verify that the device is connected.

   For example:

   # cfgadm -a ib::1730000007F51,8001,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,8001,ipib IB-VPPA connected configured ok

   ---

   Note –

   A cfgadm based configure or unconfigure operation of IB Port and HCA_SVC devices is similar to the preceding examples for an IB VPPA device.

   ---

How to Unconfigure an IB Pseudo Device

Use the following steps if you want to remove an IB pseudo device from the system.

1. Become superuser.

2. Unconfigure the IB pseudo device.

   For example:

   # cfgadm -c unconfigure ib::ibgen,0 Unconfigure the device: /devices/ib:fabric::ibgen,0 This operation will suspend activity on the IB device Continue (yes/no)? Y #

3. Verify that the device is disconnected.

   # cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected unconfigured unknown

How to Configure an IB Pseudo Device

Use the following steps to configure an IB pseudo device.

1. Become superuser.

2. Configure the IB pseudo device.

   For example:

   # cfgadm -yc configure ib::ibgen,0

3. Verify that the device is connected.

   For example:

   # cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected configured ok

How to Display Kernel IB Clients of an HCA

The following IB cfgadm plugin command can be invoked to list kernel IB clients using this HCA. Note that the last column would show a “yes” if a kernel IB client uses another HCA. IB Managers and kernel clients that do not use the HCA are shown with an Ap_Id of “-”.

1. Display kernel IB clients of an HCA.

   For example:

   $ cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,8001,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no $

How to Unconfigure IB Devices Connected to an HCA

An actual DR of an HCA is beyond the scope of the IB cfgadm plugin. Although DR of an HCA can be achieved by using the plugin of the underlying bus. For example, a PCI based HCA can use the cfgadm_pci command. For more information, see cfgadm_pci(1M).

However, the IB cfgadm plugin assists in the HCA DR by listing its kernel IB clients as illustrated in steps below.

1. Become superuser.

2. List the kernel IB clients of the HCA.

   For example:

   # cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,8001,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no

3. Unconfigure kernel IB clients, such as Port, VPPA, HCA_SVC, or IOC devices, that do not have alternate HCAs present.

   For example:

   # cfgadm -x unconfig_clients hca:1730000008070 Unconfigure Clients of HCA /devices/ib:1730000008070 This operation will unconfigure IB clients of this HCA Continue (yes/no)? y

4. Verify that the kernel IB clients of the HCA are unconfigured.

   # cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA - ibdm no - ibmf no - nfs/ib no #

Configuring an IB HCA

Invoke the bus-specific cfgadm plugin to configure the HCA. The exact details are beyond the scope of this chapter.

How to Update the IB p_key Tables

If the p_key table information of an HCA ports changes, for example, additional p_keys are enabled or disabled, InfiniBand Transport Framework (IBTF) and IBDM need to be notified so that their internal p_key databases are updated. The cfgadm command helps update the p_key databases of IBTF and IBDM. For more information, see ibtl(7D) and ibdm(7D).

1. Become superuser.

2. Update the p_key tables.

   For example:

   # cfgadm -x update_pkey_tbls -y ib

How to Display IB Communication Services

Use the following steps to display the communication services that are currently in use by the IBTF.

1. Become superuser.

2. Display IB communication services.

   For example:

   # cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs

How to Add a VPPA Communication Service

Use the following steps to add a new VPPA communication service.

Similar steps can be used to add a new HCA_SVC or a port communication service.

1. Become superuser.

2. Add a new VPPA communication service.

   For example:

   # cfgadm -o comm=vppa,service=new -x add_service ib

3. Verify that the new service has been added.

   For example:

   # cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd new HCA_SVC communication services: nfs_service #

How to Remove an Existing IB Port, HCA_SVC, or a VPPA Communication Service

Use the following steps to delete an existing IB Port, HCA_SVC, or a VPPA communication service.

1. Become superuser.

2. Remove a VPPA communication service.

   For example:

   # cfgadm -o comm=vppa,service=new -x delete_service ib

3. Verify that the communication service has been removed.

   For example:

   # cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs #

How to Update an IOC Configuration

Use the following steps to update properties of all the IOC device nodes or for a particular IOC Ap_Id. The properties that can get updated are as follows:

• port-list

• port-entries

• service-id

• service-name

For more information on these properties, see ib(7D).

Note that these properties may not get updated if there is no configuration change. The following example describes how to update a particular IOC's configuration. If you need to update the configuration of all the IOCs, then specify the static ib Ap_Id instead of the particular IOC Ap_Id.

1. Become superuser.

2. Update the configuration of an IOC.

   For example:

   # cfgadm -x update_ioc_conf ib::1730000007F5198 This operation can update properties of IOC devices. Continue (yes/no)? y #

3. Verify that the properties have been updated by running prtconf -v.

Using the uDAPL Application Interface With InfiniBand Devices

User Direct Access Programming Library (uDAPL) is a standard API that promotes data center application data messaging performance, scalability, and reliability over Remote Direct Memory Access (RDMA) capable interconnects such as InfiniBand. The uDAPL interface is defined by the DAT collaborative. For more information about the DAT collaborative, go to the following site:

http://www.datcollaborative.org

The Solaris release provides the following uDAPL features:

• A standard DAT registry library, libdat. For more information, see libdat(3LIB).

• A standard service provider registration file, dat.conf. For more information, see dat.conf(4).

• Support for multiple service providers so that each provider specifies their own uDAPL library path, version number, and so on, in their own service_provider.conf file. For more information, see, service_provider.conf(4).

• An administrative tool, the datadm command, to configure dat.conf. For more information, see datadm(1M).

• A new resource control property, project.max-device-locked-memory, to regulate the amount of locked down physical memory.

• A naming scheme that uses either IPv4 or IPv6 addresses that leverage the IP infrastructure, such as ARP in IPv4 and neighbor discovery in IPv6, for address resolution. The Solaris uDAPL Interface Adapter directly maps to an IPoIB device instance.

• Support for the standard Address Translation Scheme that is used by the DAT collaborative community.

• A uDAPL service provider library to support the Mellanox Tavor Host Channel Adapter with automatic registration to the dat.conf registration file.

• Supports both SPARC platform and x86 platforms.

How to Enable uDAPL

1. Become superuser.

2. Confirm that the following packages are installed. Or, install them, if needed.

   • SUNWib – Sun InfiniBand Framework

   • SUNWtavor – Sun Tavor HCA Driver

   • SUNWipoib – Sun IP over InfiniBand

   • SUNWudaplr – Direct Access Transport (DAT) registry package (root)

   • SUNWudaplu – Direct Access Transport (DAT) registry packages (usr)

   • SUNWudapltr – Service Provider for Tavor packages (root)

   • SUNWudapltu – Service Provider for Tavor packages (usr)

3. Select one of the following to plumb the IPoIB interfaces.

   • Manually plumb the interfaces with the ifconfig and datadm commands.

     For example:

     # ifconfig ibd1 plumb # ifconfig ibd1 192.168.0.1/24 up # datadm -a /usr/share/dat/SUNWudaplt.conf

   • Automatically plumb the interfaces by doing the following:

     • Create the following file with the appropriate IP address.

       /etc/hostname.ibd1

     • Reboot the system.

Updating the DAT Static Registry

You can use the datadm command to maintain the DAT static registry, the dat.conf file. For more information about this file, see dat.conf(4).

The datadm command can also be used to register or unregister a service provider to the dat.conf file. For more information, see datadm(1M).

When IPoIB interface adapters are added or removed, run the datadm command to update the dat.conf file to reflect the current state of the system. A new set of interface adapters for all the service providers that are currently installed will be regenerated.

How to Update the DAT Static Registry

1. Become superuser.

2. Update the DAT static registry after you add or remove IPoIP interface adapters from the system.

   # datadm -u

3. Display the updated DAT static registry.

   # datadm

How to Register a Service Provider in the DAT Static Registry

1. Become superuser.

2. Update the DAT static registry after you add Sun's service provider for the Mellanox Tavor Host Channel Adapter.

   # datadm -a /usr/share/dat/SUNWudaplt.conf

3. Display the updated DAT static registry.

   # datadm -v

How to Unregister a Service Provider from the DAT Static Registry

1. Become superuser.

2. Update the DAT static registry after you remove Sun's service provider for the Mellanox Tavor Host Channel Adapter from the system.

   # datadm -r /usr/share/dat/SUNWudaplt.conf

3. Display the updated DAT static registry.

   # datadm -v