One can configure or unconfigure an IB device from a running system by using the cfgadm CLI only. This command also provides a way to display the IB fabric, manage communication services, and update p_key table databases. For more information, see cfgadm_ib(1M).
The cfgadm CLI manages dynamic reconfiguration, referred to in this guide as DR, of the entire IB fabric as seen by a host. The cfgadm operations are supported on all the IB devices, such as Port, VPPA, HCA_SVC, IOC, and pseudo devices.
The cfgadm command displays information about attachment points (Ap_Ids), which are locations in the system where DR operations can occur. For details on the Ap_Ids that cfgadm supports, see cfgadm_ib.1M. Note that all IB Ap_Ids are shown as connected.
The cfgadm command provides the following IB device status information.
Receptacle State |
Description |
---|---|
connected/configured/ok |
The device is connected and available. The devinfo node is present. |
connected/unconfigured/unknown |
The device is unavailable and no devinfo node or device driver exists for this device. Or, the device was never configured for use by ib nexus driver. The device might be known to the IB Device Manager. |
The following sections describe how to dynamically reconfigure (DR) IB devices with the cfgadm command. All of the sample IB device information in these sections has been truncated to focus on relevant information.
You can use the prtconf command to display general information about IB devices. For example:
$ prtconf pci, instance #0 pci15b3,5a44, instance #0 ibport, instance #253 ibport, instance #254 ibport, instance #255 . . . ib, instance #0 ioc, instance #243 ioc, instance #244 ioc, instance #245 ioc, instance #246 ioc, instance #247 ioc, instance #248 ibgen, instance #249 |
In the preceding example, pci15b3,5a44 refers to an IB HCA.
Use the following steps to display specific IB device information.
Become superuser.
Display IB fabric information.
For example:
# cfgadm -a Ap_Id Type Receptacle Occupant Condition ib IB-Fabric connected configured ok hca:1730000008070 IB-HCA connected configured ok ib::1730000007F5198 IB-IOC connected configured ok ib::1730000007F5199 IB-IOC connected configured ok ib::1730000008070,0,hnfs IB-HCA_SVC connected configured ok ib::1730000008071,0,sdp IB-PORT connected configured ok ib::1730000008072,0,sdp IB-PORT connected configured ok ib::1730000008071,8001,ipib IB-VPPA connected configured ok ib::1730000008072,8001,ipib IB-VPPA connected configured ok ib::ibgen,0 IB-PSEUDO connected configured ok # |
In the preceding example output, the components are described as follows:
Identifies an IB Port device that is connected to port 2 and is bound to the sdp service.
Identifies an IB VPPA device that is connected to port 2, using a p_key value of 0x8001, and is bound to the ibd service.
Identifies an IB HCA_SVC device bound to the hnfs service.
Identifies an IOC device.
Identifies a pseudo device.
Display specific IB device information.
For example, for an IB VPPA device:
# cfgadm -al -s "cols=ap_id:info" ib::1730000008072,8001,ipib Ap_Id Information ib::1730000008072,8001,ipib ipib # |
For example, for an IB HCA device:
# cfgadm -al -s "cols=ap_id:info" hca::1730000008070 Ap_Id Information hca::1730000008070 VID: 0x15b3, PID: 0x5a44, #ports: 0x2, port1 GUID: 0x1730000008071, port2 GUID: 0x1730000008072 # |
The preceding output displays the number of ports and their GUIDs.
You can unconfigure an IB device that is still physically connected to the system, but a driver will never attach to it.
Become superuser.
For example:
# cfgadm -c unconfigure ib::1730000007F5198 Unconfigure the device: /devices/ib:fabric::1730000007F5198 This operation will suspend activity on the IB device Continue (yes/no)? y # |
Verify that the device is unconfigured.
For example:
# cfgadm -a ib::1730000007F5198 ib::1730000007F5198 IB-IOC connected unconfigured unknown # |
Become superuser.
For example:
# cfgadm -yc configure ib::1730000007F5198 |
Verify that the IB device is configured.
For example:
# cfgadm -al ib::1730000007F5198 Ap_Id Type Receptacle Occupant Condition ib::1730000007F5198 IB-IOC connected configured ok |
Use the following steps if you want to remove an IB Port, HCA_SVC, or a VPPA device from the system.
The example below illustrates how to unconfigure a VPPA device, but the same procedure applies to Port and HCA_SVC devices as well.
Become superuser.
Unconfigure the IB VPPA device.
For example:
# cfgadm -c unconfigure ib::1730000007F51,8001,ipib Unconfigure the device: /devices/ib:fabric::1730000007F51,8001,ipib This operation will suspend activity on the IB device Continue (yes/no)? Y # |
Verify that the device is disconnected.
For example:
# cfgadm -a ib::1730000007F51,8001,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,8001,ipib IB-VPPA connected unconfigured unknown # |
Use the following steps if you want to configure an IB Port, HCA_SVC, or a VPPA device on the system.
The example below illustrates how to configure a VPPA device, but similar steps can be used to configure Port and HCA_SVC devices as well.
Become superuser.
For example:
# cfgadm -c configure ib::1730000007F51,8001,ipib |
Verify that the device is connected.
For example:
# cfgadm -a ib::1730000007F51,8001,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,8001,ipib IB-VPPA connected configured ok |
A cfgadm based configure or unconfigure operation of IB Port and HCA_SVC devices is similar to the preceding examples for an IB VPPA device.
Use the following steps if you want to remove an IB pseudo device from the system.
Become superuser.
Unconfigure the IB pseudo device.
For example:
# cfgadm -c unconfigure ib::ibgen,0 Unconfigure the device: /devices/ib:fabric::ibgen,0 This operation will suspend activity on the IB device Continue (yes/no)? Y # |
Verify that the device is disconnected.
# cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected unconfigured unknown |
Use the following steps to configure an IB pseudo device.
Become superuser.
Configure the IB pseudo device.
For example:
# cfgadm -yc configure ib::ibgen,0 |
Verify that the device is connected.
For example:
# cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected configured ok |
The following IB cfgadm plugin command can be invoked to list kernel IB clients using this HCA. Note that the last column would show a “yes” if a kernel IB client uses another HCA. IB Managers and kernel clients that do not use the HCA are shown with an Ap_Id of “-”.
Display kernel IB clients of an HCA.
For example:
$ cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,8001,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no $ |
An actual DR of an HCA is beyond the scope of the IB cfgadm plugin. Although DR of an HCA can be achieved by using the plugin of the underlying bus. For example, a PCI based HCA can use the cfgadm_pci command. For more information, see cfgadm_pci(1M).
However, the IB cfgadm plugin assists in the HCA DR by listing its kernel IB clients as illustrated in steps below.
Become superuser.
List the kernel IB clients of the HCA.
For example:
# cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,8001,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no |
Unconfigure kernel IB clients, such as Port, VPPA, HCA_SVC, or IOC devices, that do not have alternate HCAs present.
For example:
# cfgadm -x unconfig_clients hca:1730000008070 Unconfigure Clients of HCA /devices/ib:1730000008070 This operation will unconfigure IB clients of this HCA Continue (yes/no)? y |
Verify that the kernel IB clients of the HCA are unconfigured.
# cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA - ibdm no - ibmf no - nfs/ib no # |
Invoke the bus-specific cfgadm plugin to configure the HCA. The exact details are beyond the scope of this chapter.
If the p_key table information of an HCA ports changes, for example, additional p_keys are enabled or disabled, InfiniBand Transport Framework (IBTF) and IBDM need to be notified so that their internal p_key databases are updated. The cfgadm command helps update the p_key databases of IBTF and IBDM. For more information, see ibtl(7D) and ibdm(7D).
Use the following steps to display the communication services that are currently in use by the IBTF.
Become superuser.
Display IB communication services.
For example:
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs |
Use the following steps to add a new VPPA communication service.
Similar steps can be used to add a new HCA_SVC or a port communication service.
Become superuser.
Add a new VPPA communication service.
For example:
# cfgadm -o comm=vppa,service=new -x add_service ib |
Verify that the new service has been added.
For example:
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd new HCA_SVC communication services: nfs_service # |
Use the following steps to delete an existing IB Port, HCA_SVC, or a VPPA communication service.
Become superuser.
Remove a VPPA communication service.
For example:
# cfgadm -o comm=vppa,service=new -x delete_service ib |
Verify that the communication service has been removed.
For example:
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs # |
Use the following steps to update properties of all the IOC device nodes or for a particular IOC Ap_Id. The properties that can get updated are as follows:
port-list
port-entries
service-id
service-name
For more information on these properties, see ib(7D).
Note that these properties may not get updated if there is no configuration change. The following example describes how to update a particular IOC's configuration. If you need to update the configuration of all the IOCs, then specify the static ib Ap_Id instead of the particular IOC Ap_Id.