Skip Navigation Links | |
Exit Print View | |
System Administration Guide: Devices and File Systems Oracle Solaris 10 1/13 Information Library |
1. Managing Removable Media (Overview/Tasks)
2. Writing CDs and DVDs (Tasks)
4. Dynamically Configuring Devices (Tasks)
5. Managing USB Devices (Tasks)
6. Using InfiniBand Devices (Overview/Tasks)
Overview of InfiniBand Devices
Dynamically Reconfiguring IB Devices (Task Map)
Dynamically Reconfiguring IB Devices (cfgadm)
How to Display IB Device Information
How to Unconfigure an IOC Device
How to Configure an IOC Device
How to Unconfigure an IB Port, HCA_SVC, or a VPPA Device
How to Configure a IB Port, HCA_SVC, or a VPPA Device
How to Unconfigure an IB Pseudo Device
How to Configure an IB Pseudo Device
How to Display Kernel IB Clients of an HCA
How to Unconfigure IB Devices Connected to an HCA
How to Update the IB p_key Tables
How to Display IB Communication Services
How to Add a VPPA Communication Service
How to Remove an Existing IB Port, HCA_SVC, or a VPPA Communication Service
Using the uDAPL Application Interface With InfiniBand Devices
Updating the DAT Static Registry
How to Update the DAT Static Registry
How to Register a Service Provider in the DAT Static Registry
How to Unregister a Service Provider from the DAT Static Registry
9. Administering Disks (Tasks)
10. SPARC: Setting Up Disks (Tasks)
11. x86: Setting Up Disks (Tasks)
12. Configuring Oracle Solaris iSCSI Targets (Tasks)
13. The format Utility (Reference)
14. Managing File Systems (Overview)
15. Creating and Mounting File Systems (Tasks)
16. Configuring Additional Swap Space (Tasks)
17. Checking UFS File System Consistency (Tasks)
18. UFS File System (Reference)
19. Backing Up and Restoring UFS File Systems (Overview/Tasks)
20. Using UFS Snapshots (Tasks)
21. Copying Files and File Systems (Tasks)
22. Managing Tape Drives (Tasks)
One can configure or unconfigure an IB device from a running system by using the cfgadm CLI only. This command also provides a way to display the IB fabric, manage communication services, and update p_key table databases. For more information, see cfgadm_ib(1M).
The cfgadm CLI manages dynamic reconfiguration, referred to in this guide as DR, of the entire IB fabric as seen by a host. The cfgadm operations are supported on all the IB devices, such as Port, VPPA, HCA_SVC, IOC, and pseudo devices.
The cfgadm command displays information about attachment points (Ap_Ids), which are locations in the system where DR operations can occur. For details on the Ap_Ids that cfgadm supports, see cfgadm_ib(1M). Note that all IB Ap_Ids are shown as connected.
The cfgadm command provides the following IB device status information.
|
The following sections describe how to dynamically reconfigure (DR) IB devices with the cfgadm command. All of the sample IB device information in these sections has been truncated to focus on relevant information.
You can use the prtconf command to display general information about IB devices. For example:
$ prtconf pci, instance #0 pci15b3,5a44, instance #0 ibport, instance #253 ibport, instance #254 ibport, instance #255 . . . ib, instance #0 ioc, instance #243 ioc, instance #244 ioc, instance #245 ioc, instance #246 ioc, instance #247 ioc, instance #248 ibgen, instance #249
In the preceding example, pci15b3,5a44 refers to an IB HCA.
Use the following steps to display specific IB device information.
For example:
# cfgadm -a Ap_Id Type Receptacle Occupant Condition ib IB-Fabric connected configured ok hca:1730000008070 IB-HCA connected configured ok ib::1730000007F5198 IB-IOC connected configured ok ib::1730000007F5199 IB-IOC connected configured ok ib::1730000008070,0,hnfs IB-HCA_SVC connected configured ok ib::1730000008071,0,sdp IB-PORT connected configured ok ib::1730000008072,0,sdp IB-PORT connected configured ok ib::1730000008071,8001,ipib IB-VPPA connected configured ok ib::1730000008072,8001,ipib IB-VPPA connected configured ok ib::ibgen,0 IB-PSEUDO connected configured ok
In the preceding example output, the components are described as follows:
Identifies an IB Port device that is connected to port 2 and is bound to the sdp service.
Identifies an IB VPPA device that is connected to port 2, using a p_key value of 0x8001, and is bound to the ibd service.
Identifies an IB HCA_SVC device bound to the hnfs service.
Identifies an IOC device.
Identifies a pseudo device.
For example, for an IB VPPA device:
# cfgadm -al -s "cols=ap_id:info" ib::1730000008072,8001,ipib Ap_Id Information ib::1730000008072,8001,ipib ipib
For example, for an IB HCA device:
# cfgadm -al -s "cols=ap_id:info" hca::1730000008070 Ap_Id Information hca::1730000008070 VID: 0x15b3, PID: 0x5a44, #ports: 0x2, port1 GUID: 0x1730000008071, port2 GUID: 0x1730000008072
The preceding output displays the number of ports and their GUIDs.
You can unconfigure an IB device that is still physically connected to the system, but a driver will never attach to it.
For example:
# cfgadm -c unconfigure ib::1730000007F5198 Unconfigure the device: /devices/ib:fabric::1730000007F5198 This operation will suspend activity on the IB device Continue (yes/no)? y
For example:
# cfgadm -a ib::1730000007F5198 ib::1730000007F5198 IB-IOC connected unconfigured unknown
For example:
# cfgadm -yc configure ib::1730000007F5198
For example:
# cfgadm -al ib::1730000007F5198 Ap_Id Type Receptacle Occupant Condition ib::1730000007F5198 IB-IOC connected configured ok
Use the following steps if you want to remove an IB Port, HCA_SVC, or a VPPA device from the system.
The example below illustrates how to unconfigure a VPPA device, but the same procedure applies to Port and HCA_SVC devices as well.
For example:
# cfgadm -c unconfigure ib::1730000007F51,8001,ipib Unconfigure the device: /devices/ib:fabric::1730000007F51,8001,ipib This operation will suspend activity on the IB device Continue (yes/no)? Y
For example:
# cfgadm -a ib::1730000007F51,8001,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,8001,ipib IB-VPPA connected unconfigured unknown
Use the following steps if you want to configure an IB Port, HCA_SVC, or a VPPA device on the system.
The example below illustrates how to configure a VPPA device, but similar steps can be used to configure Port and HCA_SVC devices as well.
For example:
# cfgadm -c configure ib::1730000007F51,8001,ipib
For example:
# cfgadm -a ib::1730000007F51,8001,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,8001,ipib IB-VPPA connected configured ok
Note - A cfgadm based configure or unconfigure operation of IB Port and HCA_SVC devices is similar to the preceding examples for an IB VPPA device.
Use the following steps if you want to remove an IB pseudo device from the system.
For example:
# cfgadm -c unconfigure ib::ibgen,0 Unconfigure the device: /devices/ib:fabric::ibgen,0 This operation will suspend activity on the IB device Continue (yes/no)? Y
# cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected unconfigured unknown
Use the following steps to configure an IB pseudo device.
For example:
# cfgadm -yc configure ib::ibgen,0
For example:
# cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected configured ok
The following IB cfgadm plugin command can be invoked to list kernel IB clients using this HCA. Note that the last column would show a “yes” if a kernel IB client uses another HCA. IB Managers and kernel clients that do not use the HCA are shown with an Ap_Id of “-”.
For example:
$ cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,8001,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no
An actual DR of an HCA is beyond the scope of the IB cfgadm plugin. Although DR of an HCA can be achieved by using the plugin of the underlying bus. For example, a PCI based HCA can use the cfgadm_pci command. For more information, see cfgadm_pci(1M).
However, the IB cfgadm plugin assists in the HCA DR by listing its kernel IB clients as illustrated in steps below.
For example:
# cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,8001,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no
For example:
# cfgadm -x unconfig_clients hca:1730000008070 Unconfigure Clients of HCA /devices/ib:1730000008070 This operation will unconfigure IB clients of this HCA Continue (yes/no)? y
# cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA - ibdm no - ibmf no - nfs/ib no
Invoke the bus-specific cfgadm plugin to configure the HCA. The exact details are beyond the scope of this chapter.
If the p_key table information of an HCA ports changes, for example, additional p_keys are enabled or disabled, InfiniBand Transport Framework (IBTF) and IBDM need to be notified so that their internal p_key databases are updated. The cfgadm command helps update the p_key databases of IBTF and IBDM. For more information, see ibtl(7D) and ibdm(7D).
Use the following steps to display the communication services that are currently in use by the IBTF.
For example:
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs
Use the following steps to add a new VPPA communication service.
Similar steps can be used to add a new HCA_SVC or a port communication service.
For example:
# cfgadm -o comm=vppa,service=new -x add_service ib
For example:
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd new HCA_SVC communication services: nfs_service
Use the following steps to delete an existing IB Port, HCA_SVC, or a VPPA communication service.
For example:
# cfgadm -o comm=vppa,service=new -x delete_service ib
For example:
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs
Use the following steps to update properties of all the IOC device nodes or for a particular IOC Ap_Id. The properties that can get updated are as follows:
port-list
port-entries
service-id
service-name
For more information on these properties, see ib(7D).
Note that these properties may not get updated if there is no configuration change. The following example describes how to update a particular IOC's configuration. If you need to update the configuration of all the IOCs, then specify the static ib Ap_Id instead of the particular IOC Ap_Id.