ibadm - administer InfiniBand Host Channel Adapters
ibadm
ibadm show-hca [[-p] -o <field>,...] [-v] [-l] [<hca>[.<vhca>]]
ibadm show-port [[-p] -o <field>,...] [<hca>[.<vhca>] [<port>]]
ibadm enable-iov [-f] [-p max-vhcas=<value>] <hca>
ibadm disable-iov [-f] <hca>
ibadm create-vhca [-f] [-p <prop>=<value>[,...] <hca>[.<vhca>]
ibadm delete-vhca [-f] <hca>.<vhca>
ibadm set-hcaprop [-f] -p <prop>=<value>[,...] <hca>[.<vhca>]
ibadm reset-hcaprop [-f] -p <prop>[,...] <hca>[.<vhca>]
ibadm show-hcaprop [[-c] -o <field>,...] [-p <prop>[,...]] [<hca>[.<vhca>]
ibadm set-portprop [-f] -p <prop>=<value>[,...] <hca>[.<vhca>] <port>
ibadm reset-portprop [-f] -p <prop>[,...] <hca>[.<vhca>] <port>
ibadm show-portprop [[-c] -o <field>,...] [-p <prop>[,...]]
[<hca>[.<vhca>] [port]]
ibadm delete-hca [-f] <hca>
ibadm help [<sub-command>]
The ibadm command is used to administer InfiniBand Host Channel Adapters (HCAs). An HCA device functions as an InfiniBand node with multiple ports that may be connected to the InfiniBand fabric.
Some HCA devices support the PCI-Express SR-IOV capability which emulates multiple virtual HCAs on a single physical HCA. Each virtual HCA can be used by a different kernel zone or LDOMS IO domain making it appear as if the virtual systems are directly attached to the InfiniBand network.
ibadm has a series of sub-commands to perform various operations. These sub-commands operate on the following objects:
A Host Channel Adapter. ibadm manages physical and virtual HCAs. The name for a physical HCA consists of a driver name and a driver instance number. The name for a virtual HCA created by Solaris over a physical HCA consists of the physical HCA name, the string ".vhca", and a VHCA instance number. For example:
Physical HCA name: hermon0 Virtual HCA name: hermon0.vhca2
When a virtual HCA is allocated to a kernel zone or LDOMS IO domain and is viewed from inside the virtual system, the virtual HCA's name consists of a driver name and driver instance number.
Inherited HCA name inside virtual system: hermon0
Some sub-commands operate only on physical HCAs, others only on virtual HCAs.
A physical or virtual point of attachment to the InfiniBand network. A port is identified by its instance number on the HCA. Most sub-commands will accept just the number, or the number appended to the word "port". Depending on context, the HCA name must be provided with the port identification.
Specific port on an HCA: hermon1 port2
The sub-commands have various options. There is both a short form and a long form of each option. For example, "–o" is the short form of "--output".
When "ibadm" is specified alone on the command line, it will display a summary of the HCAs configured on the system. This is the same display you see when entering "ibadm show-hca".
ibadm supports the following sub-commands:
show-hca
show-port
enable-iov
disable-iov
create-vhca
delete-vhca
set-hcaprop
reset-hcaprop
show-hcaprop
set-portprop
reset-portprop
show-portprop
delete-hca
help
ibadm show-hca [[-p] -o <field>,...] [-v] [-l] [<hca>[.<vhca>]]
Show HCA configuration information either for all HCAs or for the specified HCA. The specified HCA may be either a physical or virtual HCA.
A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value "all" to display all fields. By default (without –o), show-hca displays the HCA, TYPE, STATE, IOV, and ZONE fields.
Path of directory names to the HCA device node in the system's device tree.
Additional HCA status information. For each flag, the system will display either the specific flag character to indicate the condition exists, or a "-" to indicate the condition does not exist.
U - Updates pending for reboot. The currently running HCA configuration is not the same as the default or persistently defined configuration. This can happen when the --force option was used to override a blocked change. The system will use the persistently defined configuration the next time the system reboots.
T - Temporary. A virtual HCA configuration is not persistently defined. This can happen when the –force switch was used to remove the virtual HCA. The virtual HCA will not be re-created the next time the system boots.
I - Inherited. A virtual HCA inside a kernel zone or an LDOMS I/O domain was inherited from LDOMS or from a parent Solaris system.
Z - The virtual HCA was created by zoneadmd and is allocated to a kernel zone.
l - SR-IOV features on the physical HCA are under LDOMS control. If this flag is set, ibadm will not be able to create VHCAs over the physical HCA. This flag is only possible in a primary root LDOMS domain.
b - The physical HCA is used for boot. The –force switch has to be used in order to change IOV status or max-vhcas property.
The name of the physical or virtual HCA.
Maximum number of virtual functions the physical HCA device can support.
State of the SR-IOV feature on a physical HCA. Possible values are "on" or "off". When set "on", virtual HCAs may be created.
This field may contain a "(R)", "(b)", or "(l)" suffix. The "(R)" suffix with the "on" or "off" settings indicate that the IOV setting has been changed to the value shown using the –-force option. A reboot will be required to complete the configuration. The "(b)" suffix with the "on" or "off" settings indicate that this HCA is used for boot. Any changes to IOV or MAX-VHCAS need –-force option. The "(l)" suffix with the "on" setting indicates SR-IOV features on the physical HCA are under LDOMS control.
Total number of virtual functions configured on a physical HCA upon which virtual HCAs may be created.
Globally Unique Identifier of the HCA. The Node GUID is also referred to as an HCA GUID in the InfiniBand specifications.
Number of available ports on the HCA.
Specifies whether the virtual HCA can use Subnet Management Packets (SMPs). If the value of this property is "on", then SMPs are allowed for this virtual HCA. If this property is "off" then SMPs cannot be used with this virtual HCA. If the value is "readonly", then this virtual HCA can only use query SMP operations and not "set" operations.
Current state of the HCA. Possible values are:
online - The HCA driver is running in the kernel for this device.
offline - The HCA driver is not attached to this device. This may occur when an HCA card has been removed from the system or when the HCA is transitioning between IOV states in response to an enable-iov or disable-iov sub-command.
degraded - The system encountered a problem that prevented successful configuration of the requested IOV and MAX-VHCAS settings even though the settings are valid for the HCA. For example, disabling ARI Support in the BIOS on some Intel based systems effectively limits the configuration to 7 virtual HCAs on a physical HCA. When Solaris detects that something blocked a configuration attempt, it automatically falls back to an HCA configuration that the system can support. Use the show-hcaprop sub-command to view the degraded current and requested persistent property values. See the Solaris Release Notes for possible causes of degraded SR-IOV operation.
Type of HCA. Possible values are "physical" or "virtual".
Number of unused virtual functions under the current physical HCA that are available for the creation of additional virtual HCAs.
Number of virtual HCAs that are currently in-use under a physical HCA. This includes both online and offline VHCAs.
Name of the zone a virtual HCA is currently allocated to.
Display the requested fields in a machine-parseable format. The –o option is required with -p. See "Parseable Output Format", below.
Include all virtual HCAs under the specified physical HCA in the display output.
Show additional non-default fields in an alternative "long" format. The additional fields included are vhcas-inuse, vhcas-avail, max-vhcas, node-guid, device-path, num-ports, plus the port fields portN-guids and portN-pkeys for each port on the HCA. See the show-port command below for a definition of the port guids and pkeys fields. The additional fields that do not apply to a particular HCA will suppressed from the display. This option may not be used with the -o option.
ibadm show-port [[-p] -o <field>,...] [<hca>[.<vhca>] [<port>]]
Show HCA port configuration information either for all HCAs, all ports on the specified HCA, or for the specified port on the specified HCA. The specified HCA may be either a physical or virtual HCA. The port may be specified using just the port number or the port number appended to the word "port".
A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value "all" to display all fields. By default (without –o), show-port displays all fields.
Comma separated list of globally unique identifiers configured on the port.
The name of the physical or virtual HCA.
Comma separated list of InfiniBand Pkeys available on the port.
The port instance number on the HCA.
Display the requested fields in a machine-parseable format. The –o option is required with –p. See "Parseable Output Format", below.
ibadm enable-iov [-f] [-p max-vhcas=<value>] <hca>
Enable the PCI-Express SR-IOV feature on the specified physical HCA. IOV must be enabled before any virtual HCAs can be created on a physical HCA.
A comma-separated list of HCA properties to set to the specified values. The only supported property on the enable-iov sub-command is max-hcas. See "HCA Properties" below for details.
Force this configuration change if something on the actively running system is blocking the change. The requested change will be held persistently and will take effect the next time the system boots.
ibadm disable-iov [-f] <hca>
Disable the PCI-Express SR-IOV feature on the specified physical HCA.
Force this configuration change if something on the actively running system is blocking the change. The requested change will be held persistently and will take effect the next time the system boots. Any VHCAs that are configured in the global zone under this physical HCA will automatically be deleted.
ibadm create-vhca [-f] [-p <prop>=<value>[,...] <hca>[.<vhca>]
Create a virtual HCA under the specified physical HCA. If the optional virtual HCA instance is specified, the system will attempt to create it. Otherwise, the system will select a vhca instance from the device's available virtual functions. This sub-command displays the name of the virtual HCA that is allocated.
A comma-separated list of port properties to set to their specified values. The properties that may be set are:
pkeys
Port property names must be identified using "port<number>-" prefixed to the property name. For example, "port1-pkeys=FFFF". See "Entering Multiple Property Values" and "Port Properties" below.
Force this configuration change if something on the actively running system is blocking the change. The requested change will be held persistently and will take effect the next time the system boots.
ibadm delete-vhca [-f] <hca>.<vhca>
Destroy the specified virtual HCA. This sub-command will not delete virtual HCAs that were created by zoneadmd. Use zoneadm(8) and zonecfg(8) to change zone configuration.
Force this configuration change if something on the actively running system is blocking the change. The requested change will be held persistently and will take effect the next time the system boots.
ibadm set-hcaprop [-f] -p <prop>=<value>[,...] <hca>[.<vhca>]
Modifies property values on an existing physical HCA to the values specified by the user.
A comma-separated list of HCA properties to set to their specified values. See "Entering Multiple Property Values" and "HCA Properties" below.
Force this configuration change if something on the actively running system is blocking the change. The requested change will be held persistently and will take effect the next time the system boots.
ibadm reset-hcaprop [-f] -p <prop>[,...] <hca>[.<vhca>]
Reset property values on an existing physical HCA back to their default values.
A comma-separated list of HCA properties. See "Entering Multiple Property Values" and "HCA Properties" below.
Force this configuration change if something on the actively running system is blocking the change. The requested change will be held persistently and will take effect the next time the system boots.
ibadm show-hcaprop [[-c] -o <field>,...] [-p <prop>[,...]] [<hca>[.<vhca>]
Show the current and persistent values of one or more properties, either for all the existing HCAs or for the specified HCA. Several properties of interest can be retrieved at one time by providing comma-separated property names to the –p option. If the –p option is not specified, all available HCA properties are displayed.
A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value "all" to display all fields. All fields are displayed by default.
The name of the physical or virtual HCA.
The name of the property.
The read/write permissions of the property. The value shown will be r (read-only) or rw (read-and-write).
The current value of the property. Current property values will be shown as "--" on disabled interfaces to indicate "not set".
The persistent value of the property. Persistent values are the values that will be reapplied on reboot.
The default value of the property. "--" is displayed if the property has no default value.
A comma-separated list of the values the property can have. If the values span a numeric range, min - max might be displayed as short-hand. If the possible values are unknown, ? is displayed or if they are unbounded, -- is displayed.
Display the requested fields in a machine-parseable format. The –o option is required with –c. See "Parseable Output Format", below.
Comma separated list of HCA properties to display.
ibadm set-portprop [-f] -p <prop>=<value>[,...] <hca>[.<vhca>] port
Modifies property values on an existing virtual HCA port to the values specified by the user. The port may be specified using just the port number, or the port number appended to the word "port". This sub-command will not modify port properties on zoneadmd generated virtual HCAs. Use zoneadm(8) and zonecfg(8) to modify zone configuration.
A comma-separated list of HCA properties to set to their specified values. See "Entering Multiple Property Values" and "HCA Properties" below.
Force this configuration change if something on the actively running system is blocking the change. The requested change will be held persistently and will take effect the next time the system boots.
ibadm reset-portprop [-f] -p <prop>[,...] <hca>[.<vhca>] port
Reset property values on an existing virtual HCA port back to their default values. The port may be specified using just the port number, or the port number appended to the word "port". This sub-command will not modify port properties on zoneadmd generated virtual HCAs. Use zoneadm(8) and zonecfg(8) to modify zone configuration.
A comma-separated list of HCA properties. See "Entering Multiple Property Values" and "HCA Properties" below.
Force this configuration change if something on the actively running system is blocking the change. The requested change will be held persistently and will take effect the next time the system boots.
ibadm show-portprop [[-c] -o <field>,...] [-p <prop>[,...]] [<hca>[.<vhca>] [port]]
Show the current and persistent values of one or more port properties, either for all ports on the existing HCAs, all ports on the specified HCA, or the specified port on the specified HCA. The port may be specified using just the port number, or the port number appended to the word "port".
Several properties of interest can be retrieved at one time by providing comma-separated list of property names to the –p option. If the –p option is not specified, all available port properties are displayed.
A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value "all" to display all fields. All fields are displayed by default.
The name of the physical or virtual HCA.
The port instance number on the HCA.
The name of the property.
The read/write permissions of the property. The value shown will be r (read-only) or rw (read-and-write).
The current value of the property. Current property values will be shown as "--" on disabled interfaces to indicate "not set".
The persistent value of the property. Persistent values are the values that will be reapplied on reboot.
The default value of the property. "--" is displayed if the property has no default value.
A comma-separated list of the values the property can have. If the values span a numeric range, min - max might be displayed as short-hand. If the possible values are unknown, ? is displayed or if they are unbounded, -- is displayed.
Display the requested fields in a machine-parseable format. The –o option is required with –c. See "Parseable Output Format", below.
Comma separated list of HCA properties to display.
ibadm delete-hca [-f] <hca>
Clear the persistent configuration of a physical HCA and any underlying virtual HCAs associated with hardware that has been removed from the system.
The system does not automatically purge configuration information from persistent storage when you remove InfiniBand hardware. As a result, the ibadm show-hca sub-command may continue to list removed hardware as "disabled" until you explicitly clear the configuration with the delete-hca sub-command. This sub-command also works for inherited virtual HCAs.
Force this configuration change if something on the actively running system is blocking the change. The requested change will be held persistently and will take effect the next time the system boots.
ibadm help [<sub-command>]
Displays all the supported ibadm sub-commands or usage for a given sub-command. If you invoke help for a specific sub-command, the command syntax is displayed. Using ibadm help without any argument displays all of the sub-commands.
Many ibadm sub-commands have an option to enter lists of properties with their respective values after the –p option. Multiple property name/value pairs may be entered by separating each with a comma or by specifying more properties with an additional –p option.
# ibadm create-vhca -p port1-pkeys=ffff,port2-pkeys=ffff
For properties with multiple values, enter the comma-separated list of values after the property name.
# ibadm create-vhca -p port1-pkeys=8040,8060,ffff \ -p port2-pkeys=8002,8060,ffff
Many ibadm sub-commands have an option that displays output in a machine-parseable format. The output format is one or more lines of colon (:) delimited fields. Output includes only those fields requested by means of the –o option, in the order requested.
When you request multiple fields, any literal colon characters are escaped by a backslash (\) before being output. Similarly, literal backslash characters will also be escaped (\\). This escape format is parseable by using shell read() functions with the environment variable IFS=: (see EXAMPLES, below). For more information, see, read(1) man page. Note that escaping is not done when you request only a single field.
The following HCA properties are supported:
Setting this property on a physical HCA allows the user to enable/disable SR-IOV mode. This property does not apply to virtual HCAs nor to physical HCA devices that do not support SR-IOV technology. The possible values are:
Turns SR-IOV mode on. This will allow the creation of virtual HCAs.
Turns SR-IOV mode off. This will disable the ability to create virtual HCAs.
Total number of virtual functions to configure on a physical HCA upon which virtual HCAs may be created. This property does not apply to virtual HCAs nor to physical HCA devices that do not support SR-IOV technology. The possible values are 1 to hw-max-vhcas.
Specifies whether the virtual HCA can use Subnet Management Packets (SMPs). If the value of this property is "on", then SMPs are allowed for this virtual HCA. If this property is "off" then SMPs cannot be used with this virtual HCA. If the value is "readonly", then this virtual HCA can only use query SMP operations and not "set" operations. The default value is "off". The value "on" is not recommended without considering the possible security impact on the fabric. When running with "on", M_Keys should be set on fabric components.
The following HCA port properties are supported:
A comma separated list of InfiniBand Pkeys available on the port. A Pkey consists of a hex-string containing from 1 to 4 hexadecimal characters. Each Pkey entered for a virtual HCA must exist on the corresponding physical HCA port. Either the default or limited default Pkey (FFFF or 7FFF) must be included. 0 and 8000 are invalid Pkey values. The keyword "auto" may be specified without any other values to use the Pkeys defined on the physical HCA port.
The following sub-commands use the PRIV_SYS_IB_INFO privilege which is included in the basic privilege set.
show-hca, show-port, show-hcaprop, show-portprop
The following sub-commands require solaris.smf.modify.ib and solaris.smf.manage.ib authorizations, both of which are included in the "InfiniBand Management" and "Network Management" execution profiles. These sub-commands use the PRIV_SYS_DEVICES privilege.
enable-iov, disable-iov, create-vhca, delete-vhca, set-hcaprop, reset-hcaprop, set-portprop, reset-portprop, delete-hca
The SR-IOV configuration of a physical HCA may be enabled and configured either by ibadm in the Solaris host system's global zone or by LDOMS. When LDOMS enables the IOV capability on an HCA, it takes full control of VHCA configuration. The Solaris ibadm command cannot view or modify the VHCA configuration on an LDOMS controlled HCA. Likewise, when ibadm is used to enable SR-IOV, LDOMS cannot be used to view or modify the VHCA configuration on the device.
If you need to switch an HCA from an ibadm controlled SR-IOV configuration to an LDOMS controlled configuration, use the ibadm delete-vhca subcommand to remove all existing VHCAs on the device, then use the disable-iov sub-command to release SR-IOV control. Once these steps are complete, you may use LDOMS to configure SR-IOV on the device.
Likewise, to switch from an LDOMS controlled SR-IOV configuration, use the LDOMS commands to remove all virtual HCAs and disable SR-IOV before reconfiguring SR-IOV with the ibadm command.
The following command displays a summary of the HCAs configured on the system.
#ibadm HCA TYPE STATE IOV ZONE hermon2 physical online on global hermon2.vhca3 virtual online -- global hermon2.vhca5 virtual online -- kzone1 hermon3 physical online off globalExample 2 Displaying Specific HCA Configuration in Long Form
The following command displays additional details about a specific HCA in long form.
# ibadm show-hca -l hermon2 HCA TYPE STATE IOV ZONE --------------------------------------------------------- hermon2 physical online on global device-path = pci@400/pci@1/pci@0/pci@0/pciex15b3,673c@0 num-vhcas = 2 in use, 61 available, 63 maximum node-guid = 21280001A17F56 port1-guids = 21280001A17F57 port2-guids = 21280001A17F58 port1-pkeys = FFFF,8002 port2-pkeys = FFFF,8002Example 3 Displaying Specific Fields of an HCA and it's VHCAs
The following command displays specific details about hermon2 and it's underlying VHCAs.
# ibadm show-hca -v -o hca,type,state,flags,zone hermon2 HCA TYPE STATE FLAGS ZONE hermon2 physical online ----- global hermon2.vhca3 virtual online U---- global hermon2.vhca5 virtual online ---Z- kzone1Example 4 Displaying HCA Port Configuration
The following command displays the configuration of all ports available to the current system. The ports on hermon2.vhca5 are not listed because that vhca was allocated to a kernel zone.
# ibadm show-port HCA PORT GUIDS PKEYS hermon2 1 21280001A17F57 FFFF,2 hermon2 2 21280001A17F58 FFFF,2 hermon2.vhca3 1 14050000000008 FFFF hermon2.vhca3 2 14050000000009 FFFF hermon3 1 14050000000016 FFFF hermon3 2 14050000000017 FFFFExample 5 Displaying Properties to Identify Pending Changes
The following command shows that the hermon3 physical HCA has a pending configuration change but does not indicate what the change is.
# ibadm show-hca -v -o hca,type,state,iov,num-vhcas,flags hermon3 HCA TYPE IOV STATE NUM-VHCAS FLAGS hermon3 physical on online 7 U----
The following commands show both the current and persistent values of properties that apply to the HCA and it's ports. The displays show that the max-vhcas value will change from the current value of 7 to the persistent value of 63 on the next reboot. The port pkeys property can not be modified on this HCA.
# ibadm show-hcaprop hermon3 HCA PROPERTY PERM CURRENT PERSISTENT DEFAULT hermon3 iov rw on on off hermon3 max-vhcas rw 7 63 1-63
# ibadm show-portprop hermon3 HCA PORT PROPERTY PERM CURRENT PERSISTENT DEFAULT hermon3 1 pkeys r- ffff -- -- hermon3 2 pkeys r- ffff,8002 -- --Example 6 Enabling Virtualization on a Physical HCA
The following command attempts to enable virtualization on hermon3. The HCA is currently being used by at least one InfiniBand service. It therefore cannot be reconfigured immediately.
# ibadm enable-iov hermon3 ibadm: failed to update configuration entity is busy
Use the –f switch to make a change that will take effect on the next reboot.
# ibadm enable-iov -f hermon3 ibadm: failed to update configuration requested change will take effect on reboot
# ibadm HCA TYPE STATE IOV ZONE hermon2 physical online on global hermon2.vhca3 virtual online -- global hermon2.vhca5 virtual online -- kzone1 hermon3 physical online on(R) global # shutdown -rExample 7 Creating a Virtual HCA
The following command creates a virtual HCA with P-Keys FFFF,2 on both ports. It lets the system select which available virtual function to use.
# ibadm create-vhca -p port1-pkeys=ffff,8002 \ -p port2-pkeys=ffff,8002 hermon3 hermon3.vhca1Example 8 Changing Property on an HCA Port
The following command changes the P-Keys on a port2 of virtual HCA hermon3.vhca1.
# ibadm set-portprop -p pkeys=ffff,8002 hermon3.vhca1 2Example 9 Removing an HCA Card
Solaris device drivers automatically detect when an InfiniBand HCA card is physically removed from the system and will react by no longer configuring the physical device. If the system has persistent configuration information stored for the HCA, the device will continue to be shown in the show-hca sub-command display but with state "disabled". If you do not intend to replace the device, enter the following command to delete the persistent HCA configuration.
# ibadm delete-vhca hermon1Example 10 Using Parseable Output to Capture a Single Field
The following assignment saves an HCA's vhcas-avail in a shell variable named available.
# available=`ibadm show-hca -p -o vhcas-avail hermon2` # print $available 30Example 11 Using Parseable Output to Iterate over HCAs
The following script displays the online/offline state of each HCA on the system.
# ibadm show-hca -p -o hca,state | \
while IFS=: read hca state; do
print "HCA $hca is $state"
done
See attributes(7) for descriptions of the following attributes:
|