Enabling Intelligent Platform Management Interface (IPMI)

Intelligent Platform Management Interface (IPMI) provides a set of common interfaces to computer hardware and firmware that system administrators can use to monitor system health and manage the system.

Oracle Clusterware can integrate IPMI to provide failure isolation support and to ensure cluster integrity. You can configure node-termination with IPMI during installation by selecting IPMI from the Failure Isolation Support screen. You can also configure IPMI after installation with crsctl commands.

Requirements for Enabling IPMI

You must have the following hardware and software configured to enable cluster nodes to be managed with IPMI:

Each cluster member node requires a Baseboard Management Controller (BMC) running firmware compatible with IPMI version 1.5 or later, which supports IPMI over LANs, and configured for remote control using LAN.
Each cluster member node requires ipmitool or ipmiutil installed to support IPMI version 2.0 or later.
The cluster requires a management network for IPMI. This can be a shared network, but Oracle recommends that you configure a dedicated network.
Each cluster member node's port used by BMC must be connected to the IPMI management network.
Each cluster member must be connected to the management network.
Some server platforms put their network interfaces into a power saving mode when they are powered off. In this case, they may operate only at a lower link speed (for example, 100 MB, instead of 1 GB). For these platforms, the network switch port to which the BMC is connected must be able to auto-negotiate down to the lower speed, or IPMI will not function properly.

Note:

IPMI operates on the physical hardware platform through the network interface of the baseboard management controller (BMC). Depending on your system configuration, an IPMI-initiated restart of a server can affect all virtual environments hosted on the server. Contact your hardware and OS vendor for more information.

Configuring the IPMI Management Network

You can configure the BMC for DHCP, or for static IP addresses. Oracle recommends that you configure the BMC for dynamic IP address assignment using DHCP. To use this option, you must have a DHCP server configured to assign the BMC IP addresses.

For Oracle Clusterware to communicate with the BMC, the IPMI driver must be installed permanently on each node, so that it is available on system restarts. The IPMI driver is available on the Asianux Linux, Oracle Linux, Red Hat Enterprise Linux, and SUSE Linux Enterprise Server distributions supported with this release.

Note:

If you configure IPMI, and you use Grid Naming Service (GNS) you still must configure separate addresses for the IPMI interfaces. As the IPMI adapter is not seen directly by the host, the IPMI adapter is not visible to GNS as an address on the host.

Configuring the Open IPMI Driver

On Linux systems, the OpenIPMI driver is the supported driver for Oracle Clusterware deployments with IPMI. You can install and configure the driver dynamically by manually loading the required modules.

Note:

You must configure the IPMI driver either on all or none of the cluster nodes.

Contact your Linux distribution vendor for information about how to configure IPMI for your distribution. Configure the Open IPMI driver manually on Oracle Linux as follows:

Run the following commands:

# /sbin/modprobe ipmi_msghandler
# /sbin/modprobe ipmi_si 
# /sbin/modprobe ipmi_devintf

(Optional) Run the command /sbin/lsmod |grep ipmi to confirm that the IPMI modules are loaded. For example:
```
# /sbin/lsmod | grep ipmi
ipmi_devintf 12617  0 
ipmi_si 33377  0 
ipmi_msghandler  33701  2 
ipmi_devintf,ipmi_si
```
You can install the modules whether or not a BMC is present.
Open the /etc/rc.local file using a text editor, navigate to the end of the file, and enter lines similar to the following, to run the modprobe commands in step 2 automatically on system restart:
```
# START IPMI ON SYSTEM RESTART 
/sbin/modprobe ipmi_msghandler 
/sbin/modprobe ipmi_si 
/sbin/modprobe ipmi_devintf
```
On SUSE Linux Enterprise Server systems, add the modprobe commands above to /etc/init.d/boot.local.
Check to ensure that the Linux system is recognizing the IPMI device, using the following command:
```
ls -l /dev/ipmi0
```
If the IPMI device is dynamically loaded, then the output must be similar to the following:
```
# ls -l /dev/ipmi0 crw-------  1 root root 253, 0 Sep 23 06:29 /dev/ipmi0
```
If you do see the device file output, then the IPMI driver is configured, and you can ignore the following step.

If you do not see the device file output, then the udevd daemon is not set up to create device files automatically. Proceed to the next step.
Determine the device major number for the IPMI device using the command grep ipmi /proc/devices. For example:
```
# grep ipmi /proc/devices 253 ipmidev
```
The device major number is 253.
Run the mknod command to create a directory entry and i-node for the IPMI device, using the device major number. For example:
```
# mknod /dev/ipmi0 c 253 0x0
```
The permissions on /dev/ipmi0 in the example allow the device to be accessible only by root. The device should only be accessed by root, to prevent a system vulnerability.

Configuring the BMC

Configure BMC on each node for remote control using LAN for IPMI-based node fencing to function properly.

You can configure BMC from the BIOS prompt, using a distribution-specific management utility, or you can configure BMC using publicly available utilities, such as the following:

IPMItool, which is available for Linux:

http://ipmitool.sourceforge.net

IPMIutil, which is available for Linux:

http://ipmiutil.sourceforge.net

Refer to the documentation for the configuration tool you select for details about using the tool to configure the BMC.

When you configure the BMC on each node, you must complete the following:

Enable IPMI over LAN, so that the BMC can be controlled over the management network.
Enable dynamic IP addressing using DHCP or GNS, or configure a static IP address for the BMC.
Establish an administrator user account and password for the BMC.
Configure the BMC for VLAN tags, if you will use the BMC on a tagged VLAN.

The configuration tool you use does not matter, but these conditions must be met for the BMC to function properly.

Configuring BMC Using IPMItool

The following is an example of configuring BMC using ipmitool (version 1.8.6).

Log in as root.
Verify that ipmitool can communicate with the BMC using the IPMI driver by using the command bmc info, and looking for a device ID in the output. For example:
```
# ipmitool bmc info
Device ID                 : 32
.
.
.
```
If ipmitool is not communicating with the BMC, then review the section Configuring the BMC and ensure that the IPMI driver is running.
Enable IPMI over LAN using the following procedure:
1. Determine the channel number for the channel used for IPMI over LAN. Beginning with channel 1, run the following command until you find the channel that displays LAN attributes (for example, the IP address):
```
# ipmitool lan print 1

. . . 
IP Address Source       : 0x01
IP Address              : 140.87.155.89
. . .
```
2. Turn on LAN access for the channel found. For example, where the channel is 1:
```
# ipmitool -I bmc lan set 1 access on
```
Configure IP address settings for IPMI using the static IP addressing procedure:
If the BMC shares a network connection with ILOM, then the IP address must be on the same subnet. You must set not only the IP address, but also the proper values for netmask, and the default gateway. For example, assuming the channel is 1:
```
# ipmitool -I bmc lan set 1 ipaddr 192.168.0.55
# ipmitool -I bmc lan set 1 netmask 255.255.255.0
# ipmitool -I bmc lan set 1 defgw ipaddr 192.168.0.1
```
Note that the specified address (192.168.0.55) is associated only with the BMC, and does not respond to normal pings.

Establish an administration account with a username and password, using the following procedure (assuming the channel is 1):

Set BMC to require password authentication for ADMIN access over LAN. For example:
```
# ipmitool -I bmc lan set 1 auth ADMIN MD5,PASSWORD
```

List the account slots on the BMC, and identify an unused slot less than the maximum ID and not listed, for example, ID 4 in the following example. Note that some slots may be reserved and not available for reuse on some hardware.

# ipmitool user summary 1
Maximum IDs         : 20
Enabled User Count  : 3
Fixed Name Count    : 2
# ipmitool user list 1
ID  Name     Enabled Callin  Link Auth  IPMI Msg   Channel Priv Lim
1            true    false   false      true       USER
2   root     true    false   false      true       ADMINISTRATOR
3   sysoper  true    true    false      true       OPERATOR
12  default  true    true    false      true       NO ACCESS
13           true    false   true       false      CALLBACK

In the example above, there are 20 possible slots, and the first unused slot is number 4.

Assign the desired administrator user name and password and enable messaging for the identified slot. (Note that for IPMI v1.5 the user name and password can be at most 16 characters). Also, set the privilege level for that slot when accessed over LAN (channel 1) to ADMIN (level 4). For example, where username is the administrative user name, and password is the password:
```
# ipmitool user set name 4 username
# ipmitool user set password 4 password
# ipmitool user enable 4
# ipmitool channel setaccess 1 4 privilege=4
# ipmitool channel setaccess 1 4 link=on
# ipmitool channel setaccess 1 4 ipmi=on
```

Verify the setup using the command lan print 1. The output should appear similar to the following. Note that the items in bold text are the settings made in the preceding configuration steps, and comments or alternative options are indicated within brackets []:

# ipmitool lan print 1
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD
Auth Type Enable        : Callback : MD2 MD5
                        : User     : MD2 MD5
                        : Operator : MD2 MD5
                        : Admin    : MD5 PASSWORD
                        : OEM      : MD2 MD5
IP Address Source       : DHCP Address [or Static Address]
IP Address              : 192.168.0.55
Subnet Mask             : 255.255.255.0
MAC Address             : 00:14:22:23:fa:f9
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x40 Precedence=... 
Default Gateway IP      : 192.168.0.1
Default Gateway MAC     : 00:00:00:00:00:00
.
.
.
# ipmitool channel getaccess 1 4
Maximum User IDs     : 10
Enabled User IDs     : 2

User ID              : 4
User Name            : username [This is the administration user]
Fixed Name           : No
Access Available     : call-in / callback
Link Authentication  : enabled
IPMI Messaging       : enabled
Privilege Level      : ADMINISTRATOR

Verify that the BMC is accessible and controllable from a remote node in your cluster using the bmc info command. For example, if node2-ipmi is the network host name assigned the IP address of node2's BMC, then to verify the BMC on node node2 from node1, with the administrator account username, enter the following command on node1:
```
$ ipmitool -H node2-ipmi -U username lan print 1
```
You are prompted for a password. Provide the IPMI password.

If the BMC is correctly configured, then you should see information about the BMC on the remote node. If you see an error message, such as Error: Unable to establish LAN session, then you must check the BMC configuration on the remote node.
Repeat this process for each cluster member node. If the IPMI administrator account credentials on each cluster member node are not identical, then IPMI will fail during configuration.