Oracle^® Solaris Cluster Data Service for Oracle Real Application Clusters Guide

Failure of a Support for Oracle RAC Framework Resource Group

This section describes problems that can affect the Support for Oracle RAC framework resource group.

• Node Panic During Initialization of Support for Oracle RAC

• Failure of the ucmmd Daemon to Start

• How to Recover From a Failure of the ucmmd Daemon or a Related Component

Node Panic During Initialization of Support for Oracle RAC

If a fatal problem occurs during the initialization of Support for Oracle RAC, the node panics with an error messages similar to the following error message:

panic[cpu0]/thread=40037e60: Failfast: Aborting because "ucmmd" died 30 seconds ago

Description:  A component that the UCMM controls returned an error to the UCMM during a reconfiguration.

Cause:  The most common causes of this problem are as follows: A node might also panic during the initialization of Support for Oracle RAC because a reconfiguration step has timed out. For more information, see Node Panic Caused by a Timeout.

Solution:  For instructions to correct the problem, see How to Recover From a Failure of the ucmmd Daemon or a Related Component.

Failure of the ucmmd Daemon to Start

The UCMM daemon, ucmmd, manages the reconfiguration of Support for Oracle RAC. When a cluster is booted or rebooted, this daemon is started only after all components of Support for Oracle RAC are validated. If the validation of a component on a node fails, the ucmmd daemon fails to start on the node.

The most common causes of this problem are as follows:

• An error occurred during a previous reconfiguration of a component of Support for Oracle RAC.

• A step in a previous reconfiguration of Support for Oracle RAC timed out, causing the node on which the timeout occurred to panic.

For instructions to correct the problem, see How to Recover From a Failure of the ucmmd Daemon or a Related Component.

How to Recover From a Failure of the ucmmd Daemon or a Related Component

Perform this task to correct the problems that are described in the following sections:

• Node Panic During Initialization of Support for Oracle RAC

• Failure of the ucmmd Daemon to Start

1. To determine the cause of the problem, examine the log files for UCMM reconfigurations and the system messages file.

   For the location of the log files for UCMM reconfigurations, see Sources of Diagnostic Information.

   When you examine these files, start at the most recent message and work backward until you identify the cause of the problem.

   For more information about error messages that might indicate the cause of reconfiguration errors, see Oracle Solaris Cluster Error Messages Guide.

2. Correct the problem that caused the component to return an error to the UCMM.

   For example:

   • If a reconfiguration step has timed out, increase the value of the extension property that specifies the timeout for the step.

     For more information, see Node Panic Caused by a Timeout.

3. If the solution to the problem requires a reboot, reboot the node where the problem occurred.

   The solution to only certain problems requires a reboot. For example, increasing the amount of shared memory requires a reboot. However, increasing the value of a step timeout does not require a reboot.

   For more information about how to reboot a node, see Shutting Down and Booting a Single Node in a Cluster in Oracle Solaris Cluster 4.3 System Administration Guide .

4. On the node where the problem occurred, take offline and bring online the Support for Oracle RAC framework resource group.

   This step refreshes the resource group with the configuration changes you made.

   1. Assume the root role or assume a role that provides solaris.cluster.admin RBAC authorization.
   2. Type the command to take offline the Support for Oracle RAC framework resource group and its resources.

      # clresourcegroup offline -n node rac-fmwk-rg

      –n node

      Specifies the node name or node identifier (ID) of the node where the problem occurred.

      rac-fmwk-rg

      Specifies the name of the resource group that is to be taken offline.

   3. Type the command to bring online and in a managed state the Support for Oracle RAC framework resource group and its resources.

      # clresourcegroup online -eM -n node rac-fmwk-rg