Fault monitoring for the HA for Oracle Database data service is provided by the following fault monitors:
The Oracle Database server fault monitor
The Oracle Database listener fault monitor
Each fault monitor is contained in a resource whose resource type is shown in the following table.
|
Standard properties and extension properties of these resources control the behavior of the fault monitors. The default values of these properties determine the preset behavior of the fault monitors. The preset behavior should be suitable for most Oracle Solaris Cluster installations. Therefore, you should tune the HA for Oracle Database fault monitors only if you need to modify this preset behavior.
Tuning the HA for Oracle Database fault monitors involves the following tasks:
Setting the interval between fault monitor probes
Setting the timeout for fault monitor probes
Defining the criteria for persistent faults
Specifying the failover behavior of a resource
For more information, see Tuning Fault Monitors for Oracle Solaris Cluster Data Services in Planning and Administering Data Services for Oracle Solaris Cluster 4.4. Information about the HA for Oracle Database fault monitors that you need to perform these tasks is provided in the subsections that follow.
Tune the HA for Oracle Database fault monitors when you register and configure HA for Oracle Database. For more information, see Registering and Configuring HA for Oracle Database.
The fault monitor for the Oracle Database server uses a request to the server to query the health of the server.
The server fault monitor is started through pmfadm to make the monitor highly available. If the monitor is killed for any reason, the Process Monitor Facility (PMF) automatically restarts the monitor.
The server fault monitor consists of the following processes.
A main fault monitor process
A database client fault probe
This section contains the following information about the server fault monitor:
The main fault monitor determines that an operation is successful if the database is online and no errors are returned during the transaction.
The database client fault probe performs the following operations:
Monitoring the partition for archived redo logs. See Operations to Monitor the Partition for Archived Redo Logs.
If the partition is healthy, determining whether the database is operational. See Operations to Determine Whether the Database is Operational.
The probe uses the timeout value that is set in the resource property Probe_timeout to determine how much time to allocate to successfully probe Oracle Database.
The database client fault probe queries the dynamic performance view v$archive_dest to determine all possible destinations for archived redo logs. For every active destination, the probe determines whether the destination is healthy and has sufficient free space for storing archived redo logs.
If the destination is healthy, the probe determines the amount of free space in the destination's file system. If the amount of free space is less than 10% of the file system's capacity and is less than 20 Mbytes, the probe prints a message to syslog.
If the destination is in ERROR status, the probe prints a message to syslog and disables operations to determine whether the database is operational. The operations remain disabled until the error condition is cleared.
If the partition for archived redo logs is healthy, the database client fault probe queries the dynamic performance view v$sysstat to obtain database performance statistics. Changes to these statistics indicate that the database is operational. If these statistics remain unchanged between consecutive queries, the fault probe performs database transactions to determine if the database is operational. These transactions involve the creation, updating, and dropping of a table in the user table space.
The database client fault probe performs all its transactions as the Oracle Database user. The ID of this user is specified during the preparation of the Oracle Solaris Cluster nodes as explained in How to Prepare the Oracle Solaris Cluster Nodes.
If a database transaction fails, the server fault monitor performs an action that is determined by the error that caused the failure. To change the action that the server fault monitor performs, customize the server fault monitor as explained in Customizing the HA for Oracle Database Server Fault Monitor.
If the action requires an external program to be run, the program is run as a separate process in the background.
Possible actions are as follows:
Ignore. The server fault monitor ignores the error.
Stop monitoring. The server fault monitor is stopped without shutting down the database.
Restart. The server fault monitor stops and restarts the entity that is specified by the value of the Restart_type extension property:
If the Restart_type extension property is set to RESOURCE_RESTART, the server fault monitor restarts the database server resource. By default, the server fault monitor restarts the database server resource.
If the Restart_type extension property is set to RESOURCE_GROUP_RESTART, the server fault monitor restarts the database server resource group.
Switch over. The server fault monitor switches over the database server resource group to another cluster node. If no nodes are available, the attempt to switch over the resource group fails. If the attempt to switch over the resource group fails, the database server is restarted.
Oracle Database logs alerts in an alert log file. The absolute path of this file is specified by the alert_log_file extension property of the SUNW.oracle_server resource. The server fault monitor scans the alert log file for new alerts at the following times:
When the server fault monitor is started
Each time that the server fault monitor queries the health of the server
If an action is defined for a logged alert that the server fault monitor detects, the server fault monitor performs the action in response to the alert.
Preset actions for logged alerts are listed in Table 5, Preset Actions for Logged Alerts. To change the action that the server fault monitor performs, customize the server fault monitor as explained in Customizing the HA for Oracle Database Server Fault Monitor.
The Oracle Database listener fault monitor checks the status of an Oracle Database listener.
If the listener is running, the Oracle Database listener fault monitor considers a probe successful. If the fault monitor detects an error, the listener is restarted.
The listener probe is started through pmfadm to make the probe highly available. If the probe is killed, PMF automatically restarts the probe.
If a problem occurs with the listener during a probe, the probe tries to restart the listener. The value that is set for the resource property retry_count determines the maximum number of times that the probe attempts the restart. If, after trying for the maximum number of times, the probe is still unsuccessful, the probe stops the fault monitor and does not switch over the resource group.
To facilitate troubleshooting of unexplained DBMS timeouts, you can enable the fault monitor to create a core file when a probe timeout occurs. The contents of the core file relate to the fault monitor process. The fault monitor creates the core file in the root (/) directory. To enable the fault monitor to create a core file, use the coreadm command to enable set-id core dumps.
# coreadm -g /var/cores/%f.%n.%p.core -e global -e process \ -e global-setid -e proc-setid -e log
For more information, see the coreadm(8) man page.