17 Oracle Beehive Disaster Recovery with Data Guard

Using Oracle Data Guard, you can configure your Oracle Beehive deployment for disaster recovery, in which end-user traffic is diverted to a standby system in the event of a failure in the primary system.

This module contains the following topics:

Prerequisites
Configuring Oracle Beehive for Disaster Recovery
Performing Role Transitions
Tuning Options

Prerequisites

This module assumes you are familiar with installing, configuring, and maintaining Oracle databases, Oracle Beehive, and Oracle Data Guard.

To configure a disaster recovery system for Oracle Beehive, your primary deployment can either be a single-instance Oracle Beehive server, or a deployment with multiple Oracle Beehive servers. Likewise, you can use either a single instance Oracle Database, or a multi-instance RAC deployment, as your primary database.

Configuring Oracle Beehive for Disaster Recovery

These configuration steps show how to take advantage of service relocation for a switchover or failover to a local or remote standby deployment.

Planned and Unplanned Outages

In a planned outage (switchover) scenario, you perform manual steps to reverse database roles between the standby system and the primary system, and then shut down the (former) primary system. This is useful to maintain uptime when performing maintenance on the primary system, such as replacing hardware, that requries the servers to be shut down.

In a unplanned outage (failover) scenario, a failure of the primary system is detected and the standby system is made primary. Depending on your Oracle Data Guard configuration, manual steps may be required, or the failover may be performed automatically.

Database-Only versus Full Deployment

In an Oracle Beehive database-only recovery deployment, you create a standby deployment of the database used by Oracle Beehive. This provides for reduced downtime for planned and unplanned primary database outages. In this scenario, you use your primary deployment's Application tiers with the standby database, so no switchover or failover of the Oracle Beehive Application tier occurs.

In a full deployment scenario, you create a standby deployment of both the database and the Application tier(s). In a switchover or failover, you start the Oracle Beehive Application tiers on the standby system, and then redirect user traffic to the standby Oracle Beehive deployment. This provides for reduced downtime for planned and unplanned outages of the entire primary deployment.

Example Deployments

Figure Figure 17-1 shows a typical full-deployment scenario. In this scenario, a deployment with multiple Application tiers and database instances is duplicated by the standby deployment.

Figure 17-1 Oracle Beehive Full-Deployment Disaster Recovery Scenario

Description of "Figure 17-1 Oracle Beehive Full-Deployment Disaster Recovery Scenario"

Figure Figure 17-2 shows a typical database-only deployment. In this scenario, there is a standby database that duplicates the primary multi-node database, but there are no standby Application tiers. The primary Application tiers remain active and connect to the standby database. This example provides database redundancy and failover in the event of a failure in the primary database, but there is no provision for Application-tier failover.

Figure 17-2 Oracle Beehive Database-Only Recovery Scenario

Description of "Figure 17-2 Oracle Beehive Database-Only Recovery Scenario"

To configure Oracle Beehive for use in these scenarios, perform the following procedures:

Recording Environment Details
Configuring Service Relocation and Database Triggers
Configuring Oracle Beehive Application Tiers

Recording Environment Details

Having a record of the details of your primary and standby envronments will be useful during setup, configuration, and maintenance of your disaster-recovery configuration.

The following is the sample environment used in examples in this chapter. The example Primary deployment consists of a two-node RAC database and a Beehive deployment using two Application tiers. The Standby deployment duplicates this configuration.

Tip:

Refer back to this section as you read through the rest of this chapter. In all of the examples, you can use the example values to see what your actual values should be.

Example Deployment

Bootstrap Service: A common service defined on all nodes, specifically used for service relocation. In this chapter, beehivedg is the example common service name on all nodes
Primary:
- RAC Nodes (and Virtual IP): These are defined in the DNS and/or in /etc/hosts. In this chapter, pr_db1 (pr_db1_vip) and pr_db2 (pr_db2_vip) are the example primary database nodes
- ORACLE_SID: In this chapter, pr_orcl1 and pr_orcl2 are the example ORACLE_SIDs
- Application tiers: In this chapter, pr_app1 and pr_app2 are the example primary Application tiers
Standby:

See also:
MAA 10g Setup Guide: Creating a RAC Physical Standby Database for a RAC Primary Database
- RAC Nodes (and Virtual IP): In this chapter, stby_db1 (stby_db1_vip) and stby_db2 (stby_db2_vip) are the example standby database nodes
- ORACLE_SID: In this chapter, stby_orcl1 and stby_orcl2 are the example standby ORACLE_SIDs
- Application tiers: In this chapter, stby_app1 and stby_app2 are the example standby Application tiers
IP Addresses: You may find it useful to make a note of the IP addresses for each database node. IP addresses can be used as a RAC configuration method, or as a part of the DNS and/or /etc/hosts file.

Configuring Service Relocation and Database Triggers

To configure service relocation and database triggers, perform the steps in the following sections:

Create Managed Services
Create Database Services Startup Trigger
Setup FAN ONS Publisher and Database Role Change Trigger

Create Managed Services

Perform the following steps to create managed services:

If you are using a RAC database deployment, add a CRS-managed bootstrap service for the primary and standby databases by running the following commands:

For the primary, run the following command on any primary RAC node:
```
srvctl add service -s beehivedg -d pr_orcl1 -r pr_orcl1,pr_orcl2
```
For the standby, run the following command on any standby RAC node:
```
srvctl add service -s beehivedg -d stby_orcl1 -r stby_orcl1,stby_orcl2
```
Note:
This service name must be different than the name used in the database parameter SERVICE_NAMES. In this example, beehivedg is the service name.
If you are using a single-instance database deployment, create the relocation service using the dbms_service package. In SQL*Plus, connect as SYS or SYSDBA and enter the following:
```
begin
    dbms_service.create_service(service_name => 'beehivedg',
                            network_name => 'beehivedg' );
  end;
   /
```

Create Database Services Startup Trigger

When a switchover or failover to a standby database occurs, service relocation and affinity services or other CRS-managed services are not automatically started. Perform the following steps to create a trigger to start up database-related services. (In a full deployment scenario, you do not need to perform this procedure.)

If you have a multiple instance deployment, create a script to start CRS services. Example 17-1 shows an example script, which includes using affinity services, for a multiple instance deployment. Example 17-2 shows a similar script which does not include using affinity services:

Note:

Instructions for setting up affinity services are included in the RAC post-install steps in the Oracle Beehive Installation Guide for your platform.

Example 17-1 Example CRS-Managed Service Startup Script With Affinity Services StartupService.sh

#!/bin/ksh
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.1.0/db
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/ctx/lib
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/opmn/bin:/usr/local/bin:/bin:/usr/bin
export TIMESTAMP=`date '+%y%m%d%H%M%s'`
export LOGFILE=/tmp/StartupService_${TIMESTAMP}.log
 
echo "Starting $0 at `date`" > $LOGFILE
HOSTNAME=`hostname | awk -F. '{print $1}'`
case  $HOSTNAME in
   pr_db1) 
export DB_UNIQUE_NAME=bhha
export ORACLE_SID=bhha1
export AFF_SVC=aff1 
;;
   pr_db2) 
export DB_UNIQUE_NAME=bhha
export ORACLE_SID=bhha2
export AFF_SVC=aff2 
;;
   stby_db1) 
export DB_UNIQUE_NAME=bhhb
export ORACLE_SID=bhhb1
export AFF_SVC=aff1 
;;
   stby_db2) 
export DB_UNIQUE_NAME=bhhb
export ORACLE_SID=bhhb2
export AFF_SVC=aff2 
;;
    *) echo "Node $HOSTNAME not identified or not defined for database bhha " >> $LOGFILE 
       exit ;;
esac
 
echo "`date` : Starting CRS managed database service $AFF_SVC" >> $LOGFILE
srvctl start service -d $DB_UNIQUE_NAME -s $AFF_SVC -i $ORACLE_SID >> $LOGFILE 2>&1
srvctl start service -d $DB_UNIQUE_NAME -s beehivedg -i $ORACLE_SID >> $LOGFILE 2>&1
echo "`date` : Start of CRS managed database service $AFF_SVC complete" >> $LOGFILE

Example 17-2 Example CRS-Managed Service Startup Script Without Affinity Services StartupService.sh

#!/bin/ksh
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.1.0/db
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/ctx/lib
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/opmn/bin:/usr/local/bin:/bin:/usr/bin
export TIMESTAMP=`date '+%y%m%d%H%M%s'`
export LOGFILE=/tmp/StartupService_${TIMESTAMP}.log
 
echo "Starting $0 at `date`" > $LOGFILE
HOSTNAME=`hostname | awk -F. '{print $1}'`
 
echo "`date` : Starting beehivedg database service" >> $LOGFILE
srvctl start service -d $DB_UNIQUE_NAME -s beehivedg -i $ORACLE_SID >> $LOGFILE 2>&1
echo "`date` : Start of beehivedg database service complete" >> $LOGFILE

Create a database startup trigger on the primary database (Oracle Data Guard will subsequently replicate your trigger on the standby database). For example, for a multiple instance deployment:

CREATE OR REPLACE TRIGGER manage_services_start after startup on database

DECLARE
   role VARCHAR(30);
BEGIN
   SELECT DATABASE_ROLE INTO role FROM V$DATABASE;
   IF role = 'PRIMARY' THEN
      dbms_scheduler.create_job(
      job_name=>'start_dbsvc',
      job_type=>'executable',
      job_action=>'<ORACLE_HOME>/bin/StartupService.sh',
      enabled=>TRUE
      );
   END IF;
END;
/

job_action should specify the location of the script you created in Step 1

Replace <ORACLE_HOME> with the full path of your Oracle Database Oracle home.

For a single instance deployment (in which you do not need a script):

CREATE OR REPLACE TRIGGER manage_services_start after startup on database
 
DECLARE
  role VARCHAR(30);
BEGIN
  SELECT DATABASE_ROLE INTO role FROM V$DATABASE;
  IF role = 'PRIMARY' THEN
       DBMS_SERVICE.START_SERVICE('beehivedg');
  ELSE
       DBMS_SERVICE.STOP_SERVICE('beehivedg');
  END IF;
END;
/

To test the script, you can restart the database (in a RAC deployment, restart the primary RAC node database). The related services should start automatically:
```
srvctl stop database -d pr_db1
srvctl start database -d pr_db1
srvctl status service -d pr_db1
```

Setup FAN ONS Publisher and Database Role Change Trigger

You must configure the FAN ONS Publisher and Database Role Change trigger in order to enable database failover operations. (In a "full deployment" scenario, there is no switchover or failover betweeen database deployments, so you can skip this step.)

See Also:

Pages 11-13 of "Client Failover in Data Guard Configurations for Highly Available Oracle Databases: Oracle Database 10g Release 2" Oracle Maximum Availability Architecture White Paper, found at the following link on the Oracle Technology Network website:

http://www.otn.oracle.com/goto/maa

Configure the FAN ONS Publisher Configuration File in $ORACLE_HOME/dbs/cfo${ORACLE_SID}.ora (on both the primary and the standby deployments).

For example (including affinity):

bhha peer=stby_orcl
bhhb peer=pr_orcl
bhha service=beehivedg location=pr_db1,pr_orcl1:pr_db2,pr_orcl2
bhha service=aff1 location=pr_db1,pr_orcl1
bhha service=aff2 location=pr_db2,pr_orcl2
bhhb service=beehivedg location=stby_db1,stby_orcl1:stby_db2,stby_orcl2
bhhb service=aff1 location=stby_db1,stby_orcl1
bhhb service=aff2 location=stby_db2,stby_orcl2

On all prirmary and standby RAC nodes, copy the $ORACLE_HOME/dbs/cfo${ORACLE_SID}.ora file to each node and rename it with the appropriate ORACLE_SID

Create a database role change trigger:

CREATE OR REPLACE TRIGGER ons_JDBCpublish AFTER DB_ROLE_CHANGE ON DATABASE 
BEGIN 
dbms_scheduler.create_job( 
job_name=>'publish_events', 
job_type=>'executable', 
job_action=>'/u01/app/oracle/product/11.1.0/db/bin/cfo.sh', 
enabled=>TRUE 
); 
END;

Configuring Oracle Beehive Application Tiers

In a database-only deployment, configure your Oracle Beehive servers by performing the following configuration steps:

Note:

You only need to perform these steps on one Application tier.

Update bootstrap connect_string to use service relocation:

Run beectl stop --all

Run the following command:

beectl modify_bootstrap_configuration --connect_string \
"(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=yes)(FAILOVER=on)\
(ADDRESS=(PROTOCOL=TCP)(HOST=pr_db1_vip.example.com)(PORT=1521))\
(ADDRESS=(PROTOCOL=TCP)(HOST=pr_db2_vip.example.com)(PORT=1521))\
(ADDRESS=(PROTOCOL=TCP)(HOST=stby_db1_vip.example.com)(PORT=1521))\
(ADDRESS=(PROTOCOL=TCP)(HOST=stby_db2_vip.example.com)(PORT=1521)))\
(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=beehivedg.us.oracle.com)))

Update the ONS entries by modifying the ONS setup to include the standby nodes:

Note:

6200 is the current setting for remoteport in the $CRS_HOME/opmn/conf/ons.config file and should also show in the $CRS_HOME/bin/onsctl ping output

beectl> modify_property --component _current_site:Database \
--name OnsNodeConfiguration \
--value pr_db1.example.com:6200 \
--value pr_db2.example.com:6200 \
--value stby_db1.example.com:6200 \
--value stby_db2.example.com:6200

Activate the Configuration on the first Beehive Application tier by running the following commands:
```
beectl activate_configuration
beectl modify_local_configuration_files
beectl start --all
```

One at a time, run the following commands on each remaining Application tier instance:

beectl stop -all
beectl modify_local_configuration_files -restart_needed false
beectl start --all

Performing Role Transitions

An Oracle Data Guard configuration consists of one database that functions in the primary role and one or more databases that function in the standby role. Typically, the role of each database does not change. However, if Data Guard is used to maintain service in response to a primary database outage, you must initiate a role transition between the current primary database and one standby database in the configuration.

A database operates in one of the following mutually exclusive roles: primary or standby. Data Guard enables you to change these roles dynamically by issuing SQL statements, or by using either of the Data Guard broker's interfaces.

See Also:

Chapter 1, "Oracle Data Guard Broker Concepts," of Oracle Data Guard Broker

Oracle Data Guard supports the following role transitions:

Switchover

Allows the primary database to switch roles with one of its standby databases. There is no data loss during a switchover. After a switchover, each database continues to participate in the Data Guard configuration with its new role. Switchover is generally used for planned outages.
Failover

Changes a standby database to the primary role in response to a primary database failure. If the primary database was not operating in either maximum protection mode or maximum availability mode before the failure, some data loss may occur. If Flashback Database is enabled on the primary database, it can be reinstated as a standby for the new primary database once the reason for the failure is corrected. Failover is used when unplanned outages occur.

Performing a Switchover

To perform a database-only switchover operation follow the steps in MetaLink Note 751600.1.

To perform a switchover operation in a full-deployment scenario, perform the following steps:

Stop Oracle Beehive on the primary site application nodes
Follow the steps in MetaLink Note 751600.1
Start the Application tiers in your standby deployment
Once the standby Oracle Beehive system is up and running, you can redirect user traffic to the standby deployment

Performing a Failover

To perform a database-only switchover operation follow the steps in section 8.2.2, "Performing a Failover to a Physical Standby Database," in Chapter 8 of the Oracle Data Guard Concepts and Administration Guide.

To perform a switchover operation in a full-deployment scenario, perform the following steps:

Stop Oracle Beehive on the primary site application nodes
Follow the steps in section 8.2.2, "Performing a Failover to a Physical Standby Database," in Chapter 8 of the Oracle Data Guard Concepts and Administration Guide
Start the Application tiers in your standby deployment
Once the standby Oracle Beehive system is up and running, you can redirect user traffic to the standby deployment

Tuning Options

This section includes the following topics

Modifying Outbound Connect Timeout
Enabling Listener Throttling on Each Database Node

Modifying Outbound Connect Timeout

You can modify the outbound connect timeout.

See Also:

Page 5 of "Client Failover in Data Guard Configurations for Highly Available Oracle Databases: Oracle Database 10g Release 2" Oracle Maximum Availability Architecture White Paper, found at the following link on the Oracle Technology Network website:

http://www.otn.oracle.com/goto/maa

For example, to modify the outbound connect timeout to three seconds:

beectl modify_property --component _current_site:Database:DefaultNonXaPool \
--name ConnectionProperties \
--value oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR:3000

Enabling Listener Throttling on Each Database Node

Listener throttling is a performance adjustment you can make to the database listener that may be necessary for the standby database to operate efficiently during and after a switchover or failover. When the system initially moves from the primary to the standby, there will be a very high number of connections moved all at once to the standby system, which can cause "connection storm." Listener throttling is a method of controlling the number of simultaneous connections made to the database.

The appropriate value to use depends on the size of your deployment (number of concurrent users), and the capability of your database hardware.

See Also:

For complete details of this procedure, see Appendix E, "Listener Connection Rate Throttling," of "Optimizing Availability During Unplanned Outages Using Oracle Clusterware and RAC " Oracle Maximum Availability Architecture White Paper, found at the following link on the Oracle Technology Network website:

http://www.otn.oracle.com/goto/maa

The CONNECTION_RATE_<listener_name> parameter indicates the connection rate per second for all addresses with RATE_LIMIT set to YES, yes, or a specific rate.

In this example, 10 connections will be spawned per second only for the first address in the address list. The RATE_LIMIT parameter can also specify the rate itself instead of only turning throttling on at the default rate. For example, RATE_LIMIT=5 on a particular address indicates a specific setting for just that address. When specifying a RATE_LIMIT at the address level, you should not specify the CONNECTION_RATE_<listener_name> property.

Note:

If both the global rate and a specific rate are specified, the global rate is the one enforced.

Perform the following procedure to enable listener throttling on each database node:

Edit the listener.ora file. In Example 17-3, only the first address has throttling enabled. Note that the setting of 10 is an example; the actual setting depends on the details of your environment

Example 17-3 Example listener.ora File for Listener Throttling

LISTENER_PR_DB1 = 
(DESCRIPTION_LIST = 
(DESCRIPTION = 
(ADDRESS_LIST = 
(ADDRESS = (PROTOCOL = TCP)(HOST = pr_db1_vip)(PORT = 1521)(RATE_LIMIT=YES)(QUEUESIZE=1024)(IP = FIRST)) 
) 
(ADDRESS_LIST = 
(ADDRESS = (PROTOCOL = TCP)(HOST = <X.X.X.X>)(PORT = 1521)(IP = FIRST)) 
) 
(ADDRESS_LIST = 
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) 
) 
) 
) 
CONNECTION_RATE_LISTENER_PR_DB1=30

Substitute the correct IP address for <X.X.X.X>.

Implement the listener.ora changes by running the following command:
```
lsnrctl reload
```