Configuring Oracle Clusterware Management With the cluster.oracle.ini File

The information in the cluster.oracle.ini file is used to create Oracle Clusterware resources that manage TimesTen databases, TimesTen processes, user applications, and virtual IP addresses. Create an initialization file called cluster.oracle.ini as a text file.

Note:

See TimesTen Configuration Attributes for Oracle Clusterware for details on all of the attributes that can be used in the cluster.oracle.ini file.

The ttCWAdmin -create command reads this file for configuration information, so the location of the text file must be reachable and readable by ttCWAdmin. The ttCWAdmin utility is used to administer TimesTen active standby pairs in a cluster that is managed by Oracle Clusterware.

It is recommended that you place this file in the TimesTen daemon home directory on the host for the active database. However, you can place this file in any directory or shared drive on the same host as where you run the ttCWAdmin -create command.

The default location for this file is in the timesten_home/conf directory. If you place this file in another location, identify the path of the location with the -ttclusterini option.

The entry name in the cluster.oracle.ini file must be the same as an existing system DSN in the sys.odbc.ini file. For example, [basicDSN] is the entry name in the cluster.oracle.ini file described in Configuring Basic Availability. [basicDSN] must also be the DataStore and Data Source Name data store attributes in the sys.odbc.ini files on each host. For example, the sys.odbc.ini file for the basicDSN DSN on host1 might be:

[basicDSN]
DataStore=/path1/basicDSN
LogDir=/path1/log
DatabaseCharacterSet=AL32UTF8
ConnectionCharacterSet=AL32UTF8

The sys.odbc.ini file for basicDSN on host2 can have a different path, but all other attributes should be the same:

[basicDSN]
DataStore=/path2/basicDSN
LogDir=/path2/log
DatabaseCharacterSet=AL32UTF8
ConnectionCharacterSet=AL32UTF8

The following sections demonstrate sample configurations of the cluster.oracle.ini file:

Configuring Basic Availability

This example shows an active standby pair with no subscribers.

The host for the active database is the first MasterHost defined (host1) and the standby database is the second MasterHost in the list (host2). Each host in the list is delimited by commas. You can include spaces for readability, if desired.

[basicDSN]
MasterHosts=host1,host2

The following is an example of a cluster.oracle.ini file for an active standby pair with one subscriber on host3:

[basicSubscriberDSN]
MasterHosts=host1,host2
SubscriberHosts=host3

Configuring Advanced Availability

Advanced availability involves configuring spare master or subscriber hosts that are idle until needed to replace master or subscriber hosts (used in the active standby pair replication scheme) that either shut down unexpectedly or experience an unrecoverable error.

As mentioned in Configuring Basic Availability, the MasterHosts attribute in the cluster.oracle.ini file configures the hosts that are used as the master nodes. For an active standby pair replication scheme, you only need two master hosts (one to become the active and one to become the standby). In the event of a failure, the host that did not fail becomes the active (if not already the active) and the failed host is recovered and becomes the standby. However, if the failed host cannot be recovered and if you specified more than two hosts as master hosts in the cluster.oracle.ini file, then the next master host in the list can be instantiated to take the place of an unrecoverable master host.

For example, the following shows a configuration of several master hosts. The first two master hosts (host1 and host2) become the active and the standby; the latter two master hosts (host3 and host4) can be used to take the place of either host1 or host2 if either encounter an unrecoverable failure.

MasterHosts=host1,host2,host3,host4

When you configure more than two multiple hosts, you should also configure two virtual IP (VIP) addresses used only by Oracle Clusterware resources that manage TimesTen resources. With these VIP addresses, TimesTen internal processes (those that manage replication) are isolated from any master host changes that may occur because of an unrecoverable host error.

Note:

As described in Create the Oracle Clusterware Resources to Manage Virtual IP Addresses, the Oracle Clusterware resource that manage these VIP addresses (used in advanced availability) are created with the ttCWAdmin -createVIPs command.

These VIP addresses must be different from any other VIP addresses defined for Oracle Clusterware use or any VIP addresses that are to be used by user applications. Furthermore, if an application does use these VIP addresses, then the application may encounter errors when a master host fails (either recoverable or unrecoverable). These VIP addresses cannot be used by a user application as a method for client failover or as a method to isolate themselves if an active database and standby database switch.

Specify two VIP addresses in the MasterVIP parameter, one for each master host in the active standby pair replication scheme. The VIP addresses specified for the TimesTen cluster must be different from any VIP addresses already defined and used by Oracle Clusterware. In particular, the VIP addresses that are created during the Oracle Clusterware install cannot be used with TimesTen.

MasterVIP=192.168.1.1, 192.168.1.2

The following parameters are also associated with advanced availability in the cluster.oracle.ini file:

SubscriberHosts, similar to MasterHosts lists the host names that can contain subscriber databases.
SubscriberVIP, similar to MasterVIP, provides VIP addresses that can be used by TimesTen internally to manage a subscriber node.
VIPInterface is the name of the public network adaptor.
VIPNetMask defines the netmask of the virtual IP addresses.

In the following example, the hosts for the active database and the standby database are host1 and host2. The hosts available for instantiation in case of an unrecoverable error are host3 and host4. There are no subscriber nodes. VIPInterface is the name of the public network adaptor. VIPNetMask defines the netmask of the virtual IP addresses.

[advancedDSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0

The following example configures a single subscriber on host4. There is one extra host defined in SubscriberHosts that can be used for failover of the master databases and one extra node that can be used for failover of the subscriber database. MasterVIP and SubscriberVIP specify the virtual IP addresses defined for the master and subscriber hosts.

[advancedSubscriberDSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4,host5
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0

Ensure that the extra master nodes:

Have TimesTen installed
Have the direct mode application installed if this is part of the configuration. See Implementing Application Failover.

Including Cache Groups in the Active Standby Pair

If the active standby pair replicates one or more AWT or read-only cache groups, set the CacheConnect attribute to y.

This example sets the CacheConnect attribute to y. The example specifies an active standby pair with one subscriber in an advanced availability configuration. The active standby pair replicates one or more cache groups.

[advancedCacheDSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4, host5
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0
CacheConnect=y

Implementing Application Failover

TimesTen integration with Oracle Clusterware can facilitate the failover of a TimesTen application that is linked to any of the databases in the active standby pair.

TimesTen can manage both direct and client/server mode applications that are on the same host as Oracle Clusterware and TimesTen.

The required attributes in the cluster.oracle.ini file for failing over a TimesTen application are as follows:

AppName - Name of the application to be managed by Oracle Clusterware
AppStartCmd - Command line for starting the application
AppStopCmd - Command line for stopping the application
AppCheckCmd - Command line for running an application that checks the status of the application specified by AppName
AppType - Determines the database to which the application is linked. The possible values are Active, Standby, DualMaster, Subscriber (all) and Subscriber[index].

There are also several optional attributes that you can configure, such as AppFailureThreshold, DatabaseFailoverDelay, and AppScriptTimeout. Table A-3 lists and describes all optional attributes and their default values.

The TimesTen application monitor process uses the user-supplied script or program specified by AppCheckCmd to monitor the application. The script that checks the status of the application must be written to return 0 for success and a nonzero number for failure. When Oracle Clusterware detects a nonzero value, it takes action to recover the failed application.

This example shows advanced availability configured for an active standby pair with no subscribers. The reader application is an application that queries the data in the standby database. AppStartCmd, AppStopCmd and AppCheckCmd can include arguments such as start, stop and check commands.

Note:

Do not use quotes in the values for AppStartCmd, AppStopCmd and AppCheckCmd.

[appDSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
AppName=reader
AppType=Standby
AppStartCmd=/mycluster/reader/app_start.sh start
AppStopCmd=/mycluster/reader/app_stop.sh stop
AppCheckCmd=/mycluster/reader/app_check.sh check

You can configure failover for more than one application. Use AppName to name the application and provide values for AppType, AppStartCmd, AppStopCmd and AppCheckCmd immediately following the AppName attribute. You can include blank lines for readability. For example:

[app2DSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0

AppName=reader
AppType=Standby
AppStartCmd=/mycluster/reader/app_start.sh
AppStopCmd=/mycluster/reader/app_stop.sh
AppCheckCmd=/mycluster/reader/app_check.sh

AppName=update
AppType=Active
AppStartCmd=/mycluster/update/app2_start.sh
AppStopCmd=/mycluster/update/app2_stop.sh
AppCheckCmd=/mycluster/update/app2_check.sh

If you set AppType to DualMaster, the application starts on both the active and the standby hosts. The failure of the application on the active host causes the active database and all other applications on the host to fail over to the standby host. You can configure the failure interval, the number of restart attempts, and the uptime threshold by setting the AppFailureInterval, AppRestartAttempts and AppUptimeThreshold attributes. These attributes have default values. For example:

[appDualDSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
AppName=update
AppType=DualMaster
AppStartCmd=/mycluster/update/app2_start.sh
AppStopCmd=/mycluster/update/app2_stop.sh
AppCheckCmd=/mycluster/update/app2_check.sh
AppRestartAttempts=5
AppUptimeThreshold=300
AppFailureInterval=30

Note:

See TimesTen Configuration Attributes for Oracle Clusterware.

Configuring for Recovery When Both Master Nodes Permanently Fail

If both master nodes fail and then come back up, Oracle Clusterware can automatically recover the master databases.

Automatic recovery of a temporary dual failure requires the following:

RETURN TWOSAFE is not specified for the active standby pair.
AutoRecover is set to y.
RepBackupDir specifies a directory on shared storage.
RepBackupPeriod is set to a value greater than 0.

If both master nodes fail permanently, Oracle Clusterware can automatically recover the master databases to two new nodes if the following is true:

Advanced availability is configured (virtual IP addresses and at least four hosts).
The active standby pair does not replicate cache groups.
RETURN TWOSAFE is not specified.
AutoRecover is set to y.
RepBackupDir specifies a directory on shared storage.
RepBackupPeriod must be set to a value greater than 0.

TimesTen first performs a full backup of the active database and then performs incremental backups. You can specify the optional attribute RepFullBackupCycle to manage when TimesTen performs subsequent full backup. By default, TimesTen performs a full backup after every five incremental backups.

If RepBackupDir and RepBackupPeriod are configured for backups, TimesTen performs backups for any master database that becomes active. It does not delete backups that were performed for a database that used to be the active and has become the standby unless the database becomes the active again. Ensure that the shared storage has enough space for two complete database backups. The ttCWAdmin -restore command automatically chooses the correct backup files.

Incremental backups increase the amount of log records in the transaction log files. Ensure that the values of RepBackupPeriod and RepFullBackupCycle are small enough to prevent a large amount of log records in the transaction log file.

This example shows attribute settings for automatic recovery.

[autorecoveryDSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
AutoRecover=y
RepBackupDir=/shared_drive/dsbackup
RepBackupPeriod=3600

If you have cache groups in the active standby pair or prefer to recover manually from failure of both master hosts, ensure that AutoRecover is set to n (the default). Manual recovery requires the following:

RepBackupDir specifies a directory on shared storage
RepBackupPeriod must be set to a value greater than 0

This example shows attribute settings for manual recovery. The default value for AutoRecover is n, so it is not included in the file.

[manrecoveryDSN]
MasterHosts=host1,host2,host3
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
RepBackupDir=/shared_drive/dsbackup
RepBackupPeriod=3600

Using the RepDDL Attribute

The RepDDL attribute represents the SQL statement that creates the active standby pair.

The RepDDL attribute is optional. You can use it to exclude tables, cache groups and sequences from the active standby pair.

If you include RepDDL in the cluster.oracle.ini file, do not specify ReturnServiceAttribute, MasterStoreAttribute or SubscriberStoreAttribute in the cluster.oracle.ini file. Include those replication settings in the RepDDL attribute.

When you specify a value for RepDDL, use the <DSN> macro for the database file name prefix. Use the <MASTERHOST[1]> and <MASTERHOST[2]> macros to specify the master host names. TimesTen substitutes the correct values from the MasterHosts or MasterVIP attributes, depending on whether your configuration uses virtual IP addresses. Similarly, use the <SUBSCRIBERHOST[n]> macro to specify subscriber host names, where n is a number from 1 to the total number of SubscriberHosts attribute values or 1 to the total number of SubscriberVIP attribute values if virtual IP addresses are used.

Use the RepDDL attribute to exclude tables, cache groups, and sequences from the active standby pair:

[excludeDSN]
MasterHosts=host1,host2,host3,host4
SubscriberHosts=host5,host6
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0
RepDDL=CREATE ACTIVE STANDBY PAIR \
<DSN> ON <MASTERHOST[1]>, <DSN> ON <MASTERHOST[2]>
SUBSCRIBER <DSN> ON <SUBSCRIBERHOST[1]>\
EXCLUDE TABLE pat.salaries, \
EXCLUDE CACHE GROUP terry.salupdate, \
EXCLUDE SEQUENCE ttuser.empcount

The replication agent transmitter obtains route information as follows, in order of priority:

From the ROUTE clause in the RepDDL setting, if a ROUTE clause is specified. Do not specify a ROUTE clause if you are configuring advanced availability.
From Oracle Clusterware, which provides the private host names and public host names of the local and remote hosts as well as the remote daemon port number. The private host name is preferred over the public host name. If the replication agent transmitter cannot connect to the IPC socket, it attempts to connect to the remote daemon using information that Oracle Clusterware maintains about the replication scheme.
From the active and standby hosts. If they fail, then the replication agent chooses the connection method based on host name.

This is an example of specifying the ROUTE clause in RepDDL:

[routeDSN]
MasterHosts=host1,host2,host3,host4
RepDDL=CREATE ACTIVE STANDBY PAIR \
<DSN> ON <MASTERHOST[1]>, <DSN> ON <MASTERHOST[2]>\
ROUTE MASTER <DSN> ON <MASTERHOST[1]>  SUBSCRIBER <DSN> ON <MASTERHOST[2]>\
MASTERIP "192.168.1.2" PRIORITY 1\
SUBSCRIBERIP "192.168.1.3" PRIORITY 1\ 
MASTERIP "10.0.0.1" PRIORITY 2\
SUBSCRIBERIP "10.0.0.2" PRIORITY 2\
MASTERIP "140.87.11.203" PRIORITY 3\
SUBSCRIBERIP "140.87.11.204" PRIORITY 3\
ROUTE MASTER <DSN> ON <MASTERHOST[2]>  SUBSCRIBER <DSN> ON <MASTERHOST[1]>\
MASTERIP "192.168.1.3" PRIORITY 1\
SUBSCRIBERIP "192.168.1.2" PRIORITY 1\ 
MASTERIP "10.0.0.2" PRIORITY 2\
SUBSCRIBERIP "10.0.0.1" PRIORITY 2\
MASTERIP "140.87.11.204" PRIORITY 3\
SUBSCRIBERIP "140.87.11.203" PRIORITY 3\