Clusterware Management
There are certain procedures for managing clusterware when used in conjunction with TimesTen.
This section includes the following topics:
Changing User Names or Passwords When Using Oracle Clusterware
When you create the active standby pair replication scheme with the ttCWAdmin
-create
command, Oracle Clusterware prompts for the required user names and
passwords in order to manage the TimesTen environment.
Oracle Clusterware stores these user names and passwords. After modifying
any user name or password, you must run the ttCWAdmin
-reauthenticate
command to enable Oracle Clusterware to store these new
user names and passwords.
Managing Hosts in the Cluster
The following sections describe how to add or remove hosts when using a cluster.
Adding a Host to the Cluster
Adding a host requires that the cluster be configured for advanced availability.
The examples in this section use the
advancedSubscriberDSN
.
To add two spare master hosts to a cluster, enter a command similar to the following:
ttCWAdmin -addMasterHosts -hosts "host8,host9" -dsn advancedSubscriberDSN
To add a spare subscriber host to a cluster, enter a command similar to the following:
ttCWAdmin -addSubscriberHosts -hosts "subhost1" -dsn advancedSubscriberDSN
Removing a Host From the Cluster
Removing a host from the cluster requires that the cluster be configured for advanced availability.
MasterHosts
must list more than two hosts if one of the
master hosts is to be removed. SubscriberHosts
must list at least one
more host than the number of subscriber databases if one of the subscriber hosts is to
be removed.
The examples in this section use the advancedSubscriberDSN
.
To remove two spare master host from the cluster, enter a command similar to the following:
ttCWAdmin -delMasterHosts "host8,host9" -dsn advancedSubscriberDSN
To remove a spare subscriber hosts from the cluster, enter a command similar to the following:
ttCWAdmin -delSubscriberHosts "subhost1" -dsn advancedSubscriberDSN
Managing Active Standby Pairs in a Cluster
The following sections describe how to add or remove an active standby pair to a cluster.
Adding an Active Standby Pair to a Cluster
You can add an active standby pair (with or without subscribers) to a cluster that is already managing an active standby pair.
Managing Read-Only Subscribers in the Active Standby Pair
The following sections describe how to manage read-only subscribers in the active standby pair that is managed by Oracle Clusterware.
Adding a Read-Only Subscriber Managed by Oracle Clusterware
To add a read-only subscriber that is to be managed by Oracle Clusterware to an active standby pair replication scheme, perform these steps:
Removing a Read-Only Subscriber Managed by Oracle Clusterware
To remove a read-only subscriber that is managed by Oracle Clusterware from an active standby pair, perform these steps:
Adding or Dropping a Read-Only Subscriber Not Managed by Oracle Clusterware
You can add or drop a read-only subscriber that is not managed by Oracle Clusterware to or from an existing active standby pair replication scheme that is managed by Oracle Clusterware.
Using the ttCWAdmin -beginAlterSchema
command enables
you to add a subscriber without dropping and re-creating the replication scheme.
Oracle Clusterware does not manage the subscriber, because it is not part of the
configuration that was set up for Oracle Clusterware management.
Perform these steps:
If you added a subscriber, ensure that the read-only subscriber is included if the
cluster is dropped and re-created by adding the RemoteSubscriberHosts Oracle Clusterware attribute for the read-only subscriber in the
cluster.oracle.ini
file as described in Step 1 in Configure a Read-Only Subscriber That Is Not Managed by Oracle Clusterware. Alternatively, if you dropped a subscriber, remove the
RemoteSubscriberHosts
Oracle Clusterware attribute for the
dropped subscriber in the cluster.oracle.ini
file (if it is
configured).
Rebuilding a Read-Only Subscriber Not Managed by Oracle Clusterware
Perform the following tasks to destroy and rebuild a read-only subscriber that is not managed by Oracle Clusterware:
- Stop the replication agent on the subscriber host.
- Use the
ttDestroy
utility to destroy the subscriber database. - On the subscriber host, use
ttRepAdmin -duplicate
to duplicate the standby database to the read-only subscriber. See Duplicating a Database.
Reversing the Roles of the Master Databases
After a failover, the active and standby databases are on different hosts than they
were before the failover. You can use the -switch
option of the
ttCWAdmin
utility to restore the original configuration.
Optionally, you can also use the -timeout
option with the
-switch
option to set a timeout for the number of seconds to wait
for the active and standby database switch to complete.
For example:
ttCWAdmin -switch -dsn basicDSN
Ensure that there are no open transactions before using the -switch
option. If there are open transactions, the command fails.
Note:
See ttCWAdmin in the Oracle TimesTen In-Memory Database Reference.
Figure 8-6 shows the hosts for an active standby pair. The active database resides on host A, and the standby database resides on host B.
Figure 8-6 Hosts for an Active Standby Pair
Description of "Figure 8-6 Hosts for an Active Standby Pair"
The ttCWAdmin -switch
command performs these tasks:
-
Deactivates the TimesTen cluster agent (
ttCRSAgent
) on host A (the active master). -
Disables the TimesTen database monitor (
ttCRSmaster
) on host A. -
Calls the
ttRepSubscriberWait
,ttRepStop
andttRepDeactivate
built-in procedures on host A. -
Stops the active service (
ttCRSActiveService
) on host A and reports a failure event to the Oracle ClusterwareCRSD
process. -
Enables monitoring on host A and moves the active service to host B.
-
Starts the replication agent on host A, stops the standby service (
ttCRSsubservice
) on host B and reports a failure event to the Oracle ClusterwareCRSD
process on host B. -
Starts the standby service (
ttCRSsubservice
) on host A.
Modifying Connection Attribute Values
When you modify connection attributes across an active standby pair with subscribers, the connection attributes must be modified on all hosts within this configuration.
Note:
You cannot modify any DATASTORE
connection attributes since they are only allowed to be set at data store creation time. For example, this procedure can be used to change the PermSize
value.
Use the ttCWAdmin -beginAlterSchema
and
-endAlterSchema
commands to facilitate the change of any connection
attribute values on the active and standby databases and any subscribers.
-
The
ttCWAdmin -beginAlterSchema
command suspends the Oracle Clusterware management and stops the replication agents on the active and standby databases and any subscriber databases in preparation for any changes. -
After you complete all changes, the
ttCWAdmin -endAlterSchema
command resumes Oracle Clusterware management and restarts all replication agents on the active and standby databases and any subscriber databases.
Perform the following tasks when altering any connection attributes for the active standby pair when using Oracle Clusterware:
-
Suspend Oracle Clusterware and stop all replication agents for the active and standby databases with the
ttCWAdmin -beginAlterSchema
command.The active database continues to accept requests and updates, but any changes are not propagated to the standby database and any subscribers until the replication agents are restarted.
The
ttCWAdmin -beginAlterSchema
command also changes the RAM policy temporarily for the standby database and all subscriber databases to InUse with RamGrace where the grace period is set for 60 seconds to enable these databases to be unloaded by TimesTen. Once the standby and subscriber databases are unloaded from memory, the connection attributes for these databases can be modified.ttCWAdmin -beginAlterSchema -dsn advancedDSN
-
Disconnect any application connections and wait for the standby and subscriber databases to unload from memory (based on the RAM policy).
Once the standby and subscriber databases are unloaded from memory, alter any connection attributes, such as
PermSize
, on the hosts for the standby and all subscriber databases in their respectivesys.odbc.ini
files. -
Resume Oracle Clusterware and restart all replication agents for the active and standby databases with the
ttCWAdmin -endAlterSchema
command. The configured RAM policy for each TimesTen database is set back to always. The active database propagates any transactions that occurred while the standby database and subscribers were down.ttCWAdmin -endAlterSchema -dsn advancedDSN
Note:
Wait an appropriate amount of time for all changes to propagate from the active database to the standby database and all subscribers before performing the next step.
The only host that has not had the connection attribute change is the active database. You will switch the active database with the standby database so that you can modify the connection attributes on this host.
-
Suspend all application workload and disconnect all applications on the active database.
-
Switch the active and standby databases with the
ttCWAdmin -switch
command.ttCWAdmin -switch -dsn advancedDSN
Note:
-
Suspend Oracle Clusterware and stop all replication agents for all databases with the
ttCWAdmin -beginAlterSchema
command.The new active database may still accept requests and updates, but any changes are not propagated to the new standby database and any subscribers.
The RAM policy changes for the new standby database (and all subscriber databases) to inUse with RamGrace where the grace period is set for 60 seconds to enable these databases to be unloaded by TimesTen.
ttCWAdmin -beginAlterSchema -dsn advancedDSN
-
Wait for the new standby database to unload from memory. Once unloaded, alter the same connection attributes, such as
PermSize
, on the new standby database in itssys.odbc.ini
file. The connection attributes are now modified on all hosts. -
Run the
ttCWAdmin -endAlterSchema
command to resume Oracle Clusterware management and restart the replication agents on the active and standby databases. The configured RAM policy resumes to always.ttCWAdmin -endAlterSchema -dsn advancedDSN
-
Suspend all application workload and disconnect all applications on the active database.
-
If desired, you can switch the active and standby databases with the
ttCWAdmin -switch
command to restore the active standby pair to the original configuration.ttCWAdmin -switch -dsn advancedDSN
Managing the TimesTen Database RAM Policy
By default, the TimesTen database RAM policy is set to always when Oracle Clusterware manages the TimesTen database. However, if you stop Oracle Clusterware management, the TimesTen database RAM policy is set to inUse.
If you no longer use Oracle Clusterware to manage TimesTen, you should set the TimesTen RAM policy to what is appropriate for your environment. Typically, the recommended setting is manual.
See Specifying a RAM Policy in the Oracle TimesTen In-Memory Database Operations Guide.
Changing the Schema
When using Oracle Clusterware to manage an active standby pair, you can modify the schema by running DDL statements as in a normal replication environment, except that Oracle Clusterware must start and stop all replication agents, when it is necessary to do so.
Thus, when you change the schema, note the following:
-
For those DDL statements on objects that are automatically replicated, you do not need to stop the replication agents. In this case, no further action is required, since these DDL statements are automatically propagated and applied to the standby database and any subscribers. The
DDLReplicationLevel
connection attribute controls what DDL statements are replicated. -
For those objects that are a part of the replication scheme, but any DDL statements processed on these objects are not replicated (these objects are listed in Making Other Changes to an Active Standby Pair), run the Oracle Clusterware
ttCWAdmin -beginAlterSchema
command on the active database, which suspends any Oracle Clusterware management and stops the replication agents on each node in the replication scheme. Then, run the DDL statement on the active database in the replication scheme. Finally, run the Oracle ClusterwarettCWAdmin -endAlterSchema
command on the active database to restart all replication agents.Because these objects are a part of the replication scheme, but the DDL statements are not replicated, a duplicate occurs after the
ttCWAdmin -endAlterSchema
command to propagate these schema changes to the standby database and any subscribers. This is the only scenario when a duplicate is used to propagate the schema changes.Follow the instructions described in Facilitating Schema Change for Oracle Clusterware.
-
For those DDL statements on objects that are not automatically replicated and are not part of the replication scheme, run the Oracle Clusterware
ttCWAdmin -beginAlterSchema
command on the active database, which suspends any Oracle Clusterware management and stops and the replication agents on all nodes. Then, you can synchronize all nodes by manually running these DDL statements as indicated in Making DDL Changes in an Active Standby Pair. Finally, run the Oracle ClusterwarettCWAdmin -endAlterSchema
command on the active database to restart all replication agents.Follow the instructions described in Facilitating Schema Change for Oracle Clusterware.
Note:
The Making DDL Changes in an Active Standby Pair and Making Other Changes to an Active Standby Pair sections describe which DDL statements are and are not automatically replicated for an active standby pair. These sections also describe what objects are a part of the replication scheme.
Facilitating Schema Change for Oracle Clusterware
Use the ttCWAdmin -beginAlterSchema
and
-endAlterSchema
commands to facilitate a schema change on the active
and standby databases.
-
The
ttCWAdmin -beginAlterSchema
command suspends the Oracle Clusterware management and stops replication agents on both the active and standby databases in preparation for any schema changes. -
After you complete all schema changes, run the
ttCWAdmin -endAlterSchema
command. For those objects that are a part of the replication scheme, but any DDL statements processed on these objects are not automatically replicated, a duplicate occurs after thettCWAdmin -endAlterSchema
command to propagate only these schema changes to the standby database and any subscribers. This command registers the altered replication scheme, restarts the replication agents on the active and standby databases, and reinstates Oracle Clusterware control.
Perform the following tasks when altering the schema of the active standby pair when using Oracle Clusterware:
-
Suspend Oracle Clusterware and stop the replication agents on both the active and standby databases.
ttCWAdmin -beginAlterSchema -dsn advancedDSN
-
Make any desired schema changes.
If you create, alter, or drop any objects where the DDL for these objects are not replicated, you should also manually create, alter, or drop the same objects on the standby database and subscribers while the replication agents are inactive to ensure that the same objects exist on all databases in the replication scheme. For example, if you create a materialized view on the active database, create the materialized view on the standby and subscriber databases at this time.
-
If the object is not automatically replicated but is a part of the replication scheme, (such as a sequence) and you want to include it in the active standby pair replication scheme, alter the active standby pair.
ALTER ACTIVE STANDBY PAIR INCLUDE samplesequence;
-
If the object is a cache group, see Making Schema Changes to Cache Groups for instructions to create, alter, or drop a cache group.
-
Run the
ttCWAdmin -endAlterSchema
command to resume Oracle Clusterware and restart the replication agents on the active and standby databases. If you modified objects that are a part of the replication scheme, but any DDL statements processed on these objects are not automatically replicated, a duplicate occurs after thettCWAdmin -endAlterSchema
command to propagate only these schema changes to the standby database and any subscribers.ttCWAdmin -endAlterSchema -dsn advancedDSN
Making Schema Changes to Cache Groups
Use the ttCWAdmin -beginAlterSchema
and
-endAlterSchema
commands to facilitate the schema changes on cache
groups within Oracle Clusterware.
Add a Cache Group
You can add a cache group on the active database of the active standby pair.
Perform these steps on the active database of the active standby pair.
You can load the cache group at any time after you create the cache group.
Drop a Cache Group
Dropping a cache group within a Clusterware environment requires several steps.
Perform these steps to drop a cache group.
Change an Existing Cache Group
Changing an existing cache group involves dropping and adding the cache group.
To change an existing cache group, first drop the existing cache group as described in Drop a Cache Group. Then add the cache group with the desired changes as described in Add a Cache Group.
Moving a Database to a Different Host
When a cluster is configured for advanced availability, you can use the
ttCWAdmin -relocate
command to move a database from the local host to
the next available spare host specified in the MasterHosts
attribute in the
cluster.oracle.ini
file.
If the database on the local host has the active role, the
-relocate
option first reverses the roles. Thus, the newly migrated
active database becomes the standby database and the old standby database becomes the
new active database.
The ttCWAdmin -relocate
command is useful for relocating a database if you decide to take the host offline. Ensure that there are no open transactions before you use the command.
If the ttCWAdmin -relocate
command requires a role switch, then you can optionally use the -timeout
option with the -relocate
option to set a timeout for the number of seconds to wait for the role switch.
For example:
ttCWAdmin -relocate -dsn advancedDSN
Note:
See ttCWAdmin in the Oracle TimesTen In-Memory Database Reference.
Performing a Rolling Upgrade of Oracle Clusterware Software
There are methods you can run to perform a rolling upgrade of the Oracle Clusterware software.
See Oracle Clusterware Administration and Deployment Guide in the Oracle Database documentation.
Upgrading TimesTen when Using Oracle Clusterware
There are methods to upgrade TimesTen on all hosts when using Oracle Clusterware:
-
Performing an Offline TimesTen Upgrade when Using Oracle Clusterware in Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide.
-
Performing an Online TimesTen Upgrade when Using Oracle Clusterware in Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide.
Performing Host or Network Maintenance
When you need to perform host or network maintenance, you need to stop the Oracle Clusterware resources and take down one or more of the TimesTen databases in the cluster.
In order to maintain data consistency for the database, you need to ensure that the TimesTen databases included in the active standby pair are brought down properly so that no transactions are lost.
One of the decisions you will make during performing maintenance is whether you should leave Oracle Clusterware enabled or disabled. If you leave Oracle Clusterware enabled, then all Oracle Clusterware and TimesTen processes restart automatically after a system reboot. If you disable Oracle Clusterware, none of these processes restart automatically.
Perform Maintenance on All Hosts in the Cluster Simultaneously
You can perform tasks to facilitate minimal down time while performing maintenance on all hosts in the cluster.
Note:
If you have an active, standby and one or more subscriber databases, you need to run some of these tasks on each host that contains the designated database.
-
Stop Oracle Clusterware and the replication agents by running the Oracle Clusterware
crsctl stop crs
command asroot
or OS administrator on each of the hosts that contain the active, standby, and subscriber databases.Since the active database is down, all requests are refused until the replication agents are restarted.
crsctl stop crs
The
crsctl stop crs
command changes the RAM policy temporarily for the active, standby, and all subscriber databases to inUse with RamGrace where the grace period is set for 60 seconds to enable these databases to be unloaded by TimesTen. -
Optionally, you can prevent the automatic startup of Oracle Clusterware and TimesTen when the server boots by running the Oracle Clusterware
crsctl disable crs
command asroot
or OS administrator on all hosts in the cluster:crsctl disable crs
-
Disconnect any application connections and wait for the active, standby, and subscriber databases to unload from memory.
-
To gracefully shutdown each TimesTen database in the active standby pair, run the following command on each of the hosts that contain the active, standby, and subscriber databases:
ttDaemonAdmin -stop
Note:
The TimesTen main daemon process manages all databases under the same TimesTen installation, be sure to disconnect from all databases before running the above command.
See Shutting Down a TimesTen Application in Oracle TimesTen In-Memory Database Operations Guide.
-
Perform maintenance on the hosts that contain the active, standby, and subscriber databases.
-
After the maintenance is complete, either:
-
Reboot all hosts, then wait until the Oracle Clusterware and TimesTen processes are running (which can take several minutes) if you did not disable the automatic startup of Oracle Clusterware and TimesTen.
-
Perform the following tasks on each host in the cluster if you disabled the automatic startup of Oracle Clusterware and TimesTen after a reboot or if you are not rebooting the hosts after maintenance when automatic startup is enabled:
-
Start the TimesTen database by running the following command:
ttDaemonAdmin -start
-
Enable the automatic startup of Oracle Clusterware when the server boots by running
crsctl enable crs
asroot
or OS administrator:crsctl enable crs
-
Start Oracle Clusterware on the local server by running
crsctl start crs
asroot
or OS administrator. Wait until all of the Oracle Clusterware resources come up before continuing to the next step.crsctl start crs
-
Once everything is up, you can reconnect with your applications and the active starts to replicate all updates to the standby and subscriber databases. The configured RAM policy resumes to always.
-
Perform Maintenance While Still Accepting Requests
There are methods to provide minimal down time while performing maintenance on all hosts in the cluster.
Note:
If you have an active, standby and one or more subscriber databases, you need to run some of these tasks on each host that contains the designated database.
-
Stop Oracle Clusterware and the replication agents by running the Oracle Clusterware
crsctl stop crs
command asroot
or OS administrator on each of the hosts that contain the standby and subscriber databases.The active database continues to accept requests and updates, but any changes are not propagated to the standby database and any subscribers until the replication agents are restarted.
crsctl stop crs
The
crsctl stop crs
command also changes the RAM policy temporarily for the standby and all subscriber databases to inUse with RamGrace where the grace period is set for 60 seconds to enable these databases to be unloaded by TimesTen. -
Optionally, you can prevent the automatic startup of Oracle Clusterware and TimesTen when the server boots by running the Oracle Clusterware
crsctl disable crs
command asroot
or OS administrator on each of the hosts that contain the standby and subscriber databases.crsctl disable crs
-
Disconnect any application connections and wait for the standby and subscriber databases to unload from memory.
-
To gracefully shutdown a TimesTen database, run the following command on each of the hosts that contain the standby and subscriber databases:
ttDaemonAdmin -stop
Note:
The TimesTen main daemon process manages all databases under the same TimesTen installation, be sure to disconnect from all databases before running the above command.
See Shutting Down a TimesTen Application in Oracle TimesTen In-Memory Database Operations Guide.
-
Perform maintenance on the hosts that contain the standby and subscriber databases.
-
After the maintenance is complete, either:
-
Reboot all hosts, then wait until the Oracle Clusterware and TimesTen processes are running (which can take several minutes) if you did not disable the automatic startup of Oracle Clusterware and TimesTen.
-
Perform the following tasks on each host in the cluster if you disabled the automatic startup of Oracle Clusterware and TimesTen after a reboot or if you are not rebooting the hosts after maintenance when automatic startup is enabled:
-
Start the TimesTen database by running the following command:
ttDaemonAdmin -start
-
Enable the automatic startup of Oracle Clusterware when the server boots by running
crsctl enable crs
asroot
or OS administrator:crsctl enable crs
-
Start Oracle Clusterware on the local server by running
crsctl start crs
asroot
or OS administrator. Wait until all of the Oracle Clusterware resources come up before continuing to the next step.crsctl start crs
-
Once everything is up, the active replicates all updates to the standby and subscriber databases.
-
-
Switch the active and standby databases with the
ttCWAdmin -switch
command so you can perform the same maintenance on the host with the active database.ttCWAdmin -switch -dsn advancedDSN
Note:
See Reversing the Roles of the Master Databases for more details on the
ttCWAdmin -switch
command. -
Stop Oracle Clusterware and the replication agents by running the Oracle Clusterware
crsctl stop crs
command asroot
or OS administrator on the host with the new standby database.The new active database continues to accept requests and updates, but any changes are not propagated to the new standby database and any subscribers until the replication agents are restarted.
crsctl stop crs
-
Disconnect any application connections and wait for the standby and subscriber databases to unload from memory.
-
To gracefully shutdown the TimesTen database, run the following command on the host that contains the new standby database:
ttDaemonAdmin -stop
-
Perform maintenance on the host that contains the new standby database. Now the maintenance has been performed on all hosts in the cluster.
-
After the maintenance is complete, either:
-
Reboot all hosts, then wait until the Oracle Clusterware and TimesTen processes are running (which can take several minutes) if you did not disable the automatic startup of Oracle Clusterware and TimesTen.
-
Perform the following tasks on each host in the cluster if you disabled the automatic startup of Oracle Clusterware and TimesTen after a reboot or if you are not rebooting the hosts after maintenance when automatic startup is enabled:
-
Start the TimesTen database by running the following command:
ttDaemonAdmin -start
-
Enable the automatic startup of Oracle Clusterware when the server boots by running
crsctl enable crs
asroot
or OS administrator:crsctl enable crs
-
Start Oracle Clusterware on the local server by running
crsctl start crs
asroot
or OS administrator. Wait until all of the Oracle Clusterware resources come up before continuing to the next step.crsctl start crs
-
Once everything is up, the active replicates all updates to the standby and subscriber databases. The RAM policy resumes to always.
-
-
Switch back to the original configuration for the active and standby roles for the active standby pair with the
ttCWAdmin -switch
command.ttCWAdmin -switch -dsn advancedDSN
Note:
See the Oracle Clusterware Administration and Deployment Guide in the Oracle Database documentation.