Clusterware Management

Changing User Names or Passwords When Using Oracle Clusterware

When you create the active standby pair replication scheme with the ttCWAdmin -create command, Oracle Clusterware prompts for the required user names and passwords in order to manage the TimesTen environment.

Oracle Clusterware stores these user names and passwords. After modifying any user name or password, you must run the ttCWAdmin -reauthenticate command to enable Oracle Clusterware to store these new user names and passwords.

Ensure that the DDLReplicationLevel connection attribute is set to 3. This value replicates changes to the user names or passwords on the active database to the standby database.
Modify any of the user names or passwords in the same manner (and with the same restrictions) as described in Changing User Names or Passwords Used by Replication.
Ensure that all password changes are replicated to the standby database by calling the ttRepSubscriberWait built-in procedure (or the ttRepAdmin -wait command) on the active database using the DSN and host of the standby database. For example, to ensure that all transactions are replicated to the master2 standby database on the host2 host:
```
CALL ttRepSubscriberWait(NULL, NULL, 'master2', 'host2', -1);
```
Store the new passwords in Oracle Clusterware by running the ttCWAdmin -reauthenticate command.
```
ttCWAdmin -reauthenticate -dsn myDSN
```
This command prompts for the same information as requested for the ttCWAdmin -create command, which is discussed in Create an Active Standby Pair Replication Scheme.

Managing Hosts in the Cluster

The following sections describe how to add or remove hosts when using a cluster.

Adding a Host to the Cluster

Adding a host requires that the cluster be configured for advanced availability.

The examples in this section use the advancedSubscriberDSN.

To add two spare master hosts to a cluster, enter a command similar to the following:

ttCWAdmin -addMasterHosts -hosts "host8,host9" -dsn advancedSubscriberDSN

To add a spare subscriber host to a cluster, enter a command similar to the following:

ttCWAdmin -addSubscriberHosts -hosts "subhost1" -dsn advancedSubscriberDSN

Removing a Host From the Cluster

Removing a host from the cluster requires that the cluster be configured for advanced availability.

MasterHosts must list more than two hosts if one of the master hosts is to be removed. SubscriberHosts must list at least one more host than the number of subscriber databases if one of the subscriber hosts is to be removed.

The examples in this section use the advancedSubscriberDSN.

To remove two spare master host from the cluster, enter a command similar to the following:

ttCWAdmin -delMasterHosts "host8,host9" -dsn advancedSubscriberDSN

To remove a spare subscriber hosts from the cluster, enter a command similar to the following:

ttCWAdmin -delSubscriberHosts "subhost1" -dsn advancedSubscriberDSN

Managing Active Standby Pairs in a Cluster

The following sections describe how to add or remove an active standby pair to a cluster.

Adding an Active Standby Pair to a Cluster

You can add an active standby pair (with or without subscribers) to a cluster that is already managing an active standby pair.

Create and populate a database on the host where you intend the active database to reside initially. See Create and Populate a TimesTen Database on One Host.

Modify the cluster.oracle.ini file. This example adds advSub2DSN to the cluster.oracle.ini file that already contains the configuration for advancedSubscriberDSN. The new active standby pair is on different hosts from the original active standby pair.

[advancedSubscriberDSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4, host5
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0

[advSub2DSN]
MasterHosts=host6,host7,host8
SubscriberHosts=host9, host10
MasterVIP=192.168.1.4, 192.168.1.5
SubscriberVIP=192.168.1.6
VIPInterface=eth0
VIPNetMask=255.255.255.0

Create new virtual IP addresses as the root user.
```
ttCWAdmin -createVIPs -dsn advSub2DSN
```
Create the new active standby pair replication scheme.
```
ttCWAdmin -create -dsn advSub2DSN
```
Start the new active standby pair replication scheme.
```
ttCWAdmin -start -dsn advSub2DSN
```

Removing an Active Standby Pair From a Cluster

You can remove an active standby pair (with or without subscribers) from a cluster.

Stop the replication agents on all databases in the active standby pair. This example uses advSub2DSN, which was added in Adding an Active Standby Pair to a Cluster.
```
ttCWAdmin -stop -dsn advSub2DSN
```
Drop the active standby replication scheme.
```
ttCWAdmin -drop -dsn advSub2DSN
```
Drop the virtual IP addresses for the active standby pair.
```
ttCWAdmin -dropVIPs -dsn advSub2DSN
```
Modify the cluster.oracle.ini file (optional). Remove the entries for advSub2DSN.
If you want to destroy the databases, log onto each host that was included in the configuration for this active standby pair and use the ttDestroy utility.
```
ttDestroy advSub2DSN
```
See ttDestroy in Oracle TimesTen In-Memory Database Reference.

Managing Read-Only Subscribers in the Active Standby Pair

The following sections describe how to manage read-only subscribers in the active standby pair that is managed by Oracle Clusterware.

Adding a Read-Only Subscriber Managed by Oracle Clusterware

To add a read-only subscriber that is to be managed by Oracle Clusterware to an active standby pair replication scheme, perform these steps:

Stop the replication agents on all databases. This example uses the advancedSubscriberDSN, which already has a subscriber and is configured for advanced availability.
```
ttCWAdmin -stop -dsn advancedSubscriberDSN
```

Drop the active standby pair.

ttCWAdmin -drop -dsn advancedSubscriberDSN

Modify the cluster.oracle.ini file.
- Add the subscriber to the SubscriberHosts attribute.
- If the cluster is configured for advanced availability, add a virtual IP address to the SubscriberVIP attribute.
See Configuring Advanced Availability for an example using these attributes.
Create the active standby pair replication scheme.
```
ttCWAdmin -create -dsn advancedSubscriberDSN
```
Start the active standby pair replication scheme.
```
ttCWAdmin -start -dsn advancedSubscriberDSN
```

Removing a Read-Only Subscriber Managed by Oracle Clusterware

To remove a read-only subscriber that is managed by Oracle Clusterware from an active standby pair, perform these steps:

Stop the replication agents on all databases. This example uses the advancedSubscriberDSN, which has a subscriber and is configured for advanced availability.
```
ttCWAdmin -stop -dsn advancedSubscriberDSN
```

Drop the active standby pair.

ttCWAdmin -drop -dsn advancedSubscriberDSN

Modify the cluster.oracle.ini file.
- Remove the subscriber from the SubscriberHosts attribute or remove the attribute altogether if there are no subscribers left in the active standby pair.
- Remove a virtual IP from the SubscriberVIP attribute or remove the attribute altogether if there are no subscribers left in the active standby pair.
Create the active standby pair replication scheme.
```
ttCWAdmin -create -dsn advancedSubscriberDSN
```
Start the active standby pair replication scheme.
```
ttCWAdmin -start -dsn advancedSubscriberDSN
```

Adding or Dropping a Read-Only Subscriber Not Managed by Oracle Clusterware

You can add or drop a read-only subscriber that is not managed by Oracle Clusterware to or from an existing active standby pair replication scheme that is managed by Oracle Clusterware.

Using the ttCWAdmin -beginAlterSchema command enables you to add a subscriber without dropping and re-creating the replication scheme. Oracle Clusterware does not manage the subscriber, because it is not part of the configuration that was set up for Oracle Clusterware management.

Perform these steps:

Run the ttCWAdmin -beginAlterSchema command to stop the replication agent on the active and standby databases.
Using ttIsql to connect to the active database, you can add or drop the subscriber to or from the replication scheme by using an ALTER ACTIVE STANDBY PAIR statement. For example, to add a subscriber:
```
ALTER ACTIVE STANDBY PAIR ADD SUBSCRIBER ROsubDSN ON host6;
```
To drop a subscriber:
```
ALTER ACTIVE STANDBY PAIR DROP SUBSCRIBER ROsubDSN ON host6;
```
Run the ttCWAdmin -endAlterSchema command that registers the altered replication scheme and starts replication. If you are adding a subscriber, this also initiates a duplicate to the standby database.
Run the ttIsql repschemes command to verify that the read-only subscriber has been added to or dropped from the replication scheme.
Use the ttRepStateGet built-in procedure to verify that the state of the standby database is STANDBY.
If you added a subscriber, then run ttRepAdmin -duplicate on the subscriber host to duplicate the standby database to the read-only subscriber. See Duplicating a Database.
If you added a subscriber, start the replication agent on the subscriber host.

If you added a subscriber, ensure that the read-only subscriber is included if the cluster is dropped and re-created by adding the RemoteSubscriberHosts Oracle Clusterware attribute for the read-only subscriber in the cluster.oracle.ini file as described in Step 1 in Configure a Read-Only Subscriber That Is Not Managed by Oracle Clusterware. Alternatively, if you dropped a subscriber, remove the RemoteSubscriberHosts Oracle Clusterware attribute for the dropped subscriber in the cluster.oracle.ini file (if it is configured).

Rebuilding a Read-Only Subscriber Not Managed by Oracle Clusterware

Perform the following tasks to destroy and rebuild a read-only subscriber that is not managed by Oracle Clusterware:

Stop the replication agent on the subscriber host.
Use the ttDestroy utility to destroy the subscriber database.
On the subscriber host, use ttRepAdmin -duplicate to duplicate the standby database to the read-only subscriber. See Duplicating a Database.

Reversing the Roles of the Master Databases

After a failover, the active and standby databases are on different hosts than they were before the failover. You can use the -switch option of the ttCWAdmin utility to restore the original configuration.

Optionally, you can also use the -timeout option with the -switch option to set a timeout for the number of seconds to wait for the active and standby database switch to complete.

For example:

ttCWAdmin -switch -dsn basicDSN

Ensure that there are no open transactions before using the -switch option. If there are open transactions, the command fails.

Note:

See ttCWAdmin in the Oracle TimesTen In-Memory Database Reference.

Figure 8-6 shows the hosts for an active standby pair. The active database resides on host A, and the standby database resides on host B.

Figure 8-6 Hosts for an Active Standby Pair

Description of "Figure 8-6 Hosts for an Active Standby Pair"

The ttCWAdmin -switch command performs these tasks:

Deactivates the TimesTen cluster agent (ttCRSAgent) on host A (the active master).
Disables the TimesTen database monitor (ttCRSmaster) on host A.
Calls the ttRepSubscriberWait, ttRepStop and ttRepDeactivate built-in procedures on host A.
Stops the active service (ttCRSActiveService) on host A and reports a failure event to the Oracle Clusterware CRSD process.
Enables monitoring on host A and moves the active service to host B.
Starts the replication agent on host A, stops the standby service (ttCRSsubservice) on host B and reports a failure event to the Oracle Clusterware CRSD process on host B.
Starts the standby service (ttCRSsubservice) on host A.

Modifying Connection Attribute Values

When you modify connection attributes across an active standby pair with subscribers, the connection attributes must be modified on all hosts within this configuration.

Note:

You cannot modify any DATASTORE connection attributes since they are only allowed to be set at data store creation time. For example, this procedure can be used to change the PermSize value.

Use the ttCWAdmin -beginAlterSchema and -endAlterSchema commands to facilitate the change of any connection attribute values on the active and standby databases and any subscribers.

The ttCWAdmin -beginAlterSchema command suspends the Oracle Clusterware management and stops the replication agents on the active and standby databases and any subscriber databases in preparation for any changes.
After you complete all changes, the ttCWAdmin -endAlterSchema command resumes Oracle Clusterware management and restarts all replication agents on the active and standby databases and any subscriber databases.

Perform the following tasks when altering any connection attributes for the active standby pair when using Oracle Clusterware:

Suspend Oracle Clusterware and stop all replication agents for the active and standby databases with the ttCWAdmin -beginAlterSchema command.

The active database continues to accept requests and updates, but any changes are not propagated to the standby database and any subscribers until the replication agents are restarted.

The ttCWAdmin -beginAlterSchema command also changes the RAM policy temporarily for the standby database and all subscriber databases to InUse with RamGrace where the grace period is set for 60 seconds to enable these databases to be unloaded by TimesTen. Once the standby and subscriber databases are unloaded from memory, the connection attributes for these databases can be modified.
```
ttCWAdmin -beginAlterSchema -dsn advancedDSN
```
Disconnect any application connections and wait for the standby and subscriber databases to unload from memory (based on the RAM policy).

Once the standby and subscriber databases are unloaded from memory, alter any connection attributes, such as PermSize, on the hosts for the standby and all subscriber databases in their respective sys.odbc.ini files.
Resume Oracle Clusterware and restart all replication agents for the active and standby databases with the ttCWAdmin -endAlterSchema command. The configured RAM policy for each TimesTen database is set back to always. The active database propagates any transactions that occurred while the standby database and subscribers were down.
```
ttCWAdmin -endAlterSchema -dsn advancedDSN
```
Note:

Wait an appropriate amount of time for all changes to propagate from the active database to the standby database and all subscribers before performing the next step.

The only host that has not had the connection attribute change is the active database. You will switch the active database with the standby database so that you can modify the connection attributes on this host.
Suspend all application workload and disconnect all applications on the active database.
Switch the active and standby databases with the ttCWAdmin -switch command.
```
ttCWAdmin -switch -dsn advancedDSN
```
Note:

See Reversing the Roles of the Master Databases.
Suspend Oracle Clusterware and stop all replication agents for all databases with the ttCWAdmin -beginAlterSchema command.

The new active database may still accept requests and updates, but any changes are not propagated to the new standby database and any subscribers.

The RAM policy changes for the new standby database (and all subscriber databases) to inUse with RamGrace where the grace period is set for 60 seconds to enable these databases to be unloaded by TimesTen.
```
ttCWAdmin -beginAlterSchema -dsn advancedDSN
```
Wait for the new standby database to unload from memory. Once unloaded, alter the same connection attributes, such as PermSize, on the new standby database in its sys.odbc.ini file. The connection attributes are now modified on all hosts.
Run the ttCWAdmin -endAlterSchema command to resume Oracle Clusterware management and restart the replication agents on the active and standby databases. The configured RAM policy resumes to always.
```
ttCWAdmin -endAlterSchema -dsn advancedDSN
```
Suspend all application workload and disconnect all applications on the active database.
If desired, you can switch the active and standby databases with the ttCWAdmin -switch command to restore the active standby pair to the original configuration.
```
ttCWAdmin -switch -dsn advancedDSN
```

Managing the TimesTen Database RAM Policy

By default, the TimesTen database RAM policy is set to always when Oracle Clusterware manages the TimesTen database. However, if you stop Oracle Clusterware management, the TimesTen database RAM policy is set to inUse.

If you no longer use Oracle Clusterware to manage TimesTen, you should set the TimesTen RAM policy to what is appropriate for your environment. Typically, the recommended setting is manual.

See Specifying a RAM Policy in the Oracle TimesTen In-Memory Database Operations Guide.

Changing the Schema

When using Oracle Clusterware to manage an active standby pair, you can modify the schema by running DDL statements as in a normal replication environment, except that Oracle Clusterware must start and stop all replication agents, when it is necessary to do so.

Thus, when you change the schema, note the following:

For those DDL statements on objects that are automatically replicated, you do not need to stop the replication agents. In this case, no further action is required, since these DDL statements are automatically propagated and applied to the standby database and any subscribers. The DDLReplicationLevel connection attribute controls what DDL statements are replicated.
For those objects that are a part of the replication scheme, but any DDL statements processed on these objects are not replicated (these objects are listed in Making Other Changes to an Active Standby Pair), run the Oracle Clusterware ttCWAdmin -beginAlterSchema command on the active database, which suspends any Oracle Clusterware management and stops the replication agents on each node in the replication scheme. Then, run the DDL statement on the active database in the replication scheme. Finally, run the Oracle Clusterware ttCWAdmin -endAlterSchema command on the active database to restart all replication agents.

Because these objects are a part of the replication scheme, but the DDL statements are not replicated, a duplicate occurs after the ttCWAdmin -endAlterSchema command to propagate these schema changes to the standby database and any subscribers. This is the only scenario when a duplicate is used to propagate the schema changes.

Follow the instructions described in Facilitating Schema Change for Oracle Clusterware.
For those DDL statements on objects that are not automatically replicated and are not part of the replication scheme, run the Oracle Clusterware ttCWAdmin -beginAlterSchema command on the active database, which suspends any Oracle Clusterware management and stops and the replication agents on all nodes. Then, you can synchronize all nodes by manually running these DDL statements as indicated in Making DDL Changes in an Active Standby Pair. Finally, run the Oracle Clusterware ttCWAdmin -endAlterSchema command on the active database to restart all replication agents.

Follow the instructions described in Facilitating Schema Change for Oracle Clusterware.

Note:

The Making DDL Changes in an Active Standby Pair and Making Other Changes to an Active Standby Pair sections describe which DDL statements are and are not automatically replicated for an active standby pair. These sections also describe what objects are a part of the replication scheme.

Facilitating Schema Change for Oracle Clusterware

Use the ttCWAdmin -beginAlterSchema and -endAlterSchema commands to facilitate a schema change on the active and standby databases.

The ttCWAdmin -beginAlterSchema command suspends the Oracle Clusterware management and stops replication agents on both the active and standby databases in preparation for any schema changes.
After you complete all schema changes, run the ttCWAdmin -endAlterSchema command. For those objects that are a part of the replication scheme, but any DDL statements processed on these objects are not automatically replicated, a duplicate occurs after the ttCWAdmin -endAlterSchema command to propagate only these schema changes to the standby database and any subscribers. This command registers the altered replication scheme, restarts the replication agents on the active and standby databases, and reinstates Oracle Clusterware control.

Perform the following tasks when altering the schema of the active standby pair when using Oracle Clusterware:

Suspend Oracle Clusterware and stop the replication agents on both the active and standby databases.
```
ttCWAdmin -beginAlterSchema -dsn advancedDSN
```
Make any desired schema changes.

If you create, alter, or drop any objects where the DDL for these objects are not replicated, you should also manually create, alter, or drop the same objects on the standby database and subscribers while the replication agents are inactive to ensure that the same objects exist on all databases in the replication scheme. For example, if you create a materialized view on the active database, create the materialized view on the standby and subscriber databases at this time.
If the object is not automatically replicated but is a part of the replication scheme, (such as a sequence) and you want to include it in the active standby pair replication scheme, alter the active standby pair.
```
ALTER ACTIVE STANDBY PAIR INCLUDE samplesequence;
```
If the object is a cache group, see Making Schema Changes to Cache Groups for instructions to create, alter, or drop a cache group.
Run the ttCWAdmin -endAlterSchema command to resume Oracle Clusterware and restart the replication agents on the active and standby databases. If you modified objects that are a part of the replication scheme, but any DDL statements processed on these objects are not automatically replicated, a duplicate occurs after the ttCWAdmin -endAlterSchema command to propagate only these schema changes to the standby database and any subscribers.
```
ttCWAdmin -endAlterSchema -dsn advancedDSN
```

Making Schema Changes to Cache Groups

Use the ttCWAdmin -beginAlterSchema and -endAlterSchema commands to facilitate the schema changes on cache groups within Oracle Clusterware.

Add a Cache Group

You can add a cache group on the active database of the active standby pair.

Perform these steps on the active database of the active standby pair.

Suspend Oracle Clusterware management and stop the replication agents with the ttCWAdmin -beginAlterSchema command.
```
ttCWAdmin -beginAlterSchema -dsn advancedDSN
```
Create the cache group.
If the cache group is a read-only cache group, alter the active standby pair to include the cache group.
```
ALTER ACTIVE STANDBY PAIR INCLUDE CACHE GROUP samplecachegroup;
```
Resume Oracle Clusterware and start the replication agents by running the ttCWAdmin -endAlterSchema command. Since you added a cache group object, a duplicate occurs to propagate these schema changes to the standby database.
```
ttCWAdmin -endAlterSchema -dsn advancedDSN
```

You can load the cache group at any time after you create the cache group.

Drop a Cache Group

Dropping a cache group within a Clusterware environment requires several steps.

Perform these steps to drop a cache group.

Unload the cache group.
```
UNLOAD CACHE GROUP samplecachegroup;
```
On the active database of an active standby pair, enable dropping the cache group.
```
ttCWAdmin -beginAlterSchema -dsn advancedDSN
```
If the cache group is a read-only cache group, alter the active standby pair to exclude the cache group.
```
ALTER ACTIVE STANDBY PAIR EXCLUDE CACHE GROUP samplecachegroup;
```
If the cache group is a read-only cache group, set the autorefresh state to PAUSED.
```
ALTER CACHE GROUP samplecachegroup SET AUTOREFRESH STATE PAUSED;
```
Drop the cache group.
```
DROP CACHE GROUP samplecachegroup;
```
If the cache group was a read-only cache group, run the timesten_home/install/oraclescripts/cacheCleanUp.sql SQL*Plus script as the cache administration user on the Oracle database to drop the Oracle database objects used to implement autorefresh operations.
Resume Oracle Clusterware and restart the replication agents by running the ttCWAdmin -endAlterSchema command. Since you dropped a cache group object, a duplicate occurs to propagate these schema changes to the standby database.
```
ttCWAdmin -endAlterSchema -dsn advancedDSN
```

Change an Existing Cache Group

Changing an existing cache group involves dropping and adding the cache group.

To change an existing cache group, first drop the existing cache group as described in Drop a Cache Group. Then add the cache group with the desired changes as described in Add a Cache Group.

Moving a Database to a Different Host

When a cluster is configured for advanced availability, you can use the ttCWAdmin -relocate command to move a database from the local host to the next available spare host specified in the MasterHosts attribute in the cluster.oracle.ini file.

If the database on the local host has the active role, the -relocate option first reverses the roles. Thus, the newly migrated active database becomes the standby database and the old standby database becomes the new active database.

The ttCWAdmin -relocate command is useful for relocating a database if you decide to take the host offline. Ensure that there are no open transactions before you use the command.

If the ttCWAdmin -relocate command requires a role switch, then you can optionally use the -timeout option with the -relocate option to set a timeout for the number of seconds to wait for the role switch.

For example:

ttCWAdmin -relocate -dsn advancedDSN

Note:

See ttCWAdmin in the Oracle TimesTen In-Memory Database Reference.

Performing a Rolling Upgrade of Oracle Clusterware Software

There are methods you can run to perform a rolling upgrade of the Oracle Clusterware software.

See Oracle Clusterware Administration and Deployment Guide in the Oracle Database documentation.

Upgrading TimesTen when Using Oracle Clusterware

There are methods to upgrade TimesTen on all hosts when using Oracle Clusterware:

Performing an Offline TimesTen Upgrade when Using Oracle Clusterware in Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide.
Performing an Online TimesTen Upgrade when Using Oracle Clusterware in Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide.

Performing Host or Network Maintenance

When you need to perform host or network maintenance, you need to stop the Oracle Clusterware resources and take down one or more of the TimesTen databases in the cluster.

In order to maintain data consistency for the database, you need to ensure that the TimesTen databases included in the active standby pair are brought down properly so that no transactions are lost.

One of the decisions you will make during performing maintenance is whether you should leave Oracle Clusterware enabled or disabled. If you leave Oracle Clusterware enabled, then all Oracle Clusterware and TimesTen processes restart automatically after a system reboot. If you disable Oracle Clusterware, none of these processes restart automatically.

Perform Maintenance on All Hosts in the Cluster Simultaneously

You can perform tasks to facilitate minimal down time while performing maintenance on all hosts in the cluster.

Note:

If you have an active, standby and one or more subscriber databases, you need to run some of these tasks on each host that contains the designated database.

Stop Oracle Clusterware and the replication agents by running the Oracle Clusterware crsctl stop crs command as root or OS administrator on each of the hosts that contain the active, standby, and subscriber databases.

Since the active database is down, all requests are refused until the replication agents are restarted.
```
crsctl stop crs
```
The crsctl stop crs command changes the RAM policy temporarily for the active, standby, and all subscriber databases to inUse with RamGrace where the grace period is set for 60 seconds to enable these databases to be unloaded by TimesTen.
Optionally, you can prevent the automatic startup of Oracle Clusterware and TimesTen when the server boots by running the Oracle Clusterware crsctl disable crs command as root or OS administrator on all hosts in the cluster:
```
crsctl disable crs
```
Disconnect any application connections and wait for the active, standby, and subscriber databases to unload from memory.
To gracefully shutdown each TimesTen database in the active standby pair, run the following command on each of the hosts that contain the active, standby, and subscriber databases:
```
ttDaemonAdmin -stop
```
Note:

The TimesTen main daemon process manages all databases under the same TimesTen installation, be sure to disconnect from all databases before running the above command.

See Shutting Down a TimesTen Application in Oracle TimesTen In-Memory Database Operations Guide.
Perform maintenance on the hosts that contain the active, standby, and subscriber databases.
After the maintenance is complete, either:
- Reboot all hosts, then wait until the Oracle Clusterware and TimesTen processes are running (which can take several minutes) if you did not disable the automatic startup of Oracle Clusterware and TimesTen.
- Perform the following tasks on each host in the cluster if you disabled the automatic startup of Oracle Clusterware and TimesTen after a reboot or if you are not rebooting the hosts after maintenance when automatic startup is enabled:
  1. Start the TimesTen database by running the following command:
```
ttDaemonAdmin -start
```
  2. Enable the automatic startup of Oracle Clusterware when the server boots by running crsctl enable crs as root or OS administrator:
```
crsctl enable crs
```
  3. Start Oracle Clusterware on the local server by running crsctl start crs as root or OS administrator. Wait until all of the Oracle Clusterware resources come up before continuing to the next step.
```
crsctl start crs
```
Once everything is up, you can reconnect with your applications and the active starts to replicate all updates to the standby and subscriber databases. The configured RAM policy resumes to always.

Perform Maintenance While Still Accepting Requests

There are methods to provide minimal down time while performing maintenance on all hosts in the cluster.

Note:

If you have an active, standby and one or more subscriber databases, you need to run some of these tasks on each host that contains the designated database.

Stop Oracle Clusterware and the replication agents by running the Oracle Clusterware crsctl stop crs command as root or OS administrator on each of the hosts that contain the standby and subscriber databases.

The active database continues to accept requests and updates, but any changes are not propagated to the standby database and any subscribers until the replication agents are restarted.
```
crsctl stop crs
```
The crsctl stop crs command also changes the RAM policy temporarily for the standby and all subscriber databases to inUse with RamGrace where the grace period is set for 60 seconds to enable these databases to be unloaded by TimesTen.
Optionally, you can prevent the automatic startup of Oracle Clusterware and TimesTen when the server boots by running the Oracle Clusterware crsctl disable crs command as root or OS administrator on each of the hosts that contain the standby and subscriber databases.
```
crsctl disable crs
```
Disconnect any application connections and wait for the standby and subscriber databases to unload from memory.
To gracefully shutdown a TimesTen database, run the following command on each of the hosts that contain the standby and subscriber databases:
```
ttDaemonAdmin -stop
```
Note:

The TimesTen main daemon process manages all databases under the same TimesTen installation, be sure to disconnect from all databases before running the above command.

See Shutting Down a TimesTen Application in Oracle TimesTen In-Memory Database Operations Guide.
Perform maintenance on the hosts that contain the standby and subscriber databases.
After the maintenance is complete, either:
- Reboot all hosts, then wait until the Oracle Clusterware and TimesTen processes are running (which can take several minutes) if you did not disable the automatic startup of Oracle Clusterware and TimesTen.
- Perform the following tasks on each host in the cluster if you disabled the automatic startup of Oracle Clusterware and TimesTen after a reboot or if you are not rebooting the hosts after maintenance when automatic startup is enabled:
  1. Start the TimesTen database by running the following command:
```
ttDaemonAdmin -start
```
  2. Enable the automatic startup of Oracle Clusterware when the server boots by running crsctl enable crs as root or OS administrator:
```
crsctl enable crs
```
  3. Start Oracle Clusterware on the local server by running crsctl start crs as root or OS administrator. Wait until all of the Oracle Clusterware resources come up before continuing to the next step.
```
crsctl start crs
```
Once everything is up, the active replicates all updates to the standby and subscriber databases.
Switch the active and standby databases with the ttCWAdmin -switch command so you can perform the same maintenance on the host with the active database.
```
ttCWAdmin -switch -dsn advancedDSN
```
Note:

See Reversing the Roles of the Master Databases for more details on the ttCWAdmin -switch command.
Stop Oracle Clusterware and the replication agents by running the Oracle Clusterware crsctl stop crs command as root or OS administrator on the host with the new standby database.

The new active database continues to accept requests and updates, but any changes are not propagated to the new standby database and any subscribers until the replication agents are restarted.
```
crsctl stop crs
```
Disconnect any application connections and wait for the standby and subscriber databases to unload from memory.
To gracefully shutdown the TimesTen database, run the following command on the host that contains the new standby database:
```
ttDaemonAdmin -stop
```
Perform maintenance on the host that contains the new standby database. Now the maintenance has been performed on all hosts in the cluster.
After the maintenance is complete, either:
- Reboot all hosts, then wait until the Oracle Clusterware and TimesTen processes are running (which can take several minutes) if you did not disable the automatic startup of Oracle Clusterware and TimesTen.
- Perform the following tasks on each host in the cluster if you disabled the automatic startup of Oracle Clusterware and TimesTen after a reboot or if you are not rebooting the hosts after maintenance when automatic startup is enabled:
  1. Start the TimesTen database by running the following command:
```
ttDaemonAdmin -start
```
  2. Enable the automatic startup of Oracle Clusterware when the server boots by running crsctl enable crs as root or OS administrator:
```
crsctl enable crs
```
  3. Start Oracle Clusterware on the local server by running crsctl start crs as root or OS administrator. Wait until all of the Oracle Clusterware resources come up before continuing to the next step.
```
crsctl start crs
```
Once everything is up, the active replicates all updates to the standby and subscriber databases. The RAM policy resumes to always.
Switch back to the original configuration for the active and standby roles for the active standby pair with the ttCWAdmin -switch command.
```
ttCWAdmin -switch -dsn advancedDSN
```

Note:

See the Oracle Clusterware Administration and Deployment Guide in the Oracle Database documentation.