D Manual Sync Operations

The following manual sync operations must be performed if for some reason the secondary (standby) site is not synchronized with the primary site and you are performing regular backup operations of the primary site middle tier and OracleAS Infrastructure configuration files as described in Section D.1.1, "Manually Backing Up the Production Site". Then you will need to restore the backup configuration files as described in Section D.1.2, "Manually Restoring to Standby Site". After restoring the configuration files (OracleAS Infrastructure and Middle Tier) on the standby site, then proceed to Step 2 as described in Section 7.6.1.2.1, "Site Failover Operations".

D.1 Manually Synchronizing Baseline Installation with Standby Site Without Using OracleAS Guard asgctl Command-line Utility

Note:

This section and Section D.1.1, "Manually Backing Up the Production Site" and Section D.1.2, "Manually Restoring to Standby Site" are retained here for the special case as described in Step 1b in Section 7.6.1.2.1, "Site Failover Operations" where the standby site is not synchronized with the primary site. In this case, on the standby site, you must restore the most recently backed up configuration files as described in Section D.1.2, "Manually Restoring to Standby Site".

If you are using asgctl to continually synchronize the secondary (standby) site with the primary site, then both sites should already be synchronized and you do not need to manually perform a restore operation and you can begin with Step 2 in Section 7.6.1.2.1, "Site Failover Operations" to recover from an unplanned outage.

Once Oracle Data Guard has been set up between the production and standby sites, the procedure for synchronizing the two sites can be carried out. An initial synchronization should be done, before the production site is used, in order to obtain a baseline snapshot of the post-installation production site onto the standby site. This baseline can then be used to recover the production site configuration on the standby site if needed later.

In order to obtain a consistent point-in-time snapshot of the production site, the information stored in the OracleAS Infrastructure database and the Oracle Application Server-related configuration files in the middle-tier and OracleAS Infrastructure hosts must be synchronized at the same time. Synchronization of the configuration files can be done by backing up the files and restoring them on the standby hosts using the Oracle Application Server Backup and Recovery Tool. For the OracleAS Infrastructure database, synchronization is done using Oracle Data Guard by shipping the archive logs to the standby OracleAS Infrastructure and applying these logs in coordination with the restoration of the configuration files.

The sequence of steps for the baseline synchronization (which can also be used for future synchronizations) are:

Shipping OracleAS Infrastructure Database Archive Logs
Backing Up Configuration Files (OracleAS Infrastructure and Middle Tier)
Restoring Configuration Files (OracleAS Infrastructure and Middle Tier)
Restoring the OracleAS Infrastructure Database - Applying Log Files

These steps are detailed in the following two main sections.

D.1.1 Manually Backing Up the Production Site

The main strategy and approach to synchronizing configuration information between the production and standby sites is to synchronize the backup of OracleAS Infrastructure and middle-tier configuration files with the application of log information on the standby OracleAS Infrastructure database.

For Oracle Application Server, not all the configuration information is in the OracleAS Infrastructure database. The backup of the database files needs to be kept synchronized with the backup of the middle-tier and OracleAS Infrastructure configuration files. Due to this, log-apply services should not be enabled on the standby database. The log files from the production OracleAS Infrastructure are shipped to the standby OracleAS Infrastructure but are not applied.

The backup process of the production site involves backing up the configuration files in the middle-tier and OracleAS Infrastructure nodes. Additionally, the archive logs for the OracleAS Infrastructure database are shipped to the standby site.

The procedures to perform the backups and the log ship are discussed in the following sections:

Shipping OracleAS Infrastructure Database Archive Logs

Backing Up Configuration Files (OracleAS Infrastructure and Middle Tier)

IMPORTANT:

Ensure that no configuration changes are going to be made to the Oracle Application Server system (underlying configuration files and OracleAS Infrastructure database) as you perform the steps in this section.

Note:

At the minimum, the backup and restoration steps discussed in this section and the "Manually Restoring to Standby Site" section should be performed whenever there is any administration change in the production site (inclusive of changes to the OracleAS Infrastructure database and configuration files on the middle-tier and OracleAS Infrastructure nodes). On top of that, scheduled regular backups and restorations should also be done (for example, on a daily or twice weekly basis). See the Oracle Application Server Administrator's Guide for more backup and restore procedures.

D.1.1.1 Shipping OracleAS Infrastructure Database Archive Logs

After installing the OracleAS Disaster Recovery solution, Oracle Data Guard should have been installed in both the production and standby databases. The steps for shipping the archive logs from the production OracleAS Infrastructure database to the standby OracleAS Infrastructure database involve configuring Oracle Data Guard and executing several commands for both the production and standby databases. Execute the following steps to ship the logs for the OracleAS Infrastructure database:

If not disabled already, disable log-apply services by running the following SQLPLUS statement on the standby host:
```
SQL> alter database recover managed standby database cancel;
```
Run the following command to perform a log switch on the production OracleAS Infrastructure database. This ensures that the latest log file is shipped to the standby OracleAS Infrastructure database
```
SQL> alter system switch logfile;
```
In normal operation of the production site, the production database frequently ships log files to the standby database but are not applied. At the standby site, you want to apply the logs that are consistent up to the same time that the production site's configuration files are backed up. The following SQL statement encapsulates all OracleAS Infrastructure database changes into the latest log and allows the Oracle Data Guard transport services to transport this log to the OracleAS Infrastructure in the standby site:
```
SQL> select first_change# from v$log where status='CURRENT';
```
A SCN or sequence number is returned, which essentially represents the timestamp of the transported log.
Note down the SCN number as you will need this for the restoration of the production database changes on the standby site.

Continue to the next section to back up the configuration files on the middle-tier host(s) and OracleAS Infrastructure host.

D.1.1.2 Backing Up Configuration Files (OracleAS Infrastructure and Middle Tier)

Use the instructions in this section to back up the configuration files. The instructions require the use of the Oracle Application Server Backup and Recovery Tool. They assume you have installed and configured the tool on each OracleAS installation (middle tier and OracleAS Infrastructure) as it needs to be customized for each installation. Refer to Oracle Application Server Administrator's Guide for more details about that tool, including installation and configuration instructions.

For each middle-tier and OracleAS Infrastructure installation, perform the following steps (the same instructions can be used for the middle-tier and OracleAS Infrastructure configuration files):

After performing the installation and configuration steps detailed in the Oracle Application Server Administrator's Guide, for the Oracle Application Server Backup and Recovery Tool, the variables oracle_home, log_path, and config_backup_path in the tool's configuration file, config.inp, should have the appropriate values. Also, the following command for the tool should have been run to complete the configuration:
```
perl bkp_restore.pl -m configure_nodb
```
In Windows, the Perl executable can be found in <ORACLE_HOME>\perl\<perl_version>\bin\MSWin32-x86.

If you have not completed these tasks, do so before continuing with the ensuing steps.
Execute the following command to back up the configuration files from the current installation:
```
perl bkp_restore.pl -v -m backup_config
```
This command creates a directory in the location specified by the config_backup_path variable specified in the config.inp file. The directory name includes the time of the backup. For example: config_bkp_2003-09-10_13-21.
A log of the backup is also generated in the location specified by the log_path variable in the config.inp file. Check the log files for any errors that may have occurred during the backup process.
Copy the OracleAS Backup and Recovery Tool's directory structure and contents from the current node to its equivalent in the standby site. Ensure that the path structure on the standby node is identical to that on the current node.
Copy the backup directory (as defined by config_backup_path) from the current node to its equivalent in the standby site. Ensure that the path structure on the standby node is identical to that on the current node.

Repeat the steps above for each Oracle Application Server installation in the production site (middle tier and OracleAS Infrastructure).

Note:

There are two important items that should be maintained consistently between the production and standby sites. The directory names should be the same and the correlation of SCN to a given backup directory should be noted at both sites in administration procedures.

D.1.2 Manually Restoring to Standby Site

After backing up the configuration files from the middle-tier Oracle Application Server instances and OracleAS Infrastructure together with the OracleAS Infrastructure database, restore the files and database in the standby site using the instructions in this section, which consists of the following sub-sections:

Restoring Configuration Files (OracleAS Infrastructure and Middle Tier)
Restoring the OracleAS Infrastructure Database - Applying Log Files

D.1.2.1 Restoring Configuration Files (OracleAS Infrastructure and Middle Tier)

Restoring the backed up files from the production site requires the Oracle Application Server Backup and Recovery Tool that was used for the backup. The instructions in this section assume you have installed and configured the tool on each OracleAS installation in the standby site, both in the middle-tier and OracleAS Infrastructure nodes. Refer to Oracle Application Server Administrator's Guide for instructions on how to install the tool.

For each middle-tier and OracleAS Infrastructure installation in the standby site, perform the following steps (the same instructions can be used for the middle-tier and OracleAS Infrastructure configuration files):

Check that the OracleAS Backup and Recovery Tool's directory structure and the backup directory from the equivalent installation in the production site are present in the current node.
Stop the Oracle Application Server instances and their processes so that no modification of configuration files can occur during the restoration process. Use the following OPMN command:

In UNIX:
```
<ORACLE_HOME>/opmn/bin/opmnctl stopall
```
In Windows:
```
<ORACLE_HOME>\opmn\bin\opmnctl stopall
```
Check that all relevant processes are no longer running. In UNIX, use the following command:
```
ps -ef | grep <ORACLE_HOME>
```
In Windows, press <ctrl><alt><del> to bring up the Task Manager and verify that the processes have stopped.
Configure the backup utility for the Oracle home.

This can be accomplished either by configuring the OracleAS Backup and Recovery Tool for the Oracle home or copying the backup configuration file, config.inp, from the production site peer. Below is an example of running the OracleAS Backup and Recovery Tool configuration option:
```
perl bkp_restore.pl -v -m configure_nodb
```
In Windows, the Perl executable can be found in <ORACLE_HOME>\perl\<perl_version>\bin\MSWin32-x86.
Execute the following command to view a listing of the valid configuration backup locations:
```
perl bkp_restore.pl -v -m restore_config
```
Restore the configuration files using the following command:
```
perl bkp_restore.pl -v -m restore_config -t <backup_directory>
```
where <backup_directory> is the name of the directory with the backup files that was copied from the production site. For example, this could be config_bkp_2003-09-10_13-21.
Check the log file specified in config.inp for any errors that may have occurred during the restoration process.
Repeat the steps above for each Oracle Application Server installation in the production site (middle tier and OracleAS Infrastructure).

D.1.2.2 Restoring the OracleAS Infrastructure Database - Applying Log Files

During the backup phase, you executed several instructions to ship the database log files from the production site to the standby site up to the SCN number that you recorded as per instructed. To restore the standby database to that SCN number, apply the log files to the standby OracleAS Infrastructure database using the following SQLPLUS statement:

SQL> alter database recover automatic from '/private/oracle/oracleas/standby/' standby database until change <SCN>;

(In Windows, substitute the path shown above appropriately.)

With this command executed and the instructions to restore the configuration files completed on each middle-tier and OracleAS Infrastructure installation, the standby site is now synchronized with the production site. However, there are two common problems that can occur during the application of the log files: errors caused by the incorrect specification of the path and gaps in the log files that have been transported to the standby site.

The following are methods of resolving these problems:

Find the correct log path.

On the standby OracleAS Infrastructure database, try to determine location and number of received archive logs using the following SQLPLUS statement:

SQL> show parameter standby_archive_dest 

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
standby_archive_dest                 string      /private/oracle/oracleas/standby/

(The previous example shows the UNIX path. The Windows equivalent path is shown in Windows systems.)

Use the log path obtained from the previous step to ensure that all log files have been transported.

At the standby OracleAS Infrastructure database, perform the following:

standby> cd /private/oracle/oracleas/standby 
standby> ls 
1_13.dbf  1_14.dbf  1_15.dbf  1_16.dbf  1_17.dbf  1_18.dbf  1_19.dbf

(In Windows, use the command cd to change to the appropriate directory and dir to view the directory contents.)

At the production OracleAS Infrastructure database, execute the following SQLPLUS statement:

SQL> show parameter log_archive_dest_1 

NAME                                 TYPE        VALUE 
------------------------------------ ----------- ------------------------------ 
log_archive_dest1                   string      LOCATION=/private/oracle/oracleas/oradata                                                                                  MANDATORY 
log_archive_dest_10                  string

(The previous example shows the UNIX path. The Windows equivalent path is shown in Windows systems.)

Using the path specified in step 1, note the number and sequence of the log files. For example:
```
production> cd /private/oracle/oracleas/oradata 
production> ls 
1_10.dbf  1_12.dbf  1_14.dbf  1_16.dbf  1_18.dbf  asdb 
1_11.dbf  1_13.dbf  1_15.dbf  1_17.dbf  1_19.dbf
```
(In Windows, use the command cd to change to the appropriate directory and dir to view the directory contents.)

In the previous example, note the discrepancy where the standby OracleAS Infrastructure is missing files 1_10.dbf through 1_12.dbf. Since this gap in the log files happened in the past, it could be due to a problem with the historic setup involving the network used for the log transport. This problem has obviously been corrected and subsequent logs have been shipped. To correct the problem, copy (FTP) the log files to the corresponding directory on the standby OracleAS Infrastructure database host and re-attempt the SQLPLUS recovery statement shown earlier in this section.