14
Recovery Strategies and Procedures

This chapter describes Oracle Application Server recovery strategies and procedures for different types of failures and outages.

It contains the following topics:

• Recovery Strategies

• Recovery Procedures

14.1 Recovery Strategies

This section describes Oracle Application Server recovery strategies for different types of failures and outages. It contains the following topics:

• Recovery Strategies for Data Loss, Host Failure, or Media Failure (Critical)

• Recovery Strategies for Process Crashes and System Outages (Non-Critical)

14.1.1 Recovery Strategies for Data Loss, Host Failure, or Media Failure (Critical)

This section describes recovery strategies for outages that involve actual data loss or corruption, host failure, or media failure where the host or disk cannot be restarted and are permanently lost. This type of failure requires some type of data restoration before the Oracle Application Server environment (middle tier, Infrastructure, or both) can be restarted and continue with normal processing.

The strategies in this section use point-in-time recovery of the middle tier and Infrastructure. This means that, no matter where the loss occurred, the middle tier and Infrastructure are always restored together so they are in sync as they were at the time of the last backup.

Assumptions

The following assumptions apply to the recovery strategies in this section:

• ARCHIVELOG mode was enabled for all Metadata Repository backups.

• Complete recovery of the database can be performed, that is, no redo log files have been lost.

• No administrative changes were made since the last backup. If administrative changes were made since the last backup, they will need to be reapplied after recovery is complete.

  See Also: Appendix G, "Examples of Administrative Changes" to learn more about administrative changes

Determining Which Strategy To Use

Recovery strategies are listed in the following tables:

• Table 14-1, " Recovery Strategies for Data Loss, Host Failure, and Media Failure in Infrastructures"

  Use this table if you experience data loss, host failure, or media failure in an Infrastructure installation. Find the type of loss and follow the recommended procedure. The procedures apply to Infrastructure that are installed into a single Oracle home, as well as Infrastructures with Identity Management in one Oracle home and a Metadata Repository in another Oracle home or host.

• Table 14-2, "  Recovery Strategies for Data Loss, Host Failure, and Media Failure in Middle-tier Instances"

  Use this table if you experience data loss, host failure, or media failure in a middle-tier installation. Find the type of loss and follow the recommended procedure.

If the loss occurred in both the Infrastructure and middle tier, follow the Infrastructure recovery strategy first, then the middle tier.

Table 14-1  Recovery Strategies for Data Loss, Host Failure, and Media Failure in Infrastructures

Type of Loss	Recovery Strategies
Loss of host	You can restore to a new host that has the same hostname and IP address. Follow the procedure in Section 14.2.2, "Restoring an Infrastructure to a New Host".
Oracle software/binary loss or corruption	If any Oracle binaries have been lost or corrupted, you must recover the entire Infrastructure. Follow the procedure in Section 14.2.1, "Restoring an Infrastructure to the Same Host".
Database or data failure of the Metadata Repository (datafile loss, control file loss, media failure, disk corruption)	If the Metadata Repository is corrupted due to data loss or media failure, you can restore and recover it. Follow the procedure in Section 14.2.3, "Restoring and Recovering the Metadata Repository".
Deletion or corruption of configuration files	If you lose any configuration files in the Infrastructure Oracle home, you can restore them. Follow the procedure in Section 14.2.4, "Restoring Infrastructure Configuration Files".
Deletion or corruption of configuration files and data failure of the Metadata Repository	If you lose configuration files and the Metadata Repository is corrupted, you can restore and recover both. Follow these procedures: Section 14.2.4, "Restoring Infrastructure Configuration Files" Section 14.2.3, "Restoring and Recovering the Metadata Repository"

Table 14-2   Recovery Strategies for Data Loss, Host Failure, and Media Failure in Middle-tier Instances

Type of Loss	Recovery Strategies
Loss of host	If the host has been lost, you have two options: You can restore to a new host that has the same hostname and IP address. You can restore to a new host that has a different hostname and IP address. In either case, follow the procedure in Section 14.2.6, "Restoring a Middle-tier Installation to a New Host". Note that if the original host had a middle-tier installation and an Infrastructure, you cannot restore the middle-tier to a host with a different hostname or IP address.
Oracle software/binary deletion or corruption	If any Oracle binaries have been lost or corrupted, you must restore the entire middle tier to the same host. Follow the procedure in Section 14.2.5, "Restoring a Middle-tier Installation to the Same Host".
Deletion or corruption of configuration files	If you lose any configuration files in the middle tier Oracle home, you can restore them. Follow the procedure in Section 14.2.7, "Restoring Middle-tier Configuration Files".

14.1.2 Recovery Strategies for Process Crashes and System Outages (Non-Critical)

This section describes recovery strategies for process crashes and system outages. These types of outages do not involve any data loss, and therefore do not require any files to be recovered. In some cases, failure may be transparent and no manual intervention is required to recover the failed component. However, in some cases, manual intervention is required to restart a process or component. While these strategies do not strictly fit into the category of backup and recovery, they are included in this book for completeness.

Determining Which Strategy to Use

Recovery strategies for process crashes and system outages are listed in the following tables:

• Table 14-3, "  Recovery Strategies for Process Crashes and System Outages in Infrastructures"

  Use this table if you experience crash or outage in an Infrastructure. Find the type of outage and follow the recommended procedure. The procedures apply to Infrastructures that are installed into a single Oracle home, as well as Infrastructures with Identity Management in one Oracle home and a Metadata Repository in another Oracle home or host.

• Table 14-4, "  Recovery Strategies for Process Crashes and System Outages in Middle-tier Instances"

  Use this table if you experience a crash or outage on a middle-tier installation. Find the type of outage and follow the recommended procedure.

  Table 14-3   Recovery Strategies for Process Crashes and System Outages in Infrastructures

  Type of Outage	How to Check Status and Restart
  Host crash - no data loss	To restart: Reboot the host Start the Infrastructure. Refer to Section 3.2.1, "Starting an Infrastructure".
  Metadata Repository instance failure (loss of the contents of a buffer cache or data residing in memory)	To check status: Try connecting to the database using SQL*Plus. Check the state as follows: SQL> select status from v$instance; To restart: ORACLE_HOME/bin/sqlplus /nolog SQL> connect sys/password as sysdba SQL> startup SQL> quit
  Metadata Repository listener failure	To check status: ORACLE_HOME/bin/lsnrctl status To restart: ORACLE_HOME/bin/lsnrctl start
  Oracle Internet Directory server process (oidldapd) failure	To check status: ORACLE_HOME/ldap/bin/ldapcheck To restart: ORACLE_HOME/opmn/bin/opmnctl startproc ias-component=OID
  Oracle Internet Directory monitor process (oidmon) failure	To check status: ORACLE_HOME/ldap/bin/ldapcheck To restart: ORACLE_HOME/opmn/bin/opmnctl startproc ias-component=OID
  Application Server Control failure	To check status: ORACLE_HOME/bin/emctl status iasconsole To restart: ORACLE_HOME/bin/emctl start iasconsole
  Oracle HTTP Server process failure	To check status: ORACLE_HOME/opmn/bin/opmnctl status To restart: ORACLE_HOME/opmn/bin/opmnctl startproc ias-component=HTTP_Server
  OC4J instance failure	To check status: ORACLE_HOME/opmn/bin/opmnctl status To restart: ORACLE_HOME/opmn/bin/opmnctl startproc process-type=OC4J_instance_name
  DAS instance failure	To check status: ORACLE_HOME/opmn/bin/opmnctl status To restart: ORACLE_HOME/opmn/bin/opmnctl startproc ias-component=OC4J process-type=OC4J_SECURITY
  OPMN daemon failure	To check status: ORACLE_HOME/opmn/bin/opmnctl status To restart: ORACLE_HOME/opmn/bin/opmnctl start

Table 14-4   Recovery Strategies for Process Crashes and System Outages in Middle-tier Instances

Type of Outage	How to Check Status and Restart
Host crash - no data loss	To restart: Reboot Host Start the middle tier. Refer to Section 3.2.3, "Starting a Middle-Tier Instance"
Application Server Control failure	To check status: ORACLE_HOME/bin/emctl status iasconsole To restart: ORACLE_HOME/bin/emctl start iasconsole
Oracle HTTP Server process failure	To check status: ORACLE_HOME/opmn/bin/opmnctl status To restart: ORACLE_HOME/opmn/bin/opmnctl startproc ias-component=HTTP_Server
OC4J instance failure	To check status: ORACLE_HOME/opmn/bin/opmnctl status To restart: ORACLE_HOME/opmn/bin/opmnctl startproc process-type=OC4J_instance_name
OPMN daemon failure	To check status: ORACLE_HOME/opmn/bin/opmnctl status To restart: ORACLE_HOME/opmn/bin/opmnctl start
OracleAS Web Cache failure	To check status: ORACLE_HOME/opmn/bin/opmnctl status To restart: ORACLE_HOME/opmn/bin/opmnctl startproc ias-component=WebCache

14.2 Recovery Procedures

This section contains the procedures for performing different types of recovery.

It contains the following topics:

• Restoring an Infrastructure to the Same Host

• Restoring an Infrastructure to a New Host

• Restoring and Recovering the Metadata Repository

• Restoring Infrastructure Configuration Files

• Restoring a Middle-tier Installation to the Same Host

• Restoring a Middle-tier Installation to a New Host

• Restoring Middle-tier Configuration Files

14.2.1 Restoring an Infrastructure to the Same Host

This section describes how to restore an Infrastructure to the same host. You can use this procedure when you have lost some or all of your Oracle binaries.

It contains the following steps:

• Step 1: Stop the Infrastructure

• Step 2: Restore the Infrastructure Oracle Home

• Step 3: Restore Infrastructure Configuration Files

• Step 4: Restore and Recover the Metadata Repository

• Step 5: Start the Infrastructure

Step 1: Stop the Infrastructure

Refer to Section 3.2.2, "Stopping an Infrastructure" for instructions.

Step 2: Restore the Infrastructure Oracle Home

Note: If your Infrastructure is split and has Identity Management in one Oracle home, and the Metadata Repository in another Oracle home, perform this step on both Oracle homes.

Restore the backup (tar, cpio) of the Infrastructure Oracle home from your complete Oracle Application Server environment backup. Be sure your method of restoring the files preserves the original owner, group, permissions, and timestamps.

Step 3: Restore Infrastructure Configuration Files

Note: If your Infrastructure is split and has Identity Management in one Oracle home, and the Metadata Repository in another Oracle home, perform this step on both Oracle homes.

Restore all configuration files from your most recent backup. You can perform this step using your own procedure or the OracleAS Backup and Recovery Tool. For example, to do this using the tool:

cd BACKUP_TOOL_DIRECTORY
./bkp_restore.pl -m restore_config -t config_bkp_timestamp

See Also: Chapter 12, "Oracle Application Server Backup and Recovery Tool" for more information.

Step 4: Restore and Recover the Metadata Repository

Restore and recover the Metadata Repository from your latest backup. You can perform this step using your own procedure or the OracleAS Backup and Recovery Tool.

See Also: Section 12.7, "Best Practices for Restoring and Recovering the Metadata Repository" to determine the best method for restoring and recovering the Metadata Repository

Step 5: Start the Infrastructure

Refer to Section 3.2.1, "Starting an Infrastructure" for instructions.

14.2.2 Restoring an Infrastructure to a New Host

This section describes how to restore an Infrastructure to a host with the same hostname and IP address as the original host. You can use this procedure to:

• Restore an Infrastructure to the same host after the operating system has been reinstalled. The hostname and IP address must remain the same on the host.

• Restore an Infrastructure to a new host that has the same hostname and IP address as the original host.

It contains the following steps:

• Step 1: Prepare the New Host

• Step 2: Restore Oracle System Files

• Step 3: Restore the Infrastructure Oracle Home

• Step 4: Restore Infrastructure Configuration Files

• Step 5: Restore and Recover the Metadata Repository

• Step 6: Start the Infrastructure

Step 1: Prepare the New Host

If you are restoring to a new host, make sure it has an identical system configuration to the original host. Refer to the record you created in Section 13.2.2, "Creating a Record of Your Oracle Application Server Configuration".

1. On the new host, make sure the following is identical to the original host:
   • Hostname

   • Virtual hostname

   • Domain name

   • IP address

   • Hardware platform

   • Operating system release and patch levels

2. Make sure the entry for the new host in /etc/hosts is identical to the original /etc/hosts file. Make sure the values for IP address, hostname, and aliases are identical and in the same order.

3. Check port usage on the new host. Make sure there aren't any processes using the same ports as the Oracle Application Server installations you are about to restore. If there are, you must reconfigure these processes to use different ports before you begin restoring your Oracle Application Server installations.

4. On the new host, create an operating system user that is identical to the user who installed Oracle Application Server on the original host. The following attributes should be the same:
   • User name

   • Numerical userid

   • Group name

   • Numerical groupid

   • Environment profile

   • Shell

   The user may have the same password or a different password than the original user.

5. Create the Infrastructure Oracle home:
   1. Create an empty Oracle home directory using the same mount point and full path as the original Infrastructure Oracle home. Do not use symbolic links anywhere in the path.

   2. Make sure the directory is on a filesystem with enough space to hold the Infrastructure.

   3. Make sure the directory is owned by the same user and group as on the original host.

Step 2: Restore Oracle System Files

1. Restore the Oracle system files from your complete Oracle Application Server environment backup. For example, on UNIX, these files may be in /var/opt/oracle or /etc.

2. If the oraInventory directory resided in a directory that was separate from the Infrastructure Oracle home, restore it.

Step 3: Restore the Infrastructure Oracle Home

Note: If your Infrastructure is split and has Identity Management in one Oracle home, and the Metadata Repository in another Oracle home, perform this step on both Oracle homes.

Restore the backup (tar, cpio) of the Infrastructure Oracle home from your complete Oracle Application Server environment backup. Be sure your method of restoring the files preserves the original owner, group, permissions, and timestamps.

Step 4: Restore Infrastructure Configuration Files

Note: If your Infrastructure is split and has Identity Management in one Oracle home, and the Metadata Repository in another Oracle home, perform this step on both Oracle homes.

Restore all configuration files from your most recent backup. You can perform this step using your own procedure or the OracleAS Backup and Recovery Tool. For example, to do this using the tool:

cd BACKUP_TOOL_DIRECTORY
./bkp_restore.pl -m restore_config -t config_bkp_timestamp

See Also: Chapter 12, "Oracle Application Server Backup and Recovery Tool" for more information.

Step 5: Restore and Recover the Metadata Repository

Restore and recover the Metadata Repository from your latest complete Oracle Application Server environment backup or online backup, whichever was most recent.

You can perform this step using your own procedure or the OracleAS Backup and Recovery Tool.

See Also: Section 12.7, "Best Practices for Restoring and Recovering the Metadata Repository" to determine the best method for restoring and recovering the Metadata Repository

Step 6: Start the Infrastructure

1. Set file permissions by running the following command as root:

   ORACLE_HOME/root.sh
   
   

2. Start the Infrastructure.

   Refer to Section 3.2.1, "Starting an Infrastructure" for instructions.

14.2.3 Restoring and Recovering the Metadata Repository

This section describes how to restore and recover the Metadata Repository. You can use this when there has only been corruption to the Metadata Repository, and not to any other files in the Oracle home.

Restore and recover the Metadata Repository from your latest backup using your own procedure or the OracleAS Backup and Recovery Tool.

See Also: Section 12.7, "Best Practices for Restoring and Recovering the Metadata Repository" to determine the best method for restoring and recovering the Metadata Repository

14.2.4 Restoring Infrastructure Configuration Files

This section describes how to restore the configuration files in an Infrastructure Oracle home. You can use this procedure when configuration files have been lost or corrupted.

It contains the following steps:

• Step 1: Stop the Infrastructure

• Step 2: Restore Infrastructure Configuration Files

• Step 3: Apply Recent Administrative Changes

• Step 4: Start the Infrastructure

Step 1: Stop the Infrastructure

Refer to Section 3.2.2, "Stopping an Infrastructure" for instructions.

Step 2: Restore Infrastructure Configuration Files

Note: If your Infrastructure is split and has Identity Management in one Oracle home, and the Metadata Repository in another Oracle home, perform this step on both Oracle homes.

Restore all configuration files from your most recent backup. You can perform this step using your own procedure or the OracleAS Backup and Recovery Tool. For example, to do this using the tool:

cd BACKUP_TOOL_DIRECTORY
./bkp_restore.pl -m restore_config -t config_bkp_timestamp

See Also: Chapter 12, "Oracle Application Server Backup and Recovery Tool" for more information.

Step 3: Apply Recent Administrative Changes

If you made any administrative changes since the last time you did an online backup, reapply them now.

See Also: Appendix G, "Examples of Administrative Changes" to learn more about administrative changes

Step 4: Start the Infrastructure

Refer to Section 3.2.1, "Starting an Infrastructure" for instructions.

14.2.5 Restoring a Middle-tier Installation to the Same Host

This section describes how to restore a middle-tier installation to the same host. You can use this procedure when you have lost some or all of your Oracle binaries.

It contains the following steps:

• Step 1: Stop the Middle-tier Instance

• Step 2: Restore the Middle-tier Oracle Home

• Step 3: Restore Middle-tier Configuration Files

• Step 4: Start the Middle-tier Instance

Step 1: Stop the Middle-tier Instance

Refer to Section 3.2.4, "Stopping a Middle-Tier Instance" for instructions.

If the middle-tier instance uses a DCM repository (file-based or database), make sure the DCM repository is up.

Step 2: Restore the Middle-tier Oracle Home

Restore the backup (tar, cpio) of the middle-tier Oracle home from your complete Oracle Application Server environment backup. Be sure your method of restoring the files preserves the original owner, group, permissions, and timestamps.

Step 3: Restore Middle-tier Configuration Files

Restore all configuration files from your most recent backup. You can perform this step using your own procedure or the OracleAS Backup and Recovery Tool. For example, to do this using the tool:

cd BACKUP_TOOL_DIRECTORY
./bkp_restore.pl -m restore_config -t config_bkp_timestamp

See Also: Chapter 12, "Oracle Application Server Backup and Recovery Tool" for more information.

Step 4: Start the Middle-tier Instance

Refer to Section 3.2.3, "Starting a Middle-Tier Instance" for instructions.

14.2.6 Restoring a Middle-tier Installation to a New Host

This section describes how to restore and recover a middle-tier installation to a new host. You can use this procedure to

• Restore a middle-tier installation to the same host after the operating system has been reinstalled

• Restore a middle-tier installation to a new host. The new host may have the same hostname and IP address as the original host, or a different hostname, IP address, or both.

It contains the following steps:

• Step 1: Prepare the New Host

• Step 2: Restore Oracle System Files

• Step 3: Restore the Middle-tier Oracle Home

• Step 4: Restore Middle-tier Configuration Files

• Step 5: Restore the DCM File-based Repository (If Required)

• Step 6: Set the New Hostname and IP Address (If Required)

• Step 7: Start the Middle-tier Instance

Step 1: Prepare the New Host

If you are restoring to a new host, make sure it has an identical system configuration to the original host. Refer to the record you created in Section 13.2.2, "Creating a Record of Your Oracle Application Server Configuration".

1. On the new host, make sure the following is identical to the original host:
   • Hardware platform

   • Operating system release and patch levels

   The new host may have the same or different hostname and IP address.

2. Make sure the entry for the new host in /etc/hosts lists the IP address, hostname, and aliases in the same order as the original /etc/hosts file.

3. Check port usage on the new host. Make sure there aren't any processes using the same ports as the Oracle Application Server installations you are about to restore. If there are, you must reconfigure these processes to use different ports before you begin restoring your Oracle Application Server installations.

4. On the new host, create an operating system user that is identical to the user who installed Oracle Application Server on the original host. The following attributes should be the same:
   • User name

   • Numerical userid

   • Group name

   • Numerical groupid

   • Environment profile

   • Shell

   The user may have the same password or a different password than the original user.

5. Create the middle-tier Oracle home:
   1. Create an empty Oracle home directory using the same mount point and full path as the original middle-tier Oracle home. Do not use symbolic links anywhere in the path.

   2. Make sure the directory is on a filesystem with enough space to hold the middle-tier installation.

   3. Make sure the directory is owned by the same user and group as on the original host.

Step 2: Restore Oracle System Files

1. Restore the Oracle system files from your complete Oracle Application Server environment backup. For example, on UNIX, these files may be in /var/opt/oracle or /etc.

2. If the oraInventory directory resided in a directory that was separate from the middle-tier Oracle home, restore it.

Step 3: Restore the Middle-tier Oracle Home

Restore the backup (tar, cpio) of the middle-tier Oracle home from your complete Oracle Application Server environment backup. Be sure your method of restoring the files preserves the original owner, group, permissions, and timestamps.

Step 4: Restore Middle-tier Configuration Files

Restore all configuration files from your most recent backup. You can perform this step using your own procedure or the OracleAS Backup and Recovery Tool. For example, to do this using the tool:

cd BACKUP_TOOL_DIRECTORY
./bkp_restore.pl -m restore_config -t config_bkp_timestamp

See Also: Chapter 12, "Oracle Application Server Backup and Recovery Tool" for more information.

Step 5: Restore the DCM File-based Repository (If Required)

This step is required only if all of the following conditions are true:

• You are using a DCM file-based repository

• You are restoring to a new host

• You are restoring the instance that contains the DCM file-based repository (the repository host instance)

Since the DCM file-based repository on the original host was lost, you must restore (import) the DCM file-based repository to the new host as follows:

1. Stop the DCM daemon on all other instances in the farm by running the following command in the Oracle home of each instance:

   ORACLE_HOME/opmn/bin/opmnctl stopproc ias-component=dcm-daemon
   
   

2. Restore (import) the DCM file-based repository to the new host:

   ORACLE_HOME/dcm/bin/dcmctl importRepository -file file_name
   
   

   Where file_name is the repository backup you made during your most recent backup.

3. Start the DCM daemon on all other instances in the farm by running the following command in the Oracle home of each instance (do not start DCM in the instance you are currently restoring):

   ORACLE_HOME/opmn/bin/opmnctl startproc ias-component=dcm-daemon
   
   

When you run the importRepository command, the middle-tier instance you are currently restoring on the new host becomes the repository host instance. If you intend to continue to use the original host, you must notify the original host that it is no longer the repository host instance. To do this, run the following command in the middle-tier instance on the original host:

ORACLE_HOME/dcm/bin/dcmctl repositoryRelocated

See Also: Oracle Application Server 10g High Availability Guide for instructions on importing a DCM file-based repository

Step 6: Set the New Hostname and IP Address (If Required)

1. Set file permissions by running the following command as root:

   ORACLE_HOME/root.sh
   
   

2. If the new host has a different hostname or IP address as the original host, follow the procedure in Section 9.3, "Changing the Hostname and IP Address (Middle Tier)" to change the hostname, IP address, or both, as required.

Step 7: Start the Middle-tier Instance

Start OPMN and OPMN-managed processes:

ORACLE_HOME/opmn/bin/opmnctl startall

Start Application Server Control:

ORACLE_HOME/bin/emctl start iasconsole

14.2.7 Restoring Middle-tier Configuration Files

This section describes how to restore the configuration files in a middle-tier Oracle home. Use this procedure when configuration files have been lost or corrupted.

It contains the following steps:

• Step 1: Stop the Middle-tier Instance

• Step 2: Restore Middle-tier Configuration Files

• Step 3: Apply Recent Administrative Changes

• Step 4: Start the Middle-tier Instance

Step 1: Stop the Middle-tier Instance

Refer to Section 3.2.4, "Stopping a Middle-Tier Instance" for instructions.

If the middle-tier instance uses a DCM repository (file-based or database), make sure the DCM repository is up.

Step 2: Restore Middle-tier Configuration Files

Restore all configuration files from your most recent backup. You can perform this step using your own procedure or the OracleAS Backup and Recovery Tool. For example, to do this using the tool:

cd BACKUP_TOOL_DIRECTORY
./bkp_restore.pl -m restore_config -t config_bkp_timestamp

See Also: Chapter 12, "Oracle Application Server Backup and Recovery Tool" for more information

Step 3: Apply Recent Administrative Changes

If you made any administrative changes since the last time you did an online backup, reapply them now.

See Also: Appendix G, "Examples of Administrative Changes" to learn more about administrative changes

Step 4: Start the Middle-tier Instance

Refer to Section 3.2.3, "Starting a Middle-Tier Instance" for instructions.