15 Introducing Backup and Recovery

This chapter provides an introduction to backing up and recovering Oracle Fusion Middleware, including backup and recovery recommendations for Oracle Fusion Middleware components.

It contains the following sections:

Understanding Oracle Fusion Middleware Backup and Recovery
Oracle Fusion Middleware Directory Structure
Overview of the Backup Strategies
Overview of Recovery Strategies
Backup and Recovery Recommendations for Oracle Fusion Middleware Components
Assumptions and Restrictions

15.1 Understanding Oracle Fusion Middleware Backup and Recovery

The installations of an Oracle Fusion Middleware environment are interdependent in that they contain configuration information, applications, and data that are kept in synchronization. For example, when you perform a configuration change, information in configuration files is updated. When you deploy an application, you might deploy it to all Managed Servers in a domain or cluster.

It is, therefore, important to consider your entire Oracle Fusion Middleware environment when performing backup and recovery. You should back up your entire Oracle Fusion Middleware environment at once, then periodically. If a loss occurs, you can restore your environment to a consistent state.

The following topics describe concepts that are important to understanding backup and recovery:

Impact of Administration Server Failure
Managed Server Independence (MSI) Mode
Configuration Changes in Managed Servers

15.1.1 Impact of Administration Server Failure

The failure of an Administration Server does not affect the operation of Managed Servers in the domain but it does prevent you from changing the domain's configuration. If an Administration Server fails because of a hardware or software failure on its host computer, other server instances on the same computer may be similarly affected.

If an Administration Server for a domain becomes unavailable while the server instances it manages—clustered or otherwise—are running, those Managed Servers continue to run. Periodically, these Managed Servers attempt to reconnect to the Administration Server. For clustered Managed Server instances, the load balancing and failover capabilities supported by the domain configuration continue to remain available.

When you first start a Managed Server, it must be able to connect to the Administration Server to retrieve a copy of the configuration. Subsequently, you can start a Managed Server even if the Administration Server is not running. In this case, the Managed Server uses a local copy of the domain's configuration files for its starting configuration and then periodically attempts to connect with the Administration Server. When it does connect, it synchronizes its configuration state with that of the Administration Server.

15.1.2 Managed Server Independence (MSI) Mode

A Managed Server maintains a local copy of the domain configuration. When a Managed Server starts, it contacts its Administration Server to retrieve any changes to the domain configuration that were made since the Managed Server was last shut down. If a Managed Server cannot connect to the Administration Server during startup, it can use its locally cached configuration information—this is the configuration that was current at the time of the Managed Server's most recent shutdown. A Managed Server that starts without contacting its Administration Server to check for configuration updates is running in Managed Server Independence (MSI) mode. By default, MSI mode is enabled. However a Managed Server cannot be started even in MSI mode for the first time if the Administration Server is down due to non-availability of the cached configuration.

15.1.3 Configuration Changes in Managed Servers

Configuration changes are updated in a Managed Server during the following events:

On each Managed Server restart, the latest configuration is retrieved from the Administration Server. This happens even when Node Manager is down on the node where the Managed Server is running. If the Administration Server is unavailable during the Managed Server restart and if the MSI (Managed Server Independence) mode is enabled in the Managed Server, it starts by reading its local copy of the configuration and synchronizes with the Administration Server when it is available. By default MSI mode is enabled.
Upon activating every administrative change such as configuration changes, deployment or redeployment of applications, and topology changes, the Administration Server pushes the latest configuration to the Managed Server. If the Managed Server is not running, the Administration Server pushes the latest version of the configuration to the Managed Server when it does start.

15.2 Oracle Fusion Middleware Directory Structure

The following shows a simplified view of the Oracle Fusion Middleware directory structure when you have installed and configured the Oracle Fusion Middleware Infrastructure:

Description of the illustration ascon_dt_011.png

15.3 Overview of the Backup Strategies

To back up your Oracle Fusion Middleware environment, you can use:

File copy utilities such as copy, xcopy, tar, or jar. Make sure that the utilities:
- Preserve symbolic links
- Support long file names
- Preserve the permissions and ownership of the files
For example:
- On Windows, for online backups, use copy; for offline backups, use copy, xcopy, or jar. Do not use Winzip because it does not work with long filenames or extensions.
  
  Note that for some versions of Windows, any file name with more than 256 characters fails. You can use the xcopy command with the following switches to work around this issue:
```
xcopy /s/e  "C:\Temp\*.*"  "C:\copy"
```
  See the xcopy help for more information about syntax and restrictions.
- On Linux and UNIX, for online and offline backups, use tar.
Oracle Recovery Manager (RMAN) to back up database-based metadata repositories and any databases used by Oracle Fusion Middleware. With RMAN, you can perform full backups or incremental backups. See Oracle Database Backup and Recovery User's Guide for information about using RMAN to back up a database.

If you want to retain your backups for a longer duration, you may want to back up to tape, for example using Oracle Secure Backup.

You can also configure Oracle WebLogic Server to make backup copies of the configuration files. This facilitates recovery in cases where configuration changes need to be reversed or in the unlikely case that configuration files become corrupted. When the Administration Server starts, it saves a .jar file named config-booted.jar that contains the configuration files. When you make changes to the configuration files, the old files are saved in the configArchive directory under the domain directory, in a .jar file with a sequentially numbered name such as config-1.jar. However, the configuration archive is always local to the Administration Server host. It is a best practice to back up the archives to an external location.

15.3.1 Types of Backups

You can back up your Oracle Fusion Middleware environment offline or online:

An offline backup means that you must shut down the environment before backing up the files. When you perform an offline backup, the Administration Server, all Managed Servers in the domain should be shut down.

Back up the environment offline immediately after installation and after applying any patches or upgrades.
An online backup means that you do not shut down the environment before backing up the files. To avoid an inconsistent backup, do not make any configuration changes until the backup is completed. To ensure that no changes are made in the WebLogic Server domain, lock the WebLogic Server configuration, as described in Section 2.3.2.

You can perform backups on your full Oracle Fusion Middleware environment, or on the run-time artifacts, which are those files that change frequently.

To perform a full backup, you should back up the static files and directories, as well as run-time artifacts, which are described in Section 15.3.2.

15.3.2 Backup Artifacts

Backup artifacts include static files and directories and run-time artifacts.

Static files and directories are those that do not change frequently. These include:

The Oracle home (ORACLE_HOME). An Oracle home consists of product homes, such as the WebLogic Server home and an Oracle Common home. It can also contain the user_projects directories, which contains Oracle WebLogic Server domains, which are not static files.
OraInventory
On Linux and UNIX, the oraInst.loc file, which is located in the following directory:
```
(Linux and IBM AIX) /etc
(Other UNIX systems) /var/opt/oracle
```
On Linux and UNIX, the oratab file, which is located in the following directory:
```
/etc
```

The beahomelist file, which is located at:

(UNIX) user_home/bea/beahomelist
(Windows) C:\bea\beahomelist

On Windows, the following registry key:
```
HKEY_LOCAL_MACHINE\Software\oracle
```
In addition, for system components, such as Oracle HTTP Server, you must back up the following Windows Registry key:
```
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services
```

Run-time artifacts are those files that change frequently. Back up these files when you perform a full backup and on a regular basis. Run-time artifacts include:

Domain directories of the Administration Server and the Managed Servers (by default, a domain directory resides in Oracle home, but it can be configured by the user to point to a different location.)

In most cases, you do not need to back up Managed Server directories separately because the Administration Server contains information about all of the Managed Servers in its domain.
Application artifacts, such as .ear or .war files that reside outside of the domain.

You do not need to back up application artifacts in a Managed Server directory structure because they can be retrieved from the Administration Server during Managed Server startup.
Database artifacts, such as the MDS Repository.
Any database-based metadata repositories used by Oracle Fusion Middleware. You use Oracle Recovery Manager (RMAN) to back up an Oracle database.
Persistent stores, such as JMS Providers and transaction logs, which reside by default in the user_projects directory, but can be configured in a different location.

15.3.3 Recommended Backup Strategy

This section outlines the recommended strategy for performing backups. Using this strategy ensures that you can perform the recovery procedures in this book.

Perform a full offline backup: This involves backing up the entities described in Section 15.3.2. Perform a full offline backup at the following times:
- Immediately after you install Oracle Fusion Middleware
- Immediately before upgrading your Oracle Fusion Middleware environment
- Immediately after an operating system software upgrade
- Immediately after upgrading or patching Oracle Fusion Middleware
Perform an online backup of run-time artifacts: This involves backing up the run-time artifacts described in Section 15.3.2. Backing up the run-time artifacts enables you to restore your environment to a consistent state as of the time of your most recent configuration and metadata backup. To avoid an inconsistent backup, do not make any configuration changes until backup completes. Perform an online backup of run-time artifacts at the following times:
- On a regular basis. Oracle recommends that you back up run-time artifacts nightly.
- Prior to making configuration changes to a component.
- After making configuration changes to a component.
- Prior to deploying a custom Java EE application to a Managed Server or cluster.
- After a major change to the deployment architecture, such as creating servers or clusters.
Perform a full or incremental backup of your databases: Use RMAN to backup your databases. See the Oracle Database Backup and Recovery User's Guide for information about using RMAN and for suggested methods of backing up the databases.

15.4 Overview of Recovery Strategies

Recovery strategies enable you to recover from critical failures that involve actual data loss. Depending on the type of loss, they can involve recovering any combination of the following types of files:

Oracle software files
Configuration files
Oracle system files
Windows Registry keys
Application artifacts

You can recover your Oracle Fusion Middleware environment while Oracle Fusion Middleware is offline.

To recover your Oracle Fusion Middleware environment, you can use:

File copy utilities such as copy, xcopy, or tar

When you restore the files, use your preferred tool to extract the compressed files:
- On Windows, for online recovery, use copy; for offline recovery, use copy, xcopy, or jar.
  
  Note that for some versions of Windows, any file name with more than 256 characters fails. You can use the xcopy command with the following switches to work around this issue:
```
xcopy /s/e  "C:\Temp\*.*"  "C:\copy"
```
  See the xcopy help for more information about syntax and restrictions.
  
  Do not use Winzip because it does not work with long filenames or extensions.
- On Linux and UNIX, use tar.
Ensure that the tool you are using preserves the permissions and timestamps of the files.
Oracle Recovery Manager (RMAN) to recover database-based metadata repositories

15.4.1 Types of Recovery

You can recover your Oracle Fusion Middleware environment in part or in full. You can recover the following:

The Oracle home
WebLogic Server domains
The WebLogic Server Administration Server
WebLogic Server Managed Servers
A component, such as Oracle HTTP Server
WebLogic Server cluster
Deployed applications

15.4.2 Recommended Recovery Strategies

Note the following key points about recovery:

Your Oracle Fusion Middleware environment must be offline while you are performing recovery.
Rename important existing files and directories before you begin restoring the files from backup so that you do not unintentionally override necessary files.
Although, in some cases, it may appear that only one or two files are lost or corrupted, you should restore the directory structure for the entire element, such as a domain, rather than just restoring one or two files. In this way, you are more likely to guarantee a successful recovery.
Recover the database to the most current state, using point-in-time recovery (if the database is configured in Archive Log Mode). This is typically a time right before the database failure occurred.

15.5 Backup and Recovery Recommendations for Oracle Fusion Middleware Components

The following sections describe backup and recovery recommendations for specific Oracle Fusion Middleware components:

Backup and Recovery Recommendations for Oracle WebLogic Server
Backup and Recovery Recommendations for Oracle JRF Installations
Backup and Recovery Recommendations for Web Tier Installations
Backup and Recovery Recommendations for Oracle Data Integrator

For the steps you take to back up your environment, see Section 16.3. For the steps you take to recover your environment, see Chapter 17.

15.5.1 Backup and Recovery Recommendations for Oracle WebLogic Server

The following sections describe backup and recovery recommendations for Oracle WebLogic Server:

Backup and Recovery Recommendations for Oracle WebLogic Server
Backup and Recovery Recommendations for Oracle WebLogic Server JMS

15.5.1.1 Backup and Recovery Recommendations for Oracle WebLogic Server

This section describes the Oracle WebLogic Server data that must be backed up and restored.

Database Repository Dependencies

Oracle WebLogic Server does not, by default, depend on any database repository. However, applications deployed on Oracle WebLogic Server may use databases as data sources. To back up a database, see the Oracle Database Backup and Recovery User's Guide.

Backup Recommendations

Back up the Oracle home and the domain.

Recovery Recommendations

Depending on what has failed, you may need to recover the following:

The domain: See Section 17.2.2.
The Administration Server configuration: See Section 17.2.4.
A Managed Server: See Section 17.2.5.
A cluster: See Section 17.2.7.
Applications: See Section 17.2.8.

If you use Whole Server Migration, the leasing information is stored in a table in a database. If you recover Oracle WebLogic Server, you should discard the information in the leasing table. (For more information about Whole Server Migration, see "Whole Server Migration" in Administering Clusters for Oracle WebLogic Server.)

After a loss of host, you may need to recover the following:

The WebLogic domain. See Section 17.3.1.
The Administration Server host: See Section 17.3.3.
The Managed Server host: See Section 17.3.4.

15.5.1.2 Backup and Recovery Recommendations for Oracle WebLogic Server JMS

This section describes the Oracle WebLogic Server JMS data that must be backed up and restored.

Database Repository Dependencies

Only if JMS is database-based

Backup Recommendations

Back up the domain.

If you are using a database-based JMS, back up the database using RMAN.

If you are using file-based JMS, use storage snapshot techniques for taking consistent online backups. Alternatively, you can use a file system copy to perform an offline backup.

Recovery Recommendations

Recover the domain.

If the JMS persistent store is file-based, recover it from backup. If the JMS persistent store is database-based, recover the database to the most recent point in time, if needed. Note the following:

Always try to keep JMS data as current as possible. This can be achieved by using the point-in-time recovery capabilities of Oracle Database, recovering to the most recent time (in the case of database-based persistence) or using a highly available RAID-backed storage device (for example, SAN/NAS).
If you are using a file-based JMS, you can use storage snapshots to recover.
If, for whatever reason, you need to restore JMS data to a previous point in time, there are potential implications. Restoring the system state to a previous point in time not only can cause duplicate messages, but can also cause lost messages. The lost messages are messages that were enqueued before or after the system restore point time, but never processed.

Use the following procedure before recovery to drain messages in the JMS queue after persistent-store recovery to avoid processing duplicate messages:

Note:
Do not drain and discard messages without first being certain that the messages contain no data that must be preserved. The recovered messages may include unprocessed messages with important application data, in addition to duplicate messages that have already been processed.
1. Log into the Oracle WebLogic Server Administration Console.
2. Before recovery, configure JMS server to pause Production, Insertion, and consumption operations at boot time to ensure that no new messages are produced or inserted into the destination or consumed from the destination before you drain stale messages. To do this:
  1. Expand Services, then Messaging, and then click JMS Servers.
  2. On the Summary of JMS Servers page, click the JMS server you want to configure for message pausing.
  3. On the Configuration: General page, click Advanced to define the message pausing options. Select Insertion Paused At Startup, Production Paused At Startup, and Consumption Paused At Startup.
  4. Click Save.
Use the following procedure after recovery:
1. After recovering the persistent store, start the Managed Servers.
2. Drain the stale messages from JMS destinations, by taking the following steps:
  1. Expand Services, then Messaging, and then JMS Modules.
  2. Select a JMS module, then select a target.
  3. Select Monitoring, then Show Messages.
3. Click Delete All.
4. Resume operations, by taking the following steps:
  1. Expand Services, then Messaging, and then JMS Servers.
  2. On the Summary of JMS Servers page, click the JMS server you want to configure for message pausing.
  3. On the Configuration: General page, click Advanced. Deselect Insertion Paused At Startup, Production Paused At Startup, and Consumption Paused At Startup.
  4. Click Save.
If the store is not dedicated to JMS use, use the Oracle WebLogic Server JMS message management administrative tool. This tool can perform import, export, move, and delete operations from the Administration Console, MBeans, and WLST.

For applications that use publish and subscribe in addition to queuing, you should manipulate topic subscriptions in addition to queues.

For the steps to recover the domain, see Section 17.2.2 and Section 17.3.1.

15.5.2 Backup and Recovery Recommendations for Oracle JRF Installations

The following topics describe backup and recovery recommendations for components that are installed with more than one type of installation:

Backup and Recovery Recommendations for Oracle Web Services Manager
Backup and Recovery Recommendations for Oracle Platform Security Services

15.5.2.1 Backup and Recovery Recommendations for Oracle Web Services Manager

This section describes the Oracle Web Services Manager data that must be backed up and restored.

Database Repository Dependencies

If a database-based MDS Repository is used, Oracle Web Services Manager uses a partition in the MDS schema.

Backup Recommendations

Back up the Oracle Web Services Manager domain.

If Oracle Web Services Manager uses a file-based MDS Repository, back it up using a file copy mechanism. If it uses a database-based MDS Repository, back up the database using RMAN.

Recovery Recommendations

Restore the Oracle Web Services Manager Managed Server.

If Oracle Web Services Manager uses a file-based MDS Repository, restore it from the backup. If it uses a database-based MDS Repository, recover the database to the most recent point in time, if needed.

For the steps to recover components, see Section 17.2.6. For the steps specific to recovering from loss of host, see Section 17.3.5.

15.5.2.2 Backup and Recovery Recommendations for Oracle Platform Security Services

This section describes the Oracle Platform Security Services data that must be backed up and restored.

Database Repository Dependencies

If a database-based Oracle Platform Security Repository is used, Oracle Platform Security uses a partition in the OPSS schema.

If an Oracle Internet Directory based Oracle Platform Security repository is used, Oracle Platform Security uses Oracle Internet Directory.

Backup Recommendations

Back up the Administration Server domain, including the following files:

DOMAIN_HOME/config/fmwconfig/jps-config.xml
DOMAIN_HOME/config/fmwconfig/jps-config-jse.xml
DOMAIN_HOME/config/fmwconfig/cwallet.sso
DOMAIN_HOME/config/fmwconfig/bootstrap/cwallet.sso
DOMAIN_HOME/config/fmwconfig/keystores.xml
DOMAIN_HOME/config/config.xml
DOMAIN_HOME/config/fmwconfig/ids_config.xml
DOMAIN_HOME/config/fmwconfig/system-jazn-data.xml (if present)
DOMAIN_HOME/config/fmwconfig/jps_mbeans.xml

Back up Oracle Internet Directory if Oracle Platform Security uses an Oracle Internet Directory based repository.

Backup the database containing the OPSS schema if Oracle Platform Security uses a database-based repository.

Recovery Recommendations

Restore the jps-config.xml file.

If Oracle Platform Security uses a database-based repository, restore the database to the most recent point in time.

For the steps to recover components, see Section 17.2.6. For the steps specific to recovering from loss of host, see Section 17.3.5.

15.5.3 Backup and Recovery Recommendations for Web Tier Installations

The following sections describe backup and recovery recommendations for Web Tier installations:

Backup and Recovery Recommendations for Oracle HTTP Server

15.5.3.1 Backup and Recovery Recommendations for Oracle HTTP Server

This section describes the Oracle HTTP Server data that must be backed up and restored.

Database Repository Dependencies

None

Backup Recommendations

For Oracle HTTP Server in a standalone domain, back up the domain. For Oracle HTTP Server in a WebLogic Server domain, back up the domain.

Recovery Recommendations

For Oracle HTTP Server in a standalone domain, restore the domain that contains Oracle HTTP Server, as described in Section 17.2.3. For loss of host, restore the domain, as described in Section 17.3.2 and make configuration changes as described in Section 17.3.5.5.1 or Section 17.3.5.5.2.

For Oracle HTTP Server in a WebLogic Server domain, restore the domain, as described in Section 17.2.2. For loss of host, restore the domain as described in Section 17.3.1 and make configuration changes as described in Section 17.3.5.5.2.

15.5.4 Backup and Recovery Recommendations for Oracle Data Integrator

This section describes the Oracle Data Integrator data that must be backed up and restored.

Database Repository Dependencies

ODI_REPO schema

Backup Recommendations

Back up the Oracle home, domain if Oracle Data Integrator is installed in a domain, and the ODI_Oracle_Home/oracledi/agent folder for each machine where a standalone agent is installed.

Back up the database containing Oracle Data Integrator schema.

Recovery Recommendations

Recover the database to the most recent point in time, if needed.

Depending on the extent of the failure and the type of Oracle Data Integrator installation, restore the Managed Server or the Oracle home, or both.

If your environment contains the Oracle Data Integrator Standalone Agent, restore the Oracle home, as described in Section 17.2.1.

If your environment contains Oracle Data Integrator for Developers, restore the Oracle home, as described in Section 17.2.1.

If your environment contains Oracle Data Integrator deployed in a Managed Server, restore the Managed Server, as described in Section 17.2.5.

For the steps to recover Oracle Data Integrator from loss of host, see Section 17.3.5.6.

15.6 Assumptions and Restrictions

The following assumptions and restrictions apply to the backup and recovery procedures in this book. Also see the restrictions listed in Section 16.2.

Only the user who installs the product or a user who has access privileges to the directories where Oracle Fusion Middleware has been installed should be able to execute backup and recovery operations.
If a single Managed Server and Administration Server run on different hosts and the Managed Server is not in a cluster, you must use the pack and unpack commands on the Managed Server to retrieve the correct configuration.