18 Backing Up and Recovering Oracle Fusion Applications

This chapter describes recommended back up and recovery strategies and procedures for recovering your Oracle Fusion Applications environment from different types of failures and outages, such as data loss or corruption, host failure, or media failure.

This chapter contains the following topics:

• Section 18.1, "Introduction to Backup and Recovery"

• Section 18.2, "Overview of Backing Up Your Environment"

• Section 18.3, "Overview of Recovering Your Environment"

• Section 18.4, "Prerequisites for Using Cloud Control to Back Up or Restore Your Environment"

• Section 18.5, "Backup and Recovery Recommendations for Oracle Fusion Applications"

• Section 18.6, "Performing a Backup"

• Section 18.7, "Creating a Record of Your Oracle Fusion Applications Configuration"

• Section 18.8, "Recovering Using Cloud Control"

• Section 18.9, "Recovering After Data Loss, Corruption, or Media Failure"

• Section 18.10, "Recovering After Loss of Host"

• Section 18.11, "A Case Study: Recovering Oracle Fusion Customer Relationship Management"

18.1 Introduction to Backup and Recovery

An Oracle Fusion Applications environment can consist of different Oracle Fusion Applications product families. It is built on Oracle Fusion Middleware, which contains Oracle WebLogic Server domains with Java components, such as Oracle SOA Suite, and system components, such as Oracle HTTP Server. It also contains a separate Oracle WebLogic Server domain with Identity Management components, such as Oracle Internet Directory and Oracle Virtual Directory.

Oracle Fusion Applications and some Oracle Fusion Middleware components use Oracle Database instances to store data and metadata.

Note that Oracle Fusion Applications uses at least three databases:

• A database that holds the LDAP repository

• A database that holds the Oracle Fusion Applications data

• A database that holds the Oracle Identity Manager repository

The installations of an Oracle Fusion Applications environment are interdependent in that they contain configuration information, applications, and data that are kept in synchronization.

It is, therefore, important to consider your entire Oracle Fusion Applications environment when performing backup and recovery. You should back up your entire Oracle Fusion Applications environment as soon as you have installed and configured it, then periodically. If a loss occurs, you can restore your environment to a consistent state.

18.2 Overview of Backing Up Your Environment

You should back up your environment when you install and configure Oracle Fusion Applications and on a regular basis. You can back up your full environment or you can back up only parts of it. You can perform the backups in online or offline mode.

This section includes the following topics:

• Section 18.2.1, "Tools to Use to Back Up Your Environment"

• Section 18.2.2, "Modes of Backup"

• Section 18.2.3, "Types of Backups"

• Section 18.2.4, "Recommended Backup Strategy"

18.2.1 Tools to Use to Back Up Your Environment

To back up your Oracle Fusion Applications environment, you can use:

• Oracle Enterprise Manager Cloud Control

  With Cloud Control, you can back up the directories and databases that comprise a Oracle Fusion Applications installation. Because Cloud Control uses Oracle Secure Backup to perform Oracle Fusion Applications backups, all backups are done to tape, which is the media supported by Oracle Secure Backup.

• File copy utilities such as copy, xcopy, tar, or jar. Make sure that the utilities:

  • Preserve symbolic links

  • Support long file names

  • Preserve the permissions and ownership of the files

  For example:

  • On Windows, for online backups, you can use copy; for offline backups, you can use copy, xcopy, tar, or jar. Ensure that the backup utility you use can support long and unicode file names and extensions. Many of the early archiving utilities did not have this support.

    Note that for some versions of Windows, any file name with more than 256 characters will fail. You can use the xcopy command with the following switches to work around this issue:

    xcopy /s/e  "C:\Temp\*.*"  "C:\copy"
    

    See the xcopy help for more information about syntax and restrictions.

  • On UNIX, for online and offline backups, you can use tar.

  See Section 18.2.2 for descriptions of online and offline backups.

• Oracle Recovery Manager (RMAN) to back up database-based metadata repositories and any databases used by Oracle Fusion Applications. With RMAN, you can perform full backups or incremental backups. See Oracle Database Backup and Recovery User's Guide for information about using RMAN to back up a database.

If you want to retain your backups for a longer duration, consider backing up to tape, for example using Oracle Secure Backup.

For optimized backup time and fast restore times for the file system, you can use a storage snapshot feature provided by a storage vendor. Snapshots are point-in-time, read-only copies of the file system. Snapshots normally use copy-on-write mechanisms and hence do not occupy any space at the beginning. When new data is written, the old copy is written to the snapshot. Most of the storage vendors support an unlimited number of snapshots and allow creating snapshots manually or automatically without any user interventions. Snapshots can be taken quickly, thus reducing the time taken when performing a backup. Taking regular snapshots of the Oracle Fusion Applications configuration helps you restore the Oracle Fusion Applications environment quickly in the event of configuration corruption or data loss. Snapshots provide rollback capabilities. When a rollback occurs, any newer snapshots (and clones of newer snapshots) are destroyed, and the active data reverts to the state when the snapshot was taken.

18.2.2 Modes of Backup

You can back up your Oracle Fusion Applications environment offline or online:

• With an offline backup, you must shut down the environment before performing the backup. When you perform an offline backup, the Administration Server, all Managed Servers in the domain, and all system components in the Oracle instances should be shut down.

  Back up the environment offline immediately after installation and after applying upgrades.

• With an online backup, you do not shut down the environment before backing up the files. To avoid an inconsistent backup, do not make any configuration changes until the backup is completed. To ensure that no changes are made in the WebLogic Server domain, lock the WebLogic Server configuration, as described in "Locking the WebLogic Server Configuration" in the Oracle Fusion Middleware Administrator's Guide.

  During an online backup, applications can continue to run during the backup, so your business is not affected.

18.2.3 Types of Backups

You should back up your Oracle Fusion Applications file system and your Oracle Fusion Applications databases.

For the Oracle Fusion Applications file system, you can perform a full backup or you can perform a partial backup. See Figure 1-6 for a graphic of the directory structure in the Oracle Fusion Applications file system.

To perform a full backup of the file system, you should back up the binary files, as well as the configuration files.

Binary files are static files and directories that do not change frequently. These include:

• The Applications base directory, which is the top-level directory containing Oracle Fusion Applications, Middleware homes, and Oracle homes.

• The Middleware home (MW_HOME). A Middleware home consists of a WebLogic Server home, an Oracle Common home, and optionally one or more Oracle homes and one or more Oracle instances.

• OraInventory

• On UNIX, the OraInst.loc file, which is located in the following directory:

  (Linux and IBM AIX) /etc
  (Other UNIX systems) /var/opt/oracle
  

• On UNIX, the oratab file, which is located in the following directory:

  /etc
  

• The beahomelist file, which is located at:

  (UNIX) user_home/bea/beahomelist
  (Windows) C:\bea\beahomelist
  

• On Windows, the following registry key:

  HKEY_LOCAL_MACHINE\Software\oracle
  

  In addition, for system components, such as Oracle HTTP Server, you must back up the following Windows Registry key:

  HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services
  

Configuration files are those files that change frequently. Back up these files when you perform a full backup and on a regular basis. Configuration files include:

• The Applications configuration directory, which is the top-level directory containing domains and Oracle instances.

• Domain directories of the Administration Server and the Managed Servers. The Oracle Fusion Applications environment can consist of multiple domains (for example, CRMDomain and FinancialDomain). Each domain consists of an Administration Server and one or more Managed Servers.

  Unless stated in the backup recommendations for a particular type of Oracle Fusion application, you do not need to back up Managed Server directories separately because the Administration Server contains information about all of the Managed Servers in its domain.

• All Oracle instance homes, which reside by default in the MW_HOME, but can be configured to be in a different location.

For the Oracle Fusion Applications database and related databases, you can perform full or incremental backups. You use Oracle Recovery Manager to back up an Oracle Database instance.

Note that you must keep the databases synchronized when you restore them.

18.2.4 Recommended Backup Strategy

This section outlines the recommended strategy for performing backups. Using this strategy ensures that you can perform the recovery procedures in this book.

Note:

Store your backups in a secure location, that is, not on the same hardware that contains your Oracle Fusion Applications environment.

• Perform a full offline backup: Back up the binary files and directories and the configuration files described in Section 18.2.3. If the Applications base directory is shared, you only need to back it up once. If the Applications base directory is not shared, back it up on each host in your Oracle Fusion Applications environment. Perform a full offline backup at the following times:

  • Immediately after you install Oracle Fusion Applications

  • Immediately after an operating system software upgrade.

  • Immediately before upgrading your Oracle Fusion Applications environment

  • Immediately after upgrading your Oracle Fusion Applications environment

  • Immediately before patching your Oracle Fusion Applications environment.

• Perform an online backup of configuration files: Back up the configuration files described in Section 18.2.3. Backing up the configuration files enables you to restore your environment to a consistent state as of the time of your most recent configuration and metadata backup. You can back up the configuration files at the following granularity:

  • The Applications configuration directory.

  • The domain.

  • The instance. You should back up the files in the following directories:

    For the CRM Domain: instance/domains/host_name/CRMDomain
    For the HCM Domain: instance/domains/host_name/HCMDomain
    For the FIN Domain: instance/domains/host_name/FinancialDomain
    For the PRJ Domain: instance/domains/host_name/PRJDomain
    For the Webtier home: instance/CommonDomain_webtier
    For the BI Instance: instance/BIInstance
    

  To avoid an inconsistent backup, do not make any configuration changes until the backup completes.

  Perform an online backup of configuration files at the following times:

  • On a regular basis. Oracle recommends that you back up configuration files nightly.

  • Prior to making configuration changes to an Administration Server, a Managed Server, or application.

  • Immediately after patching your Oracle Fusion Applications environment.

  • After making configuration changes to an Administration Server, a Managed Server, Oracle instance, or application.

  • After a major change to the deployment architecture, such as creating servers or clusters.

• Perform a full or incremental backup of your databases: Use RMAN to backup your databases. See the Oracle Database Backup and Recovery User's Guide for information about using RMAN and for suggested methods of backing up the databases.

Note the following recommendations:

• To ensure that no changes are made in the WebLogic Server domains, lock the WebLogic Server configuration for all the domains in Oracle Fusion Applications environment, as described in "Locking the WebLogic Server Configuration" in the Oracle Fusion Middleware Administrator's Guide.

• When you create the backup, name the archive file with a unique name. Consider appending the date and time to the name. For example, if you create a backup of the Applications base directory on March 2 2012, name the backup:

  ApplBase_backup_030212.tar
  

18.3 Overview of Recovering Your Environment

If your environment suffers from critical failures that involve actual data corruption, data loss, or loss of host, you must recover all or part of your environment.

This section includes the following topics:

• Section 18.3.1, "Tools to Use to Recover Your Environment"

• Section 18.3.2, "Types of Recovery"

• Section 18.3.3, "Recommended Recovery Strategies"

18.3.1 Tools to Use to Recover Your Environment

To recover your Oracle Fusion Applications environment, you can use:

• Oracle Enterprise Manager Cloud Control

  With Cloud Control, you can restore the directories and databases that comprise a Oracle Fusion Applications installation.

• File copy utilities, such as copy, xcopy or tar.

  When you restore the files, use your preferred tool to extract the compressed files.

  For example, for online recovery on Windows, you can use copy; for offline recovery on Windows, you can use copy, xcopy, or jar. Ensure that the utility you use can support long and unicode file names and extensions. Many of the early archiving utilities did not have this support."

  For example, for UNIX, you can use tar.

• Oracle Recovery Manager (RMAN) to recover database-based metadata repositories and Oracle Fusion Applications databases. See the Oracle Database Backup and Recovery User's Guide for information about using RMAN to recover a database.

18.3.2 Types of Recovery

Recovery strategies enable you to recover from critical failures that involve actual data corruption, data loss, or loss of host. Depending on the type of loss, you can recover your Oracle Fusion Applications environment in part or in full. You can recover the following:

• The Applications base directory

• The Oracle WebLogic Server domains containing Oracle Fusion Applications

• The Administration Server for the Oracle Fusion Applications domains

• Managed Servers

• The Middleware home

• An Oracle home

• An Oracle instance home

• Any database used by Oracle Fusion Applications. You use Oracle Recovery Manager (RMAN) to recover an Oracle Database instance. See Oracle Database Backup and Recovery User's Guide for information about using RMAN to recover a database.

Note that with Oracle Fusion Applications, you can install all or some of the files on shared storage. Some of the procedures will differ depending on whether all files are on shared storage. The options are:

• All files are on a shared file system.

• All binary files are on a shared file system as well as all Administration Servers. Managed Servers are on local file systems.

18.3.3 Recommended Recovery Strategies

Note the following key points about recovery:

• All or part of your Oracle Fusion Applications environment must be offline while you are performing recovery.

  Stop the relevant processes. The processes you stop depends on the granularity of the recovery. For example, if you are recovering only one domain, shut down the corresponding Administration Server and Managed Servers.

• Rename existing files and directories before you begin restoring the files from backup so that you do not unintentionally override necessary files.

• Although, in some cases, it may appear that only one or two files are lost or corrupted, you should restore the directory structure for the entire element, such as an Oracle instance home or a component, rather than just restoring one or two files. In this way, you are more likely to guarantee a successful recovery.

• If you need to recover a database, perform a complete recovery to recover the database to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database as close to the current time as possible. (You can use point-in-time recovery if the database is configured in Archive Log Mode. This is typically a time right before the database failure occurred.) Oracle recommends using Archive Log Mode for production databases.

  You must keep the databases synchronized. See Section 18.9.9 for procedures for recovering them to the same point in time and reconciling differences.

18.4 Prerequisites for Using Cloud Control to Back Up or Restore Your Environment

Before you back up or restore an Oracle Fusion Applications environment using Oracle Enterprise Manager Cloud Control, ensure the following prerequisites are met:

• Cloud Control 12c Agents must be installed on all hosts in the Oracle Fusion Applications environment where components (such as Oracle WebLogic Server Administration and Managed Servers) are running, and the hosts must be discovered in Cloud Control. (Although it is possible to monitor an Oracle Fusion Applications environment without having Agents on every host, backup and restore requires a local Agent on each host). For more information on installing and configuring Cloud Control, refer to the Oracle Enterprise Manager Cloud Control Basic Installation Guide.

• An Oracle Secure Backup domain target must be discovered in Cloud Control. This requires an Oracle Secure Backup server installation, and all hosts that are part of the Oracle Fusion Applications environment must be configured as Oracle Secure Backup client hosts. For more information on installing Oracle Secure Backup, refer to the Oracle Secure Backup Installation and Configuration Guide.

• A Fusion Instance target representing the Oracle Fusion Applications environment must be discovered in Cloud Control. The Fusion Instance target must be discovered through a local Agent that is running on one of the hosts in the Oracle Fusion Applications environment.

• You must create a backup configuration using Cloud Control. A backup configuration contains the settings that will be used for the database and file backups, and is required to perform an Oracle Fusion Applications backup. See Section 18.6.1.1 for more details on creating a backup configuration.

• Before you perform a backup or restore through Cloud Control, you must properly shut down all Oracle WebLogic Server instances and other components in the Oracle Fusion Applications environment. Refer to Section 18.9 and Section 18.10 for details on what components must be shut down and how to shut them down. Optionally, scripts that perform the required shutdown of components on each host can be invoked as user-specified pre-backup or pre-restore scripts within the Cloud Control backup procedure. (See Section 18.6.1.2 for more details on user-specified scripts.)

• If the Oracle Fusion Applications repository or other databases are included in an Oracle Fusion Applications restore, the databases must be started in mounted mode prior to submitting the restore procedure. If a complete restore of the databases is required, it may be necessary to first perform a SPFILE or controlfile restore or both for the individual databases through the database Perform Recovery wizard, which can be accessed from the Availability menu on the Cloud Control database home page. (Go to the home page of the database that must be restored, select Availability, then select Backup and Recovery, then select Perform Recovery.)

For information about using Cloud Control to back up and recover your Oracle Fusion Applications environment, see:

• Section 18.6.1, "Performing a Backup Using Cloud Control"

• Section 18.8, "Recovering Using Cloud Control"

18.5 Backup and Recovery Recommendations for Oracle Fusion Applications

The topics in this section describe backup and recovery recommendations for specific Oracle Fusion Applications components and for components that are related to Oracle Fusion Applications.

The topics include information about configuration files for each application or component. Note that the list of files in not an exhaustive list. You do not back up or recover the individual files. Generally, you back up or recover the Applications base directory, the Applications configuration directory, Middleware home, domain, Oracle home, or an Oracle instance, as described in Section 18.6, Section 18.9, and Section 18.10.

The configuration files and database schemas are for information purposes only. You must back up the entire database.

This section includes the following topics:

• Section 18.5.1, "Backup and Recovery Recommendations for Oracle Fusion Customer Relationship Management"

• Section 18.5.2, "Backup and Recovery Recommendations for Oracle Fusion Financials"

• Section 18.5.3, "Backup and Recovery Recommendations for Oracle Fusion Human Capital Management"

• Section 18.5.4, "Backup and Recovery Recommendations for Oracle Fusion Supply Chain Management"

• Section 18.5.5, "Backup and Recovery Recommendations for Oracle Fusion Project"

• Section 18.5.6, "Backup and Recovery Recommendations for Oracle Fusion Procurement"

• Section 18.5.7, "Backup and Recovery Recommendations for Oracle Fusion Incentive Compensation"

• Section 18.5.8, "Backup and Recovery Recommendations for Oracle Fusion Applications Technology"

• Section 18.5.9, "Backup and Recovery Recommendations for Oracle Fusion Setup"

• Section 18.5.10, "Backup and Recovery Recommendations for Oracle Enterprise Scheduler"

• Section 18.5.11, "Backup and Recovery Recommendations for Oracle Enterprise Crawl and Search Framework"

• Section 18.5.12, "Backup and Recovery Recommendations for Oracle Authorization Policy Manager"

• Section 18.5.13, "Backup and Recovery Recommendations for Java Servers for Oracle E-Mail and Web Marketing"

18.5.1 Backup and Recovery Recommendations for Oracle Fusion Customer Relationship Management

This section describes the Oracle Fusion Customer Relationship Management data that must be backed up and restored.

Configuration Files

Most configuration data is stored in the Oracle WebLogic Server domain.

In addition, Oracle Fusion Customer Relationship Management uses configuration data for Java servers for Oracle E-Mail and Web Marketing. See Section 18.5.13 for information about backing up and recovering Oracle E-Mail and Web Marketing.

Dependencies on Oracle Fusion Middleware Components

Oracle Metadata Services, Oracle Business Intelligence metadata, Oracle Real-Time Decisions, an LDAP provider such as Oracle Internet Directory, Oracle WebCenter Portal (including tagging, group spaces, and the Oracle WebCenter Portal schemas), portlet metadata, Oracle SOA Suite, Oracle ADF user interface, Oracle Essbase, Oracle WebCenter Content, Oracle Enterprise Crawl and Search Framework metadata and data, including Oracle Secure Enterprise Search, Oracle Product Data Quality

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases containing Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content

Backup Recommendations

Back up the Applications base directory and the Administration Server domain in which Oracle Fusion Customer Relationship Management is deployed. Back up the standalone Java servers used for Oracle E-Mail and Web Marketing, as described in Section 18.5.13.

Back up the databases containing the Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content.

Back up Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recovery Recommendations

Recover the Managed Server to which the application is deployed. If needed, recover the standalone Java servers used for Oracle E-Mail and Web Marketing, as described in Section 18.5.13.

Depending upon the extent of failure, recovery should be performed at the desired granularity. For the steps to recover Oracle Fusion Customer Relationship Management, including for loss of host, see Section 18.9.10.1.

Recover Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recover the databases containing the Oracle Fusion Applications, Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content.

If needed, perform a complete recovery to recover the databases to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.2 Backup and Recovery Recommendations for Oracle Fusion Financials

This section describes the Oracle Fusion Financials data that must be backed up and restored.

Configuration Files

Configuration data is stored in the Oracle WebLogic Server domain.

Dependencies on Oracle Fusion Middleware Components

Oracle Application Development Framework, Oracle BPEL Process Manager, Oracle WebCenter Portal (portlets), Oracle Metadata Services, Oracle WebCenter Content: Imaging, Oracle Enterprise Scheduler, Oracle BI Enterprise Edition, Oracle Essbase, and Applications Technology Group

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases containing the Oracle Fusion Applications, the LDAP store, Oracle Essbase, and Oracle Fusion Middleware schemas

Backup Recommendations

Back up the Applications base directory and the Administration Server domain in which Oracle Fusion Financials is deployed.

Back up Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Back up the databases containing the Oracle Fusion Applications, Oracle Essbase, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content.

Recovery Recommendations

Recover the Managed Server to which the application is deployed. For information about recovering Oracle Essbase, see "Recovering Oracle Essbase After Loss of Host" in the Oracle Fusion Middleware Administrator's Guide.

Depending upon the extent of failure, recovery should be performed at the desired granularity. See Section 18.9.

Recover Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recover the databases to the most recent point in time, if needed.

If needed, perform a complete recovery to recover the databases to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.3 Backup and Recovery Recommendations for Oracle Fusion Human Capital Management

This section describes the Oracle Fusion Human Capital Management data that must be backed up and restored.

Configuration Files

Most configuration data is stored in the Oracle WebLogic Server domain.

Dependencies on Oracle Fusion Middleware Components

Oracle Application Development Framework, Oracle BPEL Process Manager, Oracle WebCenter Portal (portlets), Oracle Metadata Services, Oracle Enterprise Scheduler, Oracle Business Intelligence Publisher, an LDAP provider such as Oracle Internet Directory, Oracle Fusion Middleware Extensions for Applications, Oracle WebCenter Content

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases containing Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content

Backup Recommendations

Back up the Applications base directory and the Administration Server domain in which Oracle Fusion Human Capital Management is deployed.

Back up Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Back up the databases containing the LDAP store, Oracle Fusion Applications, Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content.

Recovery Recommendations

Recover the Managed Server to which the application is deployed.

Depending upon the extent of failure, recovery should be performed at the desired granularity. See Section 18.9.

Recover Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recover the databases to the most recent point in time, if needed. Note that the databases, including the LDAP store, the database containing Oracle Fusion Human Capital Management schemas and the Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content must be kept synchronized. If you restore one, restore the others to the same point in time.

There may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.4 Backup and Recovery Recommendations for Oracle Fusion Supply Chain Management

This section describes the Oracle Fusion Supply Chain Management data that must be backed up and restored.

Configuration Files

Configuration data is stored in the Oracle WebLogic Server domain.

Dependencies on Oracle Fusion Middleware Components

Oracle Application Development Framework, Oracle SOA Suite, Oracle Enterprise Scheduler

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases containing Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content

Backup Recommendations

Back up the Applications base directory and the Administration Server domain in which Oracle Fusion Supply Chain Management is deployed.

Back up Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Back up the databases containing the Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content.

Recovery Recommendations

Recover the Managed Server to which the application is deployed.

Recover Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recover the databases to the most recent point in time, if needed.

Depending upon the extent of failure, recovery should be performed at the desired granularity. See Section 18.9. For the steps specific to recovering from loss of host, see Section 18.10.6.1.

18.5.5 Backup and Recovery Recommendations for Oracle Fusion Project

This section describes the Oracle Fusion Project data that must be backed up and restored.

Configuration Files

Most configuration data is stored in the Oracle WebLogic Server domain. In addition, configuration data is stored in the Oracle Essbase database.

Dependencies on Oracle Fusion Middleware Components

Oracle Metadata Services, an LDAP provider such as Oracle Internet Directory, Oracle WebCenter Portal (including tagging, group spaces, and the Oracle WebCenter Portal schemas), Oracle SOA Suite, Oracle Application Development Framework user interface, Oracle Enterprise Crawl and Search Framework metadata and data, Oracle Enterprise Scheduler, Oracle Essbase

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases containing the Oracle Fusion Applications, Oracle Essbase, the LDAP store, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content

Backup Recommendations

Back up the Applications base directory and the Administration Server domain in which Oracle Fusion Project is deployed.

Back up Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Back up the Oracle Essbase database used by Oracle Fusion Project. Back up the databases containing the Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas.

Recovery Recommendations

Recover the Managed Server to which the application is deployed.

Recover Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recover the databases to the most recent point in time, if needed.

Depending upon the extent of failure, recovery should be performed at the desired granularity. See Section 18.9.

18.5.6 Backup and Recovery Recommendations for Oracle Fusion Procurement

This section describes the Oracle Fusion Procurement data that must be backed up and restored.

Configuration Files

Most configuration data is stored in the Oracle WebLogic Server domain. In addition, the certificate file, which is determined by the Profile Option, contains a certificate for SSL connections made to supplier web sites in order to let requisitioners search or browse catalogs.

Dependencies on Oracle Fusion Middleware Components

Oracle Application Development Framework, Oracle BPEL Process Manager, Oracle B2B, Oracle Fusion Middleware Extensions for Applications, Oracle Enterprise Scheduler, Oracle Business Intelligence Publisher, Oracle Streams, Advanced Queuing (AQ), Java Message Service (JMS), JMS queues with Oracle B2B

Dependencies on Third-Party Products

CUPS software, a portable printing layer for UNIX systems

Database Repository Dependencies

The databases containing Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content

Backup Recommendations

Back up the Applications base directory and the Administration Server domain in which Oracle Fusion Procurement is deployed. Back up the certificate file.

Back up Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Back up the databases containing the Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content.

Recovery Recommendations

Recover the Managed Server to which the application is deployed.

Depending upon the extent of failure, recovery should be performed at the desired granularity. For the steps to recover Oracle Fusion Procurement, see Section 18.9.10.2. For the steps specific to recovering from loss of host, see Section 18.10.6.2.

For information about recovering JMS, see "Backup and Recovery Recommendations for Oracle WebLogic Server JMS" in the Oracle Fusion Middleware Administrator's Guide.

Recover Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recover the databases to the most recent point in time, if needed.

If needed, perform a complete recovery to recover the databases to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.7 Backup and Recovery Recommendations for Oracle Fusion Incentive Compensation

This section describes the Oracle Fusion Incentive Compensation data that must be backed up and restored.

Configuration Files

Configuration data is stored in the Oracle WebLogic Server domain.

Dependencies on Oracle Fusion Middleware Components

Oracle Application Development Framework, Oracle SOA Suite, Oracle Data Integrator, Oracle BI Enterprise Edition, Oracle Enterprise Scheduler, Oracle Enterprise Crawl and Search Framework, Oracle Secure Enterprise Search

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases containing Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content

Backup Recommendations

Back up the Applications base directory and the Administration Server domain in which Oracle Fusion Incentive Compensation is deployed.

Back up Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Back up the databases containing the Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content.

Recovery Recommendations

Recover the Managed Server to which the application is deployed.

Depending upon the extent of failure, recovery should be performed at the desired granularity. See Section 18.9.

Recover Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recover the databases to the most recent point in time, if needed.

If needed, perform a complete recovery to recover the databases to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.8 Backup and Recovery Recommendations for Oracle Fusion Applications Technology

This section describes the Oracle Fusion Applications Technology data that must be backed up and restored.

Configuration Files

Configuration data is stored in the Oracle WebLogic Server domain.

Dependencies on Oracle Fusion Middleware Components

Oracle Application Development Framework, Oracle SOA Suite, Federated Worklist, Oracle WebCenter Portal (including portlets, tagging, group spaces, and forums), Oracle WebCenter Portal, MDS, portlet metadata, Oracle Enterprise Crawl and Search Framework metadata and data, and Oracle Secure Enterprise Search, Oracle BI Enterprise Edition, Oracle Fusion Topology Manager, Applications Technology Group

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases containing Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content

Backup Recommendations

Back up the Applications base directory and the Administration Server domain in which Oracle Fusion Applications Technology is deployed.

Back up Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Back up the databases containing the Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content.

Recovery Recommendations

Recover the Managed Server to which the application is deployed.

Depending upon the extent of failure, recovery should be performed at the desired granularity. See Section 18.9.

Recover Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recover the databases to the most recent point in time, if needed.

If needed, perform a complete recovery to recover the databases to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.9 Backup and Recovery Recommendations for Oracle Fusion Setup

This section describes the Oracle Fusion Setup data that must be backed up and restored.

Configuration Files

Configuration data is stored in the Oracle WebLogic Server domain.

Dependencies on Oracle Fusion Middleware Components

Oracle Application Development Framework, Oracle BPEL Process Manager, Oracle WebCenter Content, Oracle WebCenter Portal (portlets), Oracle Enterprise Scheduler, Identity Governance Framework

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases containing Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content

Backup Recommendations

Back up the Applications base directory and the Administration Server domain in which Oracle Fusion Setup is deployed.

Back up Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Back up the databases containing the Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content.

Recovery Recommendations

Recover the Managed Server to which the application is deployed.

Depending upon the extent of failure, recovery should be performed at the desired granularity. See Section 18.9.

Recover Oracle WebCenter Content, as described in "Backup and Recovery Recommendations for Oracle WebCenter Content" in the Oracle Fusion Middleware Administrator's Guide.

Recover the databases containing the Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas, including those for Oracle WebCenter Content.

If needed, perform a complete recovery to recover the databases to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.10 Backup and Recovery Recommendations for Oracle Enterprise Scheduler

This section describes the Oracle Enterprise Scheduler data that must be backed up and restored.

Configuration Files

Configuration data is stored in the Oracle WebLogic Server domain.

Dependencies on Oracle Fusion Middleware Components

None

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases containing Oracle Fusion Applications, Oracle Enterprise Scheduler, LDAP store, and Oracle Fusion Middleware schemas, including Oracle WebCenter Content

Backup Recommendations

Back up the Oracle home and the domain home.

Back up the databases containing the Oracle Fusion Applications, the LDAP store, and Oracle Fusion Middleware schemas.

Recovery Recommendations

Recover the domain home and the Oracle home, as needed.

Depending upon the extent of failure, recovery should be performed at the desired granularity. For the steps to recover Oracle Enterprise Scheduler, including for loss of host, see Section 18.9.11.2.

Recover the databases to the most recent point in time, if needed.

If needed, perform a complete recovery to recover the databases to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.11 Backup and Recovery Recommendations for Oracle Enterprise Crawl and Search Framework

This section describes the Oracle Enterprise Crawl and Search Framework data that must be backed up and restored.

Configuration Files

Configuration data is stored in the Oracle WebLogic Server domain.

Dependencies on Oracle Fusion Middleware Components

Oracle Application Development Framework

Dependencies on Third-Party Products

None

Database Repository Dependencies

The database containing Oracle Fusion Applications schemas

Backup Recommendations

Back up the domain in which Oracle Enterprise Crawl and Search Framework is deployed.

Back up the database containing the Oracle Fusion Applications schemas.

Recovery Recommendations

Recover the domain in which Oracle Enterprise Crawl and Search Framework is deployed.

Depending upon the extent of failure, recovery should be performed at the desired granularity. For the steps to recover Oracle Enterprise Crawl and Search Framework, including for loss of host, see Section 18.9.11.3.

Recover the database containing the Oracle Fusion Applications schemas.

If needed, perform a complete recovery to recover the databases to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.12 Backup and Recovery Recommendations for Oracle Authorization Policy Manager

This section describes the Oracle Authorization Policy Manager data that must be backed up and restored.

Configuration Files

The configuration files are located in the Oracle instance home.

Dependencies on Oracle Fusion Middleware Components

An LDAP provider, such as Oracle Internet Directory

Dependencies on Third-Party Products

None

Database Repository Dependencies

The databases used by Oracle Authorization Policy Manager and the LDAP store

Backup Recommendations

Back up the Oracle Authorization Policy Manager Domain home and its Oracle home.

Back up the database used by Oracle Authorization Policy Manager and the LDAP store.

Recovery Recommendations

Recover the domain in which Oracle Authorization Policy Manager is deployed. Recover the Oracle home, if necessary.

Depending upon the extent of failure, recovery should be performed at the desired granularity. For the steps to recover Oracle Authorization Policy Manager, including for loss of host, see Section 18.9.11.4.

Recover the databases used by Oracle Authorization Policy Manager and the LDAP store, if needed, to the same point in time.

If needed, perform a complete recovery to recover the databases to the most current state. However, there may be some situations where you do not have all the required logs to accomplish complete recovery. In that situation, perform an incomplete recovery (point-in-time recovery) to recover the database to as close to the current time as possible. Then, reconcile the databases, as described in Section 18.9.9.

18.5.13 Backup and Recovery Recommendations for Java Servers for Oracle E-Mail and Web Marketing

Oracle E-Mail and Web Marketing is provided with Oracle Fusion Customer Relationship Management. Oracle E-Mail and Web Marketing provides three components: Email Sending Daemon (ESD), Click -Through Daemon (CTD), and bounce handling daemon (BHD).

This section describes the Oracle E-Mail and Web Marketing data that must be backed up and restored.

Configuration Files

Configuration data is stored in the Oracle WebLogic Server domain.

Dependencies on Oracle Fusion Middleware Components

None

Dependencies on Third-Party Products

None

Database Repository Dependencies

None

Backup Recommendations

Back up the Managed Servers to which ESD and CTD are deployed. Back up the directory for BHD.

Recovery Recommendations

Restore the Managed Servers to which ESD and CTD are deployed. Restore the directory for BHD.

Depending upon the extent of failure, recovery should be performed at the desired granularity. For the steps to recover Oracle E-Mail and Web Marketing, including for loss of host, see Section 18.9.11.5.

18.6 Performing a Backup

You can perform a full offline backup or an online or offline backup of configuration files.

This section includes the following topics:

• Section 18.6.1, "Performing a Backup Using Cloud Control"

• Section 18.6.2, "Performing a Full Offline Backup Using the Command Line"

• Section 18.6.3, "Performing an Online Backup of Configuration Files Using the Command Line"

18.6.1 Performing a Backup Using Cloud Control

You can use Cloud Control to back up Oracle Fusion Applications, as described in the following topics:

• Section 18.6.1.1, "Configuring Cloud Control Backups"

• Section 18.6.1.2, "Backing Up Oracle Fusion Applications Using Cloud Control"

18.6.1.1 Configuring Cloud Control Backups

You must create a backup configuration before you perform a backup or restore using Cloud Control. A backup configuration contains settings for database and file backups. You create the configuration using the Backup Configurations page. You can create multiple backup configurations, using different settings. Then, you can use the configurations in subsequent backup and restore operations.

To create a new backup configuration using Cloud Control:

1. From the Targets menu, select Fusion Applications.

   The Fusion Applications target home page is displayed.

2. Click the Fusion Instance that you want to back up.

   The Fusion Instance home page is displayed.

3. From the Fusion Instance menu, choose Control, then Backup Configurations.

   The Backup Configurations page is displayed, as shown in the following figure:

   
   Description of the illustration backup_config.gif
   
   

4. Click Create.

   The Create Backup Configurations page is displayed. It contains three tabs: Storage, Policy, and Recovery Catalog. You can use the default settings or customize individual settings. For more information on the database backup settings in these pages, refer to the Oracle Database Backup and Recovery Reference. (For information on the Oracle Secure Backup settings in these pages, refer to the obtool backup command options in the Oracle Secure Backup Reference.)

5. On the Storage tab, you specify settings related to disk and tape backup. Because Cloud Control always backs up Oracle Fusion Applications to tape, the disk settings are not applicable.

   To specify tape settings, first select an Oracle Secure Backup Domain, after which separate tape settings for database and files are shown. Database settings include datafile and archived log copies, backup type, and media management parameters. File settings include media family and devices. Specify Oracle Secure Backup domain and host credentials.

   The following figure shows the Storage tab:

   
   Description of the illustration backup_config_store.gif
   
   

6. On the Policy tab, you specify policy settings for database and tape backups. Database settings include backup retention, compression, and encryption. Separate database encryption settings are specified for disk and tape backups. File settings include Oracle Secure Backup job and encryption settings.

   The following figure shows the Policy tab:

   
   Description of the illustration backup_config_pol.gif
   
   

7. If you are using a recovery catalog for database backups, specify the catalog on the Recovery Catalog tab, which is shown in the following figure:

   
   Description of the illustration backup_config_rec.gif
   
   

8. Click Save.

18.6.1.2 Backing Up Oracle Fusion Applications Using Cloud Control

By default, Cloud Control backs up the core directories and databases that comprise a Oracle Fusion Applications installation. You can include additional files and databases in the backup. It also allows you to execute user-defined pre-backup and post-backup scripts.

Because Cloud Control uses Oracle Secure Backup to perform the backups, all backups are done to tape, which is the media supported by Oracle Secure Backup.

To back up a Oracle Fusion Applications environment using Cloud Control:

1. Ensure that you have met the prerequisites, as described in Section 18.4.

2. From the Targets menu, choose Fusion Applications.

   The Fusion Applications target home page is displayed.

3. Click the Fusion Instance that you want to back up.

   The Fusion Instance home page is displayed.

4. From the Fusion Instance menu, choose Control, then Schedule Backup.

   The Schedule Backup wizard is displayed, with the first step, Backup Scope, as shown in the following figure:

   
   Description of the illustration backup_scope.gif
   
   

5. In the Backup Scope step:

   1. Select the Oracle Secure Backup domain to use for the backup.

   2. Select the scope of the backup. You can select:

      • Software and Configuration Directories: Backs up all directories in the Oracle Fusion Applications installation, including the Applications Home directory (the root of the Oracle Fusion Applications installation) and any other directories (for example, Domain homes) that may be installed in a different location. Cloud Control will determine the distinct file systems that comprise the installation across all hosts in the environment, and will include the top-level directories needed to constitute a complete backup.

      • Configuration Directories: Backs up only the configuration-related directories, including the Domain home of the Administrative Server for each Oracle WebLogic Server domain. You can select the following options:

        All: Back up all configuration directories.

        Custom: Narrow the scope of the backup by selecting specific Oracle Fusion Applications components. (On the next page, you can further select which of their associated directories to include in the backup.)

   Click Next.

6. In the Directories step, specify the Oracle Fusion Applications directories to include in the backup. By default, the directories that correspond to the backup scope choice (made on the Backup Scope page) are included. If you chose the Custom option, you can remove directories from the list. (The list of directories that correspond to the backup scope are automatically determined by Cloud Control. It is not possible to individually select or remove subdirectories from the backup.) Click Add to specify additional files or directories to include in the backup from outside the core Oracle Fusion Applications installation. Any directories or files that reside on a discovered Oracle Secure Backup client host can be included.

   Click Next.

7. In the Databases step, click Add to specify additional databases to add to the backup. By default, the Oracle Fusion Applications repository database is included in the backup.

   Click Next.

8. In the Configuration step, select the backup configuration to use for the backup. (See the Section 18.6.1.1 for details on creating a backup configuration.)

   Click Next.

9. In the Options step:

   1. Select the backup type:

      • Full: A full backup of the databases and files will be performed. If the database portion of the backup is to be used as the base of an incremental backup strategy, select the Use as the base of an incremental backup strategy option.

      • Incremental: An incremental backup of the databases and files will be performed.

      See the Oracle Database Backup and Recovery User's Guide for a description of the backup types.

   2. Select the backup mode:

      • Online: The databases will remain running during the backup. You must manually quiesce or shut down the Oracle Fusion Applications processes prior to submitting the backup, as this is not done automatically by Cloud Control.

      • Offline: The databases will be shutdown automatically prior to the backup. You must manually shut down the Oracle Fusion Applications processes prior to submitting the backup, as this is not done automatically by Cloud Control.

      See Section 4.4 for information about stopping Oracle Fusion Applications processes. Alternatively, you can specify scripts to shutdown the processes in Step 10.

   Click Next.

10. In the Scripts step, specify the host name and path of scripts to be run before and after the backup. Post-backup scripts can be specified to run upon success or failure of the backup. The scripts can be run on any discovered host. The scripts must be executable files that exist on the specified hosts. Any type of executable file can be specified.

    Click Next.

11. In the Credentials step, if one or more databases is included in the backup, specify the database and host credentials that will be used to back up the databases.

    Click Next.

12. In the Schedule step, specify the procedure name, description, and scheduling information.

    Click Next.

13. In the Review step, verify all the input specified in the wizard. Make changes as needed and click Submit. A Cloud Control deployment procedure is submitted to perform the backup, and a confirmation page appears with a link to the deployment procedure status page.

18.6.2 Performing a Full Offline Backup Using the Command Line

To perform a full offline backup, you copy the file system artifacts and database repositories corresponding to Oracle Fusion Applications. You use your preferred tool for archiving and compressing, as described in Section 18.2. Ensure that the tool you are using preserves the permissions of the files.

To perform a full offline backup:

1. Stop all processes. See Section 4.4.3.2.

2. Back up the Applications base directory on all hosts. For example:

   (UNIX) tar -cpf ApplBase_backup_030212.tar APPLICATIONS_BASE/*
   (Windows) jar  cf ApplBase_backup_030212.tar APPLICATIONS_BASE\*
   

3. Back up the Applications configuration directory on all hosts. For example:

   (UNIX) tar -cpf ApplConfig_backup_030212.tar APPLICATIONS_CONFIG/*
   (Windows) jar  cf ApplConfig_backup_030212.tar APPLICATIONS_CONFIG\*
   

4. If a domain is not located within the Applications configuration home, back up the domains separately. This backs up the Managed Servers that are running Java components such as Oracle SOA Suite and Oracle WebCenter Portal.

   For example:

   (UNIX) tar -cpf domain_home_backup_030212.tar APPLICATIONS_CONFIG/instance/domains/domain_name/*
   (Windows) jar  cf domain_home_backup_030212.jar APPLICATIONS_CONFIG\instance\domains\domain_name\* 
   

   In most cases, you do not need to back up the Managed Server directories separately, because the Administration Server domain contains information about the Managed Servers in its domain. See Section 18.2.4 for information about what you need to back up.

5. If the Oracle instance home is not located within the Applications base home, back up the Oracle instance home. The Oracle instance home contains configuration information about system components, such as the Global Order Promising component of Oracle Fusion Supply Chain Management.

   For example:

   (UNIX) tar -cpf instance_home_backup_030212.tar ORACLE_INSTANCE/*
   (Windows) jar  cf instance_home_backup_030212.jar ORACLE_INSTANCE\* 
   

6. If a Managed Server is not located within the domain, back up the Managed Server directory. For example:

   (UNIX) tar -cpf man_server1_backup_030212.tar APPLICATIONS_CONFIG/instance/domains/domain_name/servers/server_name/*
   (Windows) jar  cf man_server1_backup_030212.jar APPLICATIONS_CONFIG\instance\domains\domain_name\servers\server_name\* 
   

7. Back up the OraInventory directory. For example:

   (UNIX) tar -cpf Inven_home_backup_030212 /scratch/oracle/OraInventory
   (Windows) jar cf Inven_home_backup_030212.jar C:\Program Files\Oracle\Inventory
   

8. On UNIX, back up the OraInst.loc file, which is located in the following directory:

   (Linux and IBM AIX) /etc
   (Other UNIX systems) /var/opt/oracle
   

9. On UNIX, back up the oratab file, which is located in the following directory:

   /etc
   

   Note that the oratab file is located on the database host.

10. Back up the databases using the Oracle Recovery Manager (RMAN). For detailed steps, see the Oracle Database Backup and Recovery User's Guide.

11. On Windows, export the following registry key:

    HKEY_LOCAL_MACHINE\Software\oracle
    

    In addition, for system components, export the following Windows Registry key:

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services
    

    To export a key, use the following command:

    regedit /E  filename key
    

    For example:

    regedit /E C:\oracleregistry.reg HKEY_LOCAL_MACHINE/oracle 
    

    You can also use the Registry Editor to export the key. See the Registry Editor Help for more information.

12. Create a record of your Oracle Fusion Applications environment. See Section 18.7.

18.6.3 Performing an Online Backup of Configuration Files Using the Command Line

You should perform a backup of configuration files on a regular basis and at the times described in Section 18.2.4.

To back up configuration files:

1. To avoid an inconsistent backup, do not make any configuration changes until the backup is completed. To ensure that no changes are made in the WebLogic Server domain, lock the WebLogic Server configuration, as described in "Locking the WebLogic Server Configuration" in the Oracle Fusion Middleware Administrator's Guide.

2. Back up the domain directories. This backs up the Managed Servers that are running Java components such as Oracle SOA Suite and Oracle WebCenter Portal. For example:

   (UNIX) tar -cpf domain_home_backup_030212.tar APPLICATIONS_CONFIG/instance/domains/domain_name/*
   (Windows) jar  cf domain_home_backup_030212.jar APPLICATIONS_CONFIG\instance\domains\domain_name\* 
   

3. Back up the Oracle instance home. This backs up the system components, such as Oracle HTTP Server. For example:

   (UNIX) tar -cpf instance_home_backup_030212.tar ORACLE_INSTANCE/*
   (Windows) jar  cf instance_home_backup_030212.jar ORACLE_INSTANCE\* 
   

4. Back up the databases using the Oracle Recovery Manager (RMAN). For detailed steps, see the Oracle Database Backup and Recovery User's Guide.

18.7 Creating a Record of Your Oracle Fusion Applications Configuration

In the event that you need to restore and recover your Oracle Fusion Applications environment, it is important to have all the necessary information at your disposal. This is especially true in the event of a hardware loss that requires you to reconstruct all or part of your Oracle Fusion Applications environment on a new disk or host.

You should maintain an up-to-date record of your Oracle Fusion Applications environment that includes the information listed in this section. You should keep this information both in hardcopy and electronic form. The electronic form should be stored on a host or email system that is completely separate from your Oracle Fusion Applications environment.

Your Oracle Fusion Applications hardware and software configuration record should include:

• The following information for each host in your environment:

  • Host name

  • Virtual host name (if any)

  • Domain name

  • IP address

  • Hardware platform

  • Operating system release level and patch information

• The following information for each Oracle Fusion Applications installation in your environment:

  • Installation type (for example, Oracle SOA Suite, Oracle Fusion Supply Chain Management)

  • Host on which the installation resides

  • User name, user ID number, group name, group ID number, environment profile, and type of shell for the operating system user that owns the Oracle home (/etc/passwd and /etc/group entries)

  • Directory structure, mount points, and full path for the Applications base directory, Applications configuration directory, Middleware home, Oracle Common home, Oracle homes, Oracle WebLogic Server domain homes, and the Oracle instance homes

  • Amount of disk space used by the installation

  • Port numbers used by the installation

• The following information for the databases containing the metadata for components and any other databases used by Oracle Fusion Applications:

  • Host name

  • Database version and patch level

  • Base language

  • Character set

  • Global database name

  • SID

• The following information about backups:

  • The time of the backup.

  • The contents of the backup. For example, a full backup, a backup of a domain.

  • The tool used to create the backup.

  • Where the backup is stored.

18.8 Recovering Using Cloud Control

You can recover an Oracle Fusion Applications environment with Cloud Control. Cloud Control restores the directories and databases of a Oracle Fusion Applications backup. All directories and databases of the backup are restored by default. Optionally, you can select individual directories and databases to be restored.

The restore capability provided by Cloud Control is one part of a larger process that must be performed to recover a Oracle Fusion Applications environment or individual Oracle Fusion Applications components. After you restore Oracle Fusion Applications directories and databases using Cloud Control, you must perform additional manual steps depending on the component that your are restoring. Those steps are described in Section 18.9 and Section 18.10.

To recover Oracle Fusion Applications with Cloud Control:

1. Ensure that you have met the prerequisites, as described in Section 18.4.

2. From the Targets menu, choose Fusion Applications.

   The Fusion Applications target home page is displayed.

3. Click the Fusion Instance that you want to recover.

   The Fusion Instance home page is displayed.

4. From the Fusion Instance menu, choose Control, then Perform Restore.

   The Perform Restore wizard is displayed, as shown in the following figure:

   
   Description of the illustration recover_sel.gif
   
   

5. In the Select Backup step, select the backup to restore from the list of available backups for the Fusion Instance target.

   Click Next.

6. In the Components step, the Oracle Fusion Applications components and databases included in the selected backup are listed. Select one or more components to include in the restore. The directories that correspond to the selected components will be shown on the next page. If multiple components are selected that have common directories, those common directories will be included along with any directories that are specific to individual components.

   Click Next.

7. In the Directories step, the directories that correspond to the selected Oracle Fusion Applications components are listed in the first table. By default, they will all be restored to their original locations. You can remove directories that should not be included in the restore from the table. You can change the destination host name and path where the directories will be restored. (Note that Oracle Fusion Applications directories must be restored to their original paths to successfully recover the environment. However, restoring to different paths is supported to provide the option of restoring to a temporary staging location in preparation for eventual movement to the final location.)

   While you can remove complete directories from the list, you cannot select or remove individual subdirectories or files under the top-level directories shown in the list. If a more fine-grained restore granularity is desired, individual files and subdirectories can be restored by using the Perform Restore wizard available from the Oracle Secure Backup Domain home page. (From that home page, select Manage, then File System Backup/Restore, and then Perform Restore.)

   In addition, any additional files or directories included in the backup are listed in a separate table, and will be restored to their original locations by default. Directories and files can be removed from the table and you can change the destination host name and path.

   Click Next.

8. In the Configuration step, the original backup configuration used to create the backup is displayed. If the backup configuration is still available it will be used by default and no user action is required on this page. If the configuration is not available, select an alternate backup configuration that has settings similar to backup configuration used to create the backup.

   Click Next.

9. In the Scripts step, specify pre-restore and post-restore scripts that will be executed as part of the restore operation. Scripts can be run on any discovered host. The scripts must be executable files that exist on the specified hosts.

   The Oracle Fusion Applications components being restored must be shut down prior to being restored. Because this is not done automatically by Cloud Control, the appropriate shutdown scripts can be executed as part of the restore procedure by specifying them as pre-restore scripts.

   Click Next.

10. In the Credentials step, if one or more databases is included in the restore operation, specify both database and host credentials that will be used to restore the databases.

    Click Next.

11. In the Schedule step, specify the procedure name, description, and scheduling information.

    Click Next.

12. In the Review step, verify all the input specified in the wizard. Make changes as needed and click Submit.

18.9 Recovering After Data Loss, Corruption, or Media Failure

This section describes recovery strategies for outages that involve actual data loss or corruption, or media failure where the disk cannot be restored. It also describes recovery strategies for applications that are no longer functioning properly. This type of failure requires some type of data restoration before the Oracle Fusion Applications environment can be restarted and continue with normal processing.

Depending on the extent of the failure, you can recover the Applications base directory, the Middleware homes, the Administration Server, a Managed Server, or the database. Some Oracle Fusion Applications components require additional steps, which are described in subsequent sections.

This section includes the following topics:

• Section 18.9.1, "Recovering the Applications Base Directory"

• Section 18.9.2, "Recovering a Middleware Home"

• Section 18.9.3, "Recovering an Oracle WebLogic Server Domain"

• Section 18.9.4, "Recovering an Oracle Home"

• Section 18.9.5, "Recovering an Oracle Instance Home"

• Section 18.9.6, "Recovering the Administration Server Configuration"

• Section 18.9.7, "Recovering a Managed Server"

• Section 18.9.8, "Recovering the Databases"

• Section 18.9.9, "Reconciling the Data"

This section also includes the following topics which describe additional considerations you may need to take for particular components:

• Section 18.9.10, "Recovering Oracle Fusion Applications"

• Section 18.9.11, "Recovering Components Related to Oracle Fusion Applications"

See the "Recovering Your Environment" section in the Oracle Fusion Middleware Administrator's Guide for information about recovering Oracle Fusion Middleware components, such as Oracle SOA Suite.

18.9.1 Recovering the Applications Base Directory

To recover an Applications base directory that was corrupted or from which files were deleted:

1. Stop all relevant processes. That is, stop all processes that are related to Oracle Fusion Applications, such as the Administration Server, Node Manager, Managed Servers, and Oracle instances, as described in Section 4.4.3.2.

2. Recover the Applications base directory from the backup file. For example:

   (UNIX) tar -xf ApplBase_backup_030212.tar
   (Windows) jar xtf ApplBase_backup_030212.jar
   

3. Start all relevant processes. That is, start all processes that run in the Applications base home, as described in Section 4.4.3.1.

18.9.2 Recovering a Middleware Home

To recover a Middleware home that was corrupted or from which files were deleted:

1. Stop all relevant processes. That is, stop all processes that run in the Middleware home, such as the Administration Server, Node Manager, Managed Servers, and Oracle instances, as described in Section 4.4.3.2.

2. Recover the Middleware home directory from the backup file. For example:

   cd MW_HOME
   (UNIX) tar -xf mw_home_backup_030212.tar
   (Windows) jar xtf mw_home_backup_030212.jar
   

3. Start all relevant processes. That is, start all processes that run in the Middleware home, as described in Section 4.4.3.1.

18.9.3 Recovering an Oracle WebLogic Server Domain

To recover an Oracle WebLogic Server domain that was corrupted or deleted from the file system:

1. Stop all relevant processes. That is, stop all processes that are related to the domain, such as the Administration Server and Managed Servers, as described in Section 4.4.3.2.

2. Recover the domain directory from the backup file:

   cd DOMAIN_HOME
   (UNIX) tar -xf domain_backup_030212.tar 
   (Windows) jar xtf domain_backup_030212.jar 
   

3. Start all relevant processes. That is, start all processes that are related to the domain, as described in Section 4.4.3.1.

4. If you cannot start the Administration Server, recover it, as described in Section 18.9.6.

5. If you cannot start a Managed Server, recover it, as described in Section 18.9.7.

18.9.4 Recovering an Oracle Home

To recover an Oracle home from the backup file:

1. Recover the Oracle home to the original directory from a backup file. For example:

   cd ORACLE_HOME
   (UNIX) tar -xf Oracle_home_backup_030212.tar 
   (Windows) jar xtf Oracle_home_backup_030212.jar
   

2. Restart the Managed Server to which applications are deployed, using the WLST start command. For example:

   wls:/mydomain/serverConfig> start('myserver','Server')
   

18.9.5 Recovering an Oracle Instance Home

An Oracle instance home contains configuration information for system components, such as Oracle HTTP Server or Oracle Internet Directory.

This section includes the following topics:

• Section 18.9.5.1, "Recovering After Oracle Instance Home Deleted from File System"

• Section 18.9.5.2, "Recovering After Oracle Instance Home Deregistered"

18.9.5.1 Recovering After Oracle Instance Home Deleted from File System

To recover an Oracle instance home that was corrupted or deleted from the file system:

1. Stop all relevant processes. That is, kill all processes that are related to that Oracle instance.

2. Recover the Oracle instance home directory from a backup file. For example:

   cd ORACLE_INSTANCE
   (UNIX) tar -xf instance_home_backup_030212.tar 
   (Windows) jar xtf instance_home_backup_030212.jar
   

3. Start all relevant processes. That is, start all processes that are related to that Oracle instance:

   opmnctl startall
   

18.9.5.2 Recovering After Oracle Instance Home Deregistered

An Oracle instance must be registered with the domain. To recover an Oracle instance home that was deregistered from the domain:

1. Recover the Oracle instance home directory from a backup file. For example:

   cd ORACLE_INSTANCE
   (UNIX) tar -xf instance_home_backup_030212.tar 
   (Windows) jar xtf instance_home_backup_030212.jar
   

2. Register the Oracle instance, along with all of its components, with the Administration Server, using the opmnctl registerinstance command. For example:

   opmnctl registerinstance -adminHost admin_server_host 
        -adminPort admin_server_port -adminUsername username 
        -adminPassword password
        -oracleInstance ORACLE_INSTANCE_dir -oracleHome ORACLE_HOME_dir
        -instanceName Instance_name -wlserverHome Middleware_Home
   

18.9.6 Recovering the Administration Server Configuration

If the Administration Server configuration has been lost because of file deletion or file system corruption, the Administration Server console continues to function if it was already started when the problem occurred. The Administration Server directory is regenerated automatically, except for security information. As a result, whenever you start the Administration Server, it prompts for a user name and password. To prevent this, you can recover the configuration.

Caution:

Performing a domain-level recovery can impact other aspects of a running system and all of the configuration changes performed after the backup was taken will be lost.

To recover the Administration Server configuration:

1. Stop all processes, including the Administration Server, Managed Servers, and Node Manager, if they are started, as described in Section 4.4.3.2.

2. Recover the Administration Server configuration by recovering the domain home backup to a temporary location. Then, restore the config directory to the following location:

   (UNIX) DOMAIN_HOME/config
   (Windows) DOMAIN_HOME\config
   

3. Start the Administration Server as described in 0, "Starting the Administration Servers and Managed Servers" in Section 4.4.3.1.

4. Verify that the Administration Server starts properly and is accessible.

5. Start other processes, as described in Section 4.4.3.1.

On the next configuration change, the configuration from the Administration Server is pushed to the Managed Servers. On each Managed Server restart, the configuration is retrieved from the Administration Server.

18.9.7 Recovering a Managed Server

For many Oracle Fusion Applications, you recover the Managed Server in which the application is deployed. In addition, for some components, you may need to take additional steps, which are described in Section 18.9.10 and Section 18.9.11.

In this scenario, the Managed Server does not operate properly or cannot be started because the configuration has been deleted or corrupted or the configuration was mistakenly changed and you cannot ascertain what was changed.

To recover a Managed Server:

1. If the Administration Server is not reachable, recover the Administration Server, as described in Section 18.9.6.

2. If the Managed Server fails to start or if the file system is lost:

   1. Recover the Middleware home from the backup file, if required. For example:

      (UNIX) tar -xf mw_home_backup_030212.tar 
      (Windows) jar xtf mw_home_backup_030212.jar
      

   2. Create a domain template jar file for the Administration Server, using the pack utility.

      For example, on UNIX:

      pack.sh -domain=APPLICATIONS_CONFIG/instance/domains/domain_name
         -template=/scratch/temp.jar -template_name=domain1 
         -template_author=myname -log=/scratch/logs/my.log -managed=true
      

      For example, on Windows:

      pack.cmd -domain=APPLICATIONS_CONFIG\instance\domains\domain_name
         -template=C:\temp\temp.jar -template_name=domain1 
         -template_author=myname -log=C:\temp\logs\my.log -managed=true
      

      Specifying the -managed=true option packs up only the Managed Servers. If you want to pack the entire domain, omit this option.

   3. Unpack the domain template JAR file, using the unpack utility.

      For example, on UNIX:

      unpack.sh -template=/scratch/temp.jar
         -domain=APPLICATIONS_CONFIG/instance/domains/domain_name
         -log=/scratch/logs/new.log -log_priority=info
      

      For example, on Windows:

      unpack.cmd -template=C:\temp\temp.jar
         -domain=APPLICATIONS_CONFIG\instance\domains\domain_name
         -log=C:\temp\logs\new.log -log_priority=info
      

   4. Ensure that the application artifacts are accessible from the Managed Server host. That is, if the application artifacts are not on the same server as the Managed Server, they must be in a location accessible by the Managed Server.

      Note:

      • For stage mode applications, the Administration Server copies the application bits to the staged directories on the Managed Server hosts.

      • For nostage and external-stage mode applications, ensure that application files are available in the stage directories of the Managed Server.

      See Oracle Fusion Middleware Deploying Applications to Oracle WebLogic Server for information about stage, nostage, and external-stage mode applications.

   5. Start the Managed Server, as described in Section 4.4.4.1.

      The Managed Server connects to the Administration Server and updates its configuration changes.

18.9.8 Recovering the Databases

You can recover the databases used by Oracle Fusion Applications using RMAN. You can recover the databases at whatever level is appropriate by performing a restore and recover of the full database, a tablespace, or a data file.

Oracle Fusion Customer Relationship Management uses the following databases: Oracle Fusion Applications database, LDAP database, Oracle WebCenter Content, and Oracle Identity Manager database. You must maintain consistency among the databases. If any of these databases are recovered to a different point in time, you may need to reconcile the databases.

For information about recovering databases, see the Oracle Database Backup and Recovery User's Guide.

For information about reconciling the databases, see Section 18.9.9.

18.9.9 Reconciling the Data

The following topics describe how to recover and reconcile the databases to maintain consistency among the Oracle Fusion Applications database, LDAP database, and Oracle Identity Manager database.

• Section 18.9.9.1, "Recovering the Oracle Identity Manager Database and Reconciling It with the LDAP Database"

• Section 18.9.9.2, "Recovering the Oracle Fusion Applications Database and Reconciling It with the LDAP Database"

• Section 18.9.9.3, "Recovering the LDAP Database Using Multimaster Replication"

18.9.9.1 Recovering the Oracle Identity Manager Database and Reconciling It with the LDAP Database

Oracle Identity Manager users, role categories, role hierarchies, and role memberships are stored in the Oracle Identity Manager database. When a change in the information about users or roles takes place in Oracle Identity Manager, this information is propagated to the LDAP identity store. If the change takes place in the LDAP identity store directly, these changes are synchronized into Oracle Identity Manager. The LDAP identity store can be Oracle Internet Directory or any third party solutions such as Active Directory.

To recover the Oracle Identity Manager database, use RMAN to perform a point-in-time recovery of the Oracle Identity Manager database. See Oracle Database Backup and Recovery User's Guide for information on recovering a database.

If you restore the Oracle Identity Manager database to a different point in time than the LDAP store, the reconciliation engine checks the change logs and reapplies all the changes that happened in the time period between the restore of the LDAP store and the Oracle Identity Manager database. For example, if the Oracle Identity Manager database is restored so that is 10 hours behind the LDAP store, the reconciliation engine checks the change logs and reapplies all the changes that happened in the last 10 hours in the LDAP store to the Oracle Identity Manager database.

You do not need to explicitly trigger the reconciliation. LDAP synchronization is set up as a scheduled task to submit reconciliation events periodically. You can also start the reconciliation process manually and monitor the reconciliation events from the Oracle Identity Manager console. See "Reconciliation Configuration" in Oracle Fusion Middleware User's Guide for Oracle Identity Manager.

Note:

Oracle recommends that you make sure that the Oracle Identity Manager application is unavailable to the end users when a bulk reconciliation is occurring (as in the above recovery scenario). When the bulk reconciliation is complete, make sure that the Oracle Identity Manager application is again available to the end users. You can monitor the reconciliation with the Oracle Identity Manager console.

18.9.9.2 Recovering the Oracle Fusion Applications Database and Reconciling It with the LDAP Database

If the Oracle Fusion Applications database fails, and you restore it to a different point in time than the Oracle Identity Manager database, you may lose some user and roles data. For example, if you have created some users and roles in the Oracle Identity Manager database before the Oracle Fusion Applications database is restored, those users and roles will also exist in the LDAP database, but they will not exist in the Oracle Fusion Applications database.

To reconcile the Oracle Fusion Applications database with the Oracle Identity Manager and LDAP databases:

1. Restore the Oracle Fusion Applications database using RMAN to perform a point-in-time recovery. See Oracle Database Backup and Recovery User's Guide for information about recovering a database.

2. Synchronize the user and role information:

   1. Log in to the HcmCore application as the HR_SPEC_ALL user.

   2. Click EssLink.

   3. Click Retrieve Latest LDAP Changes.

   4. Click Submit.

   This is an asynchronous process that runs in the background.

3. When the process completes, find the users that were created in Oracle Identity Manager before the Oracle Fusion Applications database was restored. Use the following SQL command, where creation_date is the time you started running the synchronization process in Step 2:

   select * from per_users where creation_date > creation_date
   

4. For each user returned in the previous step, you create the user in Oracle Fusion Human Capital Management. For information, see "Creating and Updating Person and Employment Records" in the HCM Foundation help.

5. For each user, create the user's roles, as described in "User and Role Provisioning" in the HCM Foundation help.

18.9.9.3 Recovering the LDAP Database Using Multimaster Replication

Oracle recommends that you use multi-master replication for your LDAP database, as described in the "Setting up Multimaster Replication" section in the Oracle Fusion Middleware High Availability Guide.

When you use multimaster replication, if one LDAP node fails, the LDAP traffic is automatically routed to another node. You do not need to take any further action.

18.9.10 Recovering Oracle Fusion Applications

In most cases, to recover Oracle Fusion Applications, you recover a Middleware home, a domain, a server, an Oracle home, or an Oracle instance, depending on the extent of the failure. However, you may need to take additional steps for particular components.

This section includes the following topics:

• Section 18.9.10.1, "Recovering Oracle Fusion Customer Relationship Management"

• Section 18.9.10.2, "Recovering Oracle Fusion Procurement"

18.9.10.1 Recovering Oracle Fusion Customer Relationship Management

To recover Oracle Fusion Customer Relationship Management:

1. Recover the Managed Server to which Oracle Fusion Customer Relationship Management is deployed, as described in Section 18.9.7.

2. Recover the Java servers for Oracle E-Mail and Web Marketing, as described in Section 18.9.11.5.

3. Note that the Oracle Product Data Quality repository (the FUSION_DQ schema) and the customer source must be kept synchronized. If you restore one, restore the other to the same point in time. In addition, the Data Quality engine server artifacts in the file system, the Data Quality Admin Configuration data in the customer master (fusion), and the deployment topology must be kept synchronized.

For a complete case study for recovering Oracle Fusion Customer Relationship Management in different recovery scenarios, see Section 18.11.

18.9.10.2 Recovering Oracle Fusion Procurement

To recover Oracle Fusion Procurement:

1. Recover the Managed Server to which the application is deployed, as described in Section 18.9.7.

2. If you use a certificate file to make SSL connections to supplier's web sites, make sure that the certificate file exists in the location that you specified. If it does not, recover it from the backup file.

18.9.11 Recovering Components Related to Oracle Fusion Applications

You may need to recover components related to Oracle Fusion Applications.

This section includes the following topics:

• Section 18.9.11.1, "Recovering Oracle HTTP Server"

• Section 18.9.11.2, "Recovering Oracle Enterprise Scheduler"

• Section 18.9.11.3, "Recovering Oracle Enterprise Crawl and Search Framework"

• Section 18.9.11.4, "Recovering Oracle Authorization Policy Manager"

• Section 18.9.11.5, "Recovering Oracle E-Mail and Web Marketing"

18.9.11.1 Recovering Oracle HTTP Server

To recover Oracle HTTP Server, you recover the Oracle instance that contains Oracle HTTP Server, as described in Section 18.9.5.

18.9.11.2 Recovering Oracle Enterprise Scheduler

To recover Oracle Enterprise Scheduler:

1. Recover the domain directory from the backup file, as described in Section 18.9.3.

2. Recover the Oracle home to the original directory from the backup file, as described in Section 18.9.4.

3. Recover the database containing the Oracle Fusion applications and MDS schemas to the most recent point in time, if needed, as described in Section 18.9.8.

18.9.11.3 Recovering Oracle Enterprise Crawl and Search Framework

To recover Oracle Enterprise Crawl and Search Framework:

1. Recover the domain directory from the backup file, as described in Section 18.9.3.

2. Recover the database containing schemas related to Oracle Enterprise Crawl and Search Framework, as described in Section 18.9.8.

18.9.11.4 Recovering Oracle Authorization Policy Manager

To recover Oracle Authorization Policy Manager:

1. Stop all relevant processes. That is, stop all processes that are related to the domain, such as the Administration Server and Managed Servers, as described in Section 4.4.3.2.

2. Recover the domain directory from the backup file, as described in Section 18.9.3.

3. Recover the Oracle home to the original directory from the backup file, as described in Section 18.9.4.

4. Recover the LDAP store, if necessary.

   See the "Recovering Components" section in the Oracle Fusion Middleware Administrator's Guide.

5. Recover the database, if necessary. See Section 18.9.8.

6. Start all relevant processes. That is, start all processes that are related to the domain, as described in Section 4.4.3.1.

18.9.11.5 Recovering Oracle E-Mail and Web Marketing

To recover Oracle E-Mail and Web Marketing:

1. Recover the Managed Servers to which the Email Sending Daemon and Click Thru Daemon are deployed, as described in Section 18.9.7.

2. Recover the installation directory for the Bounce Handling Daemon.

18.10 Recovering After Loss of Host

This section describes recovery strategies after losing the original operating environment. For example, you could have a serious system malfunction or loss of media.

Depending on the extent of the failure, you can recover the Applications base directory, the Administration Server, a Managed Server, or the database. Some Oracle Fusion Applications components require additional steps, which are described in subsequent sections.

This section includes the following topics:

• Section 18.10.1, "Recovering the Applications Base Directory After Loss of Host"

• Section 18.10.2, "Recovering After Loss of Administration Server Host"

• Section 18.10.3, "Recovering After Loss of Managed Server Host"

• Section 18.10.4, "Recovering the Databases After Loss of Host"

• Section 18.10.5, "Additional Actions for Recovering Entities After Loss of Host"

This section also includes the following topics, which describe additional considerations you may need to take for Oracle Fusion Applications components:

• Section 18.10.6, "Recovering Oracle Fusion Applications After Loss of Host"

• Section 18.10.7, "Recovering Components Related to Oracle Fusion Applications"

18.10.1 Recovering the Applications Base Directory After Loss of Host

You can recover the Applications base directory if you lose the host that contains the directory.

To recover the Applications base directory:

1. Recover the Applications base directory from the backup file. For example:

   cd APPLICATIONS_BASE
   (UNIX) tar -xf ApplBase_backup_030212.tar
   (Windows) jar xtf ApplBase_backup_030212.jar
   

2. Start all relevant processes. That is, start all processes that run in the Applications base home, as described in Section 4.4.3.1.

18.10.2 Recovering After Loss of Administration Server Host

If you lose a host that contains the Administration Server, you can recover it to the same host or a different host.

This section includes the following topics:

• Section 18.10.2.1, "Recovering the Administration Server to the Same Host"

• Section 18.10.2.2, "Recovering the Administration Server to a Different Host"

18.10.2.1 Recovering the Administration Server to the Same Host

In this scenario, you recover the Administration Server either to the same host after the operating system has been reinstalled or to a new host that has the same host name. For example, the Administration Server is running on Host A and the Managed Server is running on Host B. Host A has failed for some reason and the Administration Server needs to be recovered.

To recover the Administration Server:

1. Recover the file system. For example, recover the domain containing the Administration Server, as described in Section 18.9.3.

2. Attempt to start the Administration Server, as described in Section 4.4.4.1.

   If the Administration Server starts, you do not need to take any further steps.

3. If the Administration Server fails to start, take the following steps on Host A:

   1. Stop all relevant processes. That is, stop all processes that are related to the domain, such as the Administration Server and Managed Servers, as described in Section 4.4.3.2.

   2. Recover the domain directory from the backup file:

      cd DOMAIN_HOME
      (UNIX) tar -xf domain_backup_030212.tar 
      (Windows) jar xtf domain_backup_030212.jar
      

      This restores the Administration Server as well as the Managed Servers in the domain.

   3. Start the Administration Server and Managed Servers, as described in Section 4.4.4.1.

   4. Start the Node Manager:

      java weblogic.WLST
      wls:/offline> startNodeManager()
      

18.10.2.2 Recovering the Administration Server to a Different Host

In this scenario, the Administration Server is running on Host A and the Managed Server is running on Host B. Host A has failed for some reason and the Administration Server needs to be moved to Host C.

Note:

Note that this scenario assumes that the shared location containing the binary files and the Administration Server configuration is intact. If it is not, follow the steps in Section 18.9.2 through Section 18.9.5.

To recover the Administration Server to a different host:

1. Because you have included a new host, mount the file system to the new host, Host C.

2. If the Administration Server has a Listen address, create a new machine with the new host name, as described in Section 18.10.5.3.

3. When you move .jks files to another host, you may receive warnings from the host name verification. You can regenerate those SSL certificates in the new host to set up SSL and configure a custom host name verifier. See the "Configure a Custom Host Name Verifier" section in the Oracle WebLogic Server Administration Console Help, which is located at:

   http://download.oracle.com/docs/cd/E21764_01/apirefs.1111/e13952/taskhelp/security/ConfigureACustomHostNameVerifier.html
   

   Also, see the "Configuring SSL" chapter in Oracle Fusion Middleware Securing Oracle WebLogic Server.

   The .jks files are located in the following directory:

   (UNIX) APPLICATIONS_BASE/fusionapps/wlserver_10.3/server/lib 
   (Windows) APPLICATIONS_BASE\fusionapps\wlserver_10.3\server\lib
   

4. Start the Node Manager on Host C if it was configured on the original host:

   java weblogic.WLST
   wls:/offline> startNodeManager()
   

5. Start the Administration Server, as described in Section 4.4.4.1.

6. If the Managed Servers are on the same failed host as the Administration Server, restore the Managed Servers, as described in Section 18.10.3.2.

7. Start the Managed Servers, as described in Section 4.4.4.1.

   The "Restarting a Failed Administration Server" section in the Oracle Fusion Middleware Managing Server Startup and Shutdown for Oracle WebLogic Server describes different ways to restart them, depending on how they were configured.

8. Ensure that additional application artifacts are available. For example, if the deployment mode is nostage or external-stage, applications may reside in directories outside of the domain directory. Make your application files available to the new Administration Server by copying them from backups or by using a shared disk. Your application files should be available in the same relative location on the new file system as on the file system of the original Administration Server.

   If the application is staged, the Administration Server copies the application bits to the staged directories on the Managed Server hosts.

9. If your environment contains Oracle HTTP Server, modify the FusionVirtualHost_x.conf file, as described in Section 18.10.5.2.

10. Update Oracle Inventory, as described in Section 18.10.5.4.

11. Edit the targets.xml file for Fusion Middleware Control, as described in Section 18.10.5.1.

18.10.3 Recovering After Loss of Managed Server Host

If you lose a host that contains a Managed Server, you can recover it to the same host or a different host.

This section includes the following topics:

• Section 18.10.3.1, "Recovering a Managed Server to the Same Host"

• Section 18.10.3.2, "Recovering a Managed Server to a Different Host"

18.10.3.1 Recovering a Managed Server to the Same Host

In this scenario, you recover a Managed Server to the same host after the operating system has been reinstalled or to a new host that has the same host name. The Administration Server is running on Host A and the Managed Server is running on Host B. Host B failed for some reason and the Managed Server needs to be recovered to Host B.

1. Start the Node Manager on Host B:

   java weblogic.WLST
   wls:/offline> startNodeManager()
   

2. Start the Managed Server, as described in Section 4.4.4.1.

   If the Managed Server starts, it connects to the Administration Server and updates its configuration changes. You do not need to take any further steps.

3. If the Managed Server fails to start or if the file system is lost:

   1. Stop the Node Manager:

      java weblogic.WLST
      wls:/offline> stopNodeManager()
      

   2. Recover the Middleware home to Host B from the backup file, if required:

      cd MW_HOME
      (UNIX) tar -xf mw_home_backup_030212.tar 
      (Windows) jar xtf mw_home_backup_030212.jar
      

   3. Create a domain template jar file for the Administration Server running in Host A, using the pack utility.

      For example, on UNIX:

      pack.sh -domain=APPLICATIONS_CONFIG/instance/domains/domain_name
         -template=/scratch/temp.jar -template_name=domain1
         -template_author=myname -log=/scratch/logs/my.log -managed=true
      

      For example, on Windows:

      pack.cmd -domain=APPLICATIONS_CONFIG\instance\domains\domain_name
         -template=C:\temp\temp.jar -template_name=domain1
         -template_author=myname -log=C:\temp\logs\my.log -managed=true
      

      Specifying the -managed=true option packs up only the Managed Servers. If you want to pack the entire domain, omit this option.

   4. Unpack the domain template jar file in Host B, using the unpack utility.

      For example, on UNIX:

      unpack.sh -template=/scratch/temp.jar
         -domain=APPLICATIONS_CONFIG/instance/domains/domain_name
         -log=/scratch/logs/new.log -log_priority=info
      

      For example, on Windows:

      unpack.cmd -template=C:\temp\temp.jar
         -domain=APPLICATIONS_CONFIG\instance\domains\domain_name
         -log=C:\temp\logs\new.log -log_priority=info
      

   5. Ensure that the application artifacts are accessible from the Managed Server host. That is, if the application artifacts are not on the same server as the Managed Server, they must be in a location accessible by the Managed Server.

      Note:

      • For applications that are deployed in nostage or external-stage mode, copy the application artifacts from the Administration Server host directory.

      • For applications that are deployed in stage mode, the Administration Server copies the application bits to the staged directories on the Managed Server hosts.

      See Oracle Fusion Middleware Deploying Applications to Oracle WebLogic Server for information about deploying applications.

   6. If the Node Manager is not started, start it:

      java weblogic.WLST
      wls:/offline> startNodeManager()
      

   7. Start the Managed Server, as described in Section 4.4.4.1.

      The Managed Server connects to the Administration Server and updates its configuration changes.

18.10.3.2 Recovering a Managed Server to a Different Host

In this scenario, the Administration Server is running on Host A and the Managed Server is running on Host B. Host B failed for some reason and the Managed Server needs to be recovered to Host C.

Important:

Recover the Middleware home to the same location as the original.

To recover a Managed Server to a different host:

1. If the Managed Server configuration files are local to the host (that is, they are not on a shared file system):

   1. Recover the Middleware home for the Managed Server to Host C.

      cd MW_HOME
      (UNIX) tar -xf mw_home_backup_030212.tar 
      (Windows) jar xtf mw_home_backup_030212.jar
      

      Note that when you restore the Middleware home, you are restoring all of the domains because they are on a shared file system.

   2. Create a domain template jar file from the Administration Server running in Host A, using the pack utility.

      For example, on UNIX:

      pack.sh -domain=APPLICATIONS_CONFIG/instance/domains/domain_name
         -template=/scratch/temp.jar -template_name=domain1
         -template_author=myname -log=/scratch/logs/my.log -managed=true
      

      For example, on Windows:

      pack.cmd -domain=APPLICATIONS_CONFIG\instance\domains\domain_name
         -template=C:\temp\temp.jar -template_name=domain1 
         -template_author=myname -log=C:\temp\logs\my.log -managed=true
      

      Specifying the -managed=true option packs up only the Managed Servers. If you want to pack the entire domain, omit this option.

   3. Unpack the domain template jar file on Host C, using the unpack utility.

      For example, on UNIX:

      unpack.sh -template=/scratch/temp.jar
         -domain=APPLICATIONS_CONFIG/instance/domains/domain_name
         -log=/scratch/logs/new.log -log_priority=info
      

      For example, on Windows:

      unpack.cmd -template=C:\temp\temp.jar
         -domain=APPLICATIONS_CONFIG\instance\domains\domain_name
         -log=C:\temp\logs\new.log -log_priority=info
      

      If you are recovering to a different domain home, use the -app_dir switch in the unpack command.

2. Ensure that the application artifacts are accessible from the Managed Server host. That is, if the application artifacts are not on the same server as the Managed Server, they must be in a location accessible by the Managed Server.

   Note:

   • For applications that are deployed in nostage or external-stage mode, copy the application artifacts from the Administration Server host directory.

   • For applications that are deployed in stage mode, the Administration Server copies the application bits to the staged directories on the Managed Server hosts.

   See Oracle Fusion Middleware Deploying Applications to Oracle WebLogic Server for information about deploying applications.

3. If the Managed Server is not co-located with the Administration Server, take the following steps. (These steps are not needed if the Managed Server is co-located with the Administration Server.)

   1. Edit the nodemanager.domains file, specifying the domain names and domain directories. Use the following format:

      domain_name=domain_directory
      

   2. Start the Node Manager on Host C, if it is not started:

      java weblogic.WLST
      wls:/offline> startNodeManager()
      

   3. Using WLST, connect to the Administration Server and then enroll the Node Manager running in the new host with the Administration Server:

      connect('username','password','host:port')
      nmEnroll('APPLICATIONS_CONFIG/instance/domains/domain_name',
        'MW_HOME/wlserver_n/common/nodemanager/instance_name')
      

   4. Change the Managed Server configuration to point to the new host:

      • In the WebLogic Server Administration Console, create a machine, which is a logical representation of the computer that hosts one or more WebLogic Servers, and point it to the new host. (From the Home page, select Machines. Then, click New.) Follow the directions in the Administration Console help.

        If you identify the Listen Address by IP address, you must disable Host Name Verification on the Administration Servers that access Node Manager. For more information, see the "Using Host Name Verification" section in Oracle Fusion Middleware Securing Oracle WebLogic Server.

      • Change the Managed Server configuration to point to the new machine. (From the left pane of the Console, expand Environment and then Servers. Then, select the name of the server. Select the Configuration tab, then the General tab. In the Machine field, select the machine to which you want to assign the server.)

        Change Listen Address to the new host. (If the listening address was set to blank, you do not need to change it.)

      You only need to take these steps once for all Managed Servers on the same host.

4. When you move .jks files to another host, you may receive warnings from the host name verification. You can regenerate those SSL certificates in the new host to set up SSL and configure a custom host name verifier. See the "Configure a Custom Host Name Verifier" section in the Oracle WebLogic Server Administration Console Help. Also, see the "Configuring SSL" chapter in Oracle Fusion Middleware Securing Oracle WebLogic Server.

   The .jks files are located in the following directory:

   (UNIX) APPLICATIONS_BASE/fusionapps/wlserver_10.3/server/lib 
   (Windows) APPLICATIONS_BASE\fusionapps\wlserver_10.3\server\lib
   

5. Start the Managed Server, as described in Section 4.4.4.1.

   The Managed Server connects to the Administration Server and updates its configuration changes.

6. If your environment contains Oracle HTTP Server, modify the FusionVirtualHost_x.conf file, as described in Section 18.10.5.2.

7. Edit the targets.xml file for Fusion Middleware Control, as described in Section 18.10.5.1.

Now you can start and stop the Managed Server on Host C using the Administration Server running on Host A.

18.10.4 Recovering the Databases After Loss of Host

If the physical host where your database resides is lost, you can recover the database using RMAN.

See "Restoring a Database on a New Host" in the Oracle Database Backup and Recovery User's Guide to learn how to use RMAN to recover your database in the event of a complete failure on the primary database host computer.

Oracle Fusion Customer Relationship Management uses the following databases: Oracle Fusion Applications database, LDAP database, Oracle WebCenter Content, and Oracle Identity Manager database. You must maintain consistency among the databases. If any of these databases are recovered to a different point in time, you may need to reconcile the databases.

For information about reconciling the LDAP database, the Oracle Fusion Applications database, and the Oracle Identity Manager database, see Section 18.9.9.

18.10.5 Additional Actions for Recovering Entities After Loss of Host

Depending on the entity that you are recovering, you may need to take additional actions after loss of host. The sections about each entity may require you to follow one or more of the following procedures. If so, that is noted in the section describing how to recover the entity.

This section includes the following topics:

• Section 18.10.5.1, "Changing the Host Name in the targets.xml File for Fusion Middleware Control"

• Section 18.10.5.2, "Modifying the FusionVirtualHost_x.conf File"

• Section 18.10.5.3, "Creating a New Machine for the New Host Name"

• Section 18.10.5.4, "Updating Oracle Inventory"

18.10.5.1 Changing the Host Name in the targets.xml File for Fusion Middleware Control

When you recover a component to a different host, you must update the targets.xml file for Fusion Middleware Control. The file is located at:

(UNIX) APPLICATIONS_CONFIG/instance/domains/hostname/domain_name/sysman/state/targets.xml
(Windows) APPLICATIONS_CONFIG\instance\domains\hostname\domain_name\sysman\state\targets.xml

In the file, change the host name to the new host name for components that are recovered to a different host.

18.10.5.2 Modifying the FusionVirtualHost_x.conf File

When you recover an Administration Server or a Managed Server to a different host and your environment includes Oracle HTTP Server, you must modify the FusionVirtualHost_x.conf file on the new host. There is a separate file for each domain, for example FusionVirtualHost_fin.conf. The files are located in:

(UNIX) APPLICATIONS_CONFIG/instance/CommonDomain_webtier/config/OHS/ohs_name/moduleconf/FusionVirtualHost_x.conf
(Windows) APPLICATIONS_CONFIG\instance\CommonDomain_webtier\config\OHS\ohs_name\moduleconf\FusionVirtualHost_x.conf

Modify all of the instances of the host name and clusters (elements such as WebLogicHost and WebLogicCluster) entries in that file. For example:

<Location /console>
    SetHandler weblogic-handler
    WebLogicHost Admin_Host
    WeblogicPort Admin_Port
    WLProxySSL ON
    WLProxySSLPassThrough ON
</Location>
 .
 .
 .
<Location /soa-infra>
   SetHandler weblogic-handler
   WebLogicCluster SOAHOST1:8001,*SOAHOST2*:*8001*
   WLProxySSL ON
   WLProxySSLPassThrough ON
</Location> 

18.10.5.3 Creating a New Machine for the New Host Name

If the Administration Server has an Listen address, you must create a new machine with the new host name and set the Listen address, before you start the Administration Server. A machine is a logical representation of the computer that hosts one or more WebLogic Servers.

To create a new machine:

1. Create a new machine with the new host name. Use the following WLST commands, in offline mode:

   readDomain('Domain_Home')
   machine = create('newhostname', 'Machine')
   cd('/Machine/newhostname')
   nm = create('newhostname', 'NodeManager')
   
   cd('/Machine/newhostname/NodeManager/newhostname')
   set('ListenAddress', 'newhostname')
   updateDomain()
   

2. For the Administration Server, set the machine with the new host name, using the following WLST command, in offline mode:

   readDomain('DomainHome')
   cd ('/Machine/newhostname')
   machine = cmo
   cd ('/Server/AdminServer')
   set('Machine', machine)
   updateDomain()
   

3. Set the Listen address for the Administration Server:

   readDomain("DomainHome")
   cd("servers/AdminServer")
   cmo.setListenPort(8001)
   updateDomain()
   exit()
   

18.10.5.4 Updating Oracle Inventory

For many components, when you recover to a different host, as in the case of loss of host, you must update the Oracle Inventory on UNIX. To do so, execute the following script:

ORACLE_COMMON_HOME/oui/bin/attachHome.sh 

In addition, you must update beahomelist to edit the location of a Middleware home. Edit the following file to update the Middleware home information:

(UNIX) user_home/bea/beahomelist
(Windows) C:\bea\beahomelist

18.10.6 Recovering Oracle Fusion Applications After Loss of Host

In most cases, to recover Oracle Fusion Applications, you recover the entire Middleware home, a domain, a server, an Oracle home, or an Oracle instance, depending on the extent of the failure. However, you may need to take additional steps you for particular components.

This section includes the following topics:

• Section 18.10.6.1, "Recovering Oracle Fusion Supply Chain Management After Loss of Host"

• Section 18.10.6.2, "Recovering Oracle Fusion Procurement After Loss of Host"

18.10.6.1 Recovering Oracle Fusion Supply Chain Management After Loss of Host

If you lose a host that contains Oracle Fusion Supply Chain Management, you can recover it to the same host or a different host.

To recover Oracle Fusion Supply Chain Management to the same host, recover the Managed Server to which the application is deployed, as described in Section 18.10.3.

To recover Oracle Fusion Supply Chain Management to a different host:

1. Recover the Managed Server to which the application is deployed, as described in Section 18.10.3.

2. Recover the Oracle instance for Global Order Promising, as described in Section 18.9.5.

18.10.6.2 Recovering Oracle Fusion Procurement After Loss of Host

If you lose a host that contains Oracle Fusion Procurement, you can recover it to the same host or a different host.

To recover Oracle Fusion Procurement to the same host, recover the Managed Server to which the application is deployed, as described in Section 18.10.3.

To recover Oracle Fusion Procurement to a different host:

1. Recover the Managed Server to which the application is deployed, as described in Section 18.10.3.

2. Ensure that Oracle Business Intelligence Publisher and CUPS IP address and port number reflects the different host.

18.10.7 Recovering Components Related to Oracle Fusion Applications

In most cases, to recover components related to Oracle Fusion Applications, you recover the entire Middleware home, a domain, a server, an Oracle home, or an Oracle instance, depending on the extent of the failure. You may need to take additional steps for particular components related to Oracle Fusion Applications.

This section includes the following topics:

• Section 18.10.7.1, "Recovering Oracle HTTP Server After Loss of Host"

• Section 18.10.7.2, "Recovering Oracle Enterprise Scheduler After Loss of Host"

• Section 18.10.7.3, "Recovering Oracle Enterprise Crawl and Search Framework After Loss of Host"

• Section 18.10.7.4, "Recovering Oracle Authorization Policy Manager After Loss of Host"

• Section 18.10.7.5, "Recovering Oracle E-Mail and Web Marketing After Loss of Host"

• Section 18.10.7.6, "Recovering Oracle Essbase In Clustered Environment After Loss of Host"

18.10.7.1 Recovering Oracle HTTP Server After Loss of Host

To recover Oracle HTTP Server to the same host, recover the Oracle instance, as described in Section 18.9.5.

To recover Oracle HTTP Server to a different host:

1. Recover the Middleware home, as described in Section 18.9.2.

2. Start all relevant processes, as described in Section 4.4.1.

3. Update the registration of the Oracle instance with the Administration Server, using the opmnctl updateinstanceregistration command on the new host. For example:

   opmnctl updateinstanceregistration -adminHost admin_server_host 
   

   This command updates the OPMN instance.properties file.

4. Update the registration of the component with the Administration Server, using the opmnctl updatecomponentregistration command on the new host. For example, to update the registration for Oracle HTTP Server, use the following command:

   opmnctl updatecomponentregistration -Host new_host -Port nonSSLPort
       -componentName ohs1 -componentType OHS
   

5. Edit the targets.xml file for Fusion Middleware Control, as described in Section 18.10.5.1.

6. Modify the ServerName entry in the following file to have the new host name:

   (UNIX) ORACLE_INSTANCE/config/OHS/ohs_name/httpd.conf
   (Windows) ORACLE_INSTANCE\config\OHS\ohs_name\httpd.conf
   

7. Modify the FusionVirtualHost_x.conf file, as described in Section 18.10.5.2.

8. If the Managed Server is in a cluster, modify the following files (there is more than one file, with the name ear_name.ear):

   (UNIX) APPLICATIONS_CONFIG/fusionapps/applications/domain_name/deploy/ear_name.ear/APP-INF/classes/wf_client_config.xml
   (Windows) APPLICATIONS_CONFIG\fusionapps\applications\domain_name\deploy\ear_name.ear\APP-INF\classes\wf_client_config.xml
   

   Change the rootEndPointURL element so that it points to the Oracle HTTP Server or Load Balancer for the cluster on the new environment.

18.10.7.2 Recovering Oracle Enterprise Scheduler After Loss of Host

To recover Oracle Enterprise Scheduler, follow the procedure in Section 18.9.11.2.

18.10.7.3 Recovering Oracle Enterprise Crawl and Search Framework After Loss of Host

To recover Oracle Enterprise Crawl and Search Framework to a different host, follow the procedure in Section 18.9.11.3.

18.10.7.4 Recovering Oracle Authorization Policy Manager After Loss of Host

To recover Oracle Authorization Policy Manager, follow the procedure in Section 18.9.11.4.

18.10.7.5 Recovering Oracle E-Mail and Web Marketing After Loss of Host

To recover Oracle E-Mail and Web Marketing, follow the procedure in Section 18.9.11.5.

18.10.7.6 Recovering Oracle Essbase In Clustered Environment After Loss of Host

To recover Oracle Essbase after loss of host, you recover the entire Middleware home, a domain, a server, an Oracle home, or an Oracle instance, depending on the extent of the failure.

However, if the failed node contained Oracle Essbase and was clustered, you must configure secondary instances of Oracle Essbase Agent so that they are distributed for high availability. For more information, see "Configuring Oracle Essbase Clustering" in the Oracle Fusion Applications Customer Relationship Management Enterprise Deployment Guide.

18.11 A Case Study: Recovering Oracle Fusion Customer Relationship Management

This section provides a case study of recovering Oracle Fusion Customer Relationship Management and all of the other Oracle Fusion Applications offerings and Oracle Fusion Middleware components that are installed when you install Oracle Fusion Customer Relationship Management.

You can recover all or part of your Oracle Fusion Applications environment, as described in Section 18.3.

This section includes the following topics:

• Section 18.11.1, "The Recovery Case Study Scenario"

• Section 18.11.2, "Recovering the Middleware Home Containing Java EE Components"

• Section 18.11.3, "Recovering the Web Tier Middleware Home"

• Section 18.11.4, "Recovering the Oracle Fusion Customer Relationship Management Domain"

• Section 18.11.5, "Recovering Servers When the Installation Directory Is Shared Between Hosts"

• Section 18.11.6, "Recovering Servers When the Managed Server Configuration Is Local"

• Section 18.11.7, "Recovering an Oracle Instance in an Oracle Fusion Customer Relationship Management Installation"

• Section 18.11.8, "Recovering an Oracle Fusion Customer Relationship Management Cluster"

• Section 18.11.9, "Recovering Databases for Oracle Fusion Customer Relationship Management"

18.11.1 The Recovery Case Study Scenario

In this case study, the following domains were created when you installed and provisioned Oracle Fusion Customer Relationship Management:

Oracle Fusion Applications Component	Domain Name
Oracle Fusion Customer Relationship Management	CRMDomain
Oracle Fusion Financials	FinancialDomain
Oracle Fusion Human Capital Management	HCMDomain
Oracle Fusion Supply Chain Management	SCMDomain
Oracle Fusion Setup	CommonDomain

In addition, Oracle Fusion Customer Relationship Management uses the following Oracle Fusion Middleware components:

• Oracle WebCenter Portal

• Oracle Business Intelligence

• Oracle Essbase

• Oracle Enterprise Scheduler

• Oracle SOA Suite

• Oracle WebCenter Content

• Oracle Application Development Framework

Oracle Fusion Customer Relationship Management is installed in the following directory:

/scratch/oracle/APPLICATIONS_BASE

The APPLICATIONS_BASE directory contains:

• The Oracle Fusion Applications Middleware home directory, which is located at:

  /scratch/oracle/APPLICATIONS_BASE/fusionapps
  

• One Middleware home that hosts the Java EE components, including Oracle SOA Suite and Oracle WebCenter Portal, and is located at:

  /scratch/oracle/APPLICATIONS_BASE/fusionapps
  

• A second Middleware home that hosts system components such as Oracle HTTP Server and is located at:

  /scratch/oracle/APPLICATIONS_BASE/webtier_mw_home
  

Note:

In this case study, the Oracle Fusion application files are installed in shared storage, which is mounted on all hosts that run the applications. A single copy of binary files is shared across multiple hosts. The Administration Server configuration is also shared. However, the Managed Server configurations are not shared.

18.11.2 Recovering the Middleware Home Containing Java EE Components

In this scenario, you recover the Oracle Fusion Applications Middleware home, which contains Oracle Fusion Applications instances, and the Java EE components, Oracle SOA Suite and Oracle WebCenter Portal, after data loss or corruption.

To recover the Middleware home:

1. Stop all relevant processes using Fusion Applications Control. That is, stop all Managed Servers that are related to Oracle SOA Suite and Oracle WebCenter Portal. For example, stop the FINSOAServer_1 (in the FinancialDomain), CRMSOAServer_1 (in the CRMDomain), SCMSOAServer_1 (in the SCMDomain), and FSSOAServer (in the HCMDomain). Stop the Administration Server, as described in Section 4.4.4.2.

2. From the original Middleware home directory, move the files to a backup directory:

   mv /scratch/oracle/APPLICATIONS_BASE/fusionapps
       /scratch/oracle/APPLICATIONS_BASE/fusionapps_backup
   

   This ensures that any needed files are not overwritten.

3. Create a staging directory and restore the Applications base directory to that directory:

   cd /scratch/stage/
   tar -xvf crm_ApplBase_backup_030212.tar
   

4. From that stage directory, copy the Middleware home to the original location:

   cp /scratch/stage/APPLICATIONS_BASE/fusionapps
      /scratch/oracle/APPLICATIONS_BASE/fusionapps
   

5. Start all relevant processes, as described in Section 4.4.3.1.

18.11.3 Recovering the Web Tier Middleware Home

In this scenario, you recover the Web tier Middleware home that contains system components, such as Oracle HTTP Server, after data loss or corruption.

To recover the Middleware home:

1. Stop all relevant processes, such as all processes running in the Oracle instance, using the following command:

   opmnctl stopall
   

2. From the original Middleware home directory, move the files to a backup directory:

   mv /scratch/oracle/APPLICATIONS_BASE/webtier_mw_home
     /scratch/oracle/APPLICATIONS_BASE/webtier_mw_home_backup
   

   This ensures that any needed files are not overwritten.

3. Create a staging directory and restore the Applications base directory to that directory:

   cd /scratch/stage
   tar -xvf crm_ApplBase_backup_030212.tar
   

4. From that stage directory, copy the Middleware home to the original location:

   cp /scratch/stage/APPLICATIONS_BASE/webtier_mwhome
     /scratch/oracle/APPLICATIONS_BASE/webtier_mwhome
   

5. Start all relevant processes, using the following command:

   opmnctl startall
   

18.11.4 Recovering the Oracle Fusion Customer Relationship Management Domain

In this scenario, you recover the domain that contains Oracle Fusion Customer Relationship Management after data loss or corruption. This procedure shows recovering the domain CRMDomain, but you can use the same procedure for other Oracle Fusion Applications domains.

To recover the domain:

1. Stop all Managed Servers in the CRMDomain, as described in Section 4.4.4.2.

2. Stop the Administration Server for the CRMDomain, as described in Section 4.4.4.2.

3. From the original domain directory, move the files to a backup directory:

   mv /scratch/oracle/APPLICATIONS_CONFIG/instance/domains/CRMDomain
     /scratch/oracle/APPLICATIONS_CONFIG/instance/domains/CRMDomain_backup
   

   This ensures that any needed files are not overwritten.

4. Create a staging directory and restore the Applications base directory to that directory:

   cd /scratch/stage
   tar -xvf crm_ApplBase_backup_030212.tar
   

5. From that stage directory, copy the domain to the original location:

   cp /scratch/stage/APPLICATIONS_CONFIG/instance/domains/CRMDomain
     /scratch/oracle/APPLICATIONS_CONFIG/instance/domains/CRMDomain
   

6. Start the Administration Server, as described in Section 4.4.4.1.

7. Start all Managed Servers, as described in Section 4.4.4.1.

18.11.5 Recovering Servers When the Installation Directory Is Shared Between Hosts

When the applications directory is shared between hosts, and you need to recover an Administration Server or a Managed Server, you must recover the entire domain.

The following table shows a sample domain mapping and provides guidelines for recovering an Oracle WebLogic Server Administration Server or any Managed Server running in the domain. The list of servers in the table is not complete. Refer to the particular Oracle Fusion application offering for the complete list of servers.

Domain Name	Servers Running in Domain	What to Recover
CRMDomain	AdminServer CRMCommonServer_1 CRMPerformanceServer_1 CRMAnalyticsServer_1 CRMSearchServer_1 CRMODIServer_1 CRMSOAServer_1 CustomerServer_1 SalesServer_1 MarketingServer_1 OrderCaptureServer_1	The entire CRMDomain. For example: APPLICATIONS_CONFIG/instance/domains/CRMDomain
FinancialDomain	AdminServer FinCommonServer_1 FinAnalyticsServer_1 FinSOAServer_1 LedgerServer_1 ESSServer_1	The entire FinancialDomain. For example: APPLICATIONS_CONFIG/instance/domains/FinancialDomain
SCMDomain	AdminServer ProductManagementServer_1 SCMSOAServer_1	The entire SCMDomain. For example: APPLICATIONS_CONFIG/instance/domains/SCMDomain
HCMDomain	AdminServer CoreSetupServer_1 FINSOAServer_1 TalentServer_1 ESSServer_1 HCMAnalyticsServer_1	The entire HCMDomain. For example: APPLICATIONS_CONFIG/instance/domains/HCMDomain
SetupDomain	AdminServer HomePageServer_1 HelpPortalServer_1 FunctionalSetupServer_1 WCServices_1 SpacesServer_1	The entire SetupDomain. For example: APPLICATIONS_CONFIG/instance/domains/SEtupDomain

18.11.6 Recovering Servers When the Managed Server Configuration Is Local

If domain directory of the Managed Server is different than the Administration Server domain directory, and you need to recover a Managed Server, you use pack and unpack to recover the Managed Server.

For example, to pack and unpack the Managed Server FinCommonServer_1:

1. If the Administration Server is not reachable, recover the Administration Server, as described in Section 18.9.6.

2. If the Managed Server fails to start or if the file system is lost:

   1. Recover the Middleware home from the backup file, if required. For example:

      cd MW_HOME
      tar -xf mw_home_backup_030212.tar 
      

   2. Create a domain template jar file for the FinancialDomain Administration Server, using the pack utility. For example:

      pack.sh -domain=APPLICATIONS_CONFIG/instance/domains/FinancialDomain
         -template=/scratch/temp.jar -template_name=domain1 
         -template_author=myname -log=/scratch/logs/my.log -managed=true
      

      Specifying the -managed=true option packs up only the Managed Servers. If you want to pack the entire domain, omit this option.

   3. Unpack the domain template JAR file, using the unpack utility. For example:

      unpack.sh -template=/scratch/temp.jar
         -domain=APPLICATIONS_CONFIG/instance/domains/FinancialDomain
         -log=/scratch/logs/my.lognew.log -log_priority=info
      

   4. Ensure that the application artifacts are accessible from the Managed Server host. That is, if the application artifacts are not on the same server as the Managed Server, they must be in a location accessible by the Managed Server.

      Note:

      • For stage mode applications, the Administration Server copies the application bits to the staged directories on the Managed Server hosts.

      • For nostage and external-stage mode applications, ensure that application files are available in the stage directories of the Managed Server.

      See Oracle Fusion Middleware Deploying Applications to Oracle WebLogic Server for information about stage, nostage, and external-stage mode applications.

   5. Start the Managed Server, as described in Section 4.4.4.1.

      The Managed Server connects to the Administration Server and updates its configuration changes.

18.11.7 Recovering an Oracle Instance in an Oracle Fusion Customer Relationship Management Installation

To recover the Oracle instance ohs in an Oracle Fusion Customer Relationship Management installation:

1. Stop all relevant processes using opmnctl or kill all processes that are related to that Oracle instance. For example:

   opmnctl stopall
   

2. Copy the original Oracle instance to a backup location in case you need to retrieve some configuration data:

   cp /scratch/CRM/APPLICATIONS_CONFIG/webtier_mw_home/webtier/ohs 
      /scratch/CRM/APPLICATIONS_CONFIG/webtier_mw_home/webtier/ohs_backup
   

3. Restore the Oracle instance home from the backup file to a stage directory by restoring the Applications configuration directory:

   mkdir stage
   cd stage
   tar -xvf crm_ApplConfig_backup_030212.tar
   

4. Copy the restored Oracle instance from step 3 to the original location:

   cp /scratch/stage/APPLICATIONS_CONFIG/webtier_mw_home/webtier/ohs/  
       /scratch/CRM/APPLICATIONS_CONFIG/webtier_mw_home/webtier/ohs
   

5. Start all relevant processes. That is, start all processes that are related to that Oracle instance:

   opmnctl startall
   

Note:

If you are recovering the Oracle instance containing Oracle HTTP Server to a new host after loss of host, see Section 18.10.7.1.

18.11.8 Recovering an Oracle Fusion Customer Relationship Management Cluster

In this scenario, the cluster CRMPerformanceCluster has been erroneously deleted or the cluster-level configuration, such as the JMS configuration or container-level data sources, was mistakenly changed and committed. The server CRMPerformanceServer_1 cannot be started or does not operate properly or the services running inside the server are not starting. You cannot ascertain what was changed.

Caution:

Performing a domain-level recovery can impact other aspects of a running system and all of the configuration changes performed after the backup was taken will be lost.

If the configuration changes are few, then the easiest way is to redo the configuration changes. If that is not feasible, restore the domain to which the cluster belongs. (See the table in Section 18.11.5 for Managed Servers to domain mappings.) Take the following steps:

1. Stop the cluster:

   stop('CRMPerformanceCluster', 'Cluster')
   

2. Stop the Administration Server, as described in 0, "Stopping the Administration Servers and Managed Servers" in Section 4.4.3.2.

3. Copy the original CRMDomain to a backup location in case you need to retrieve some configuration data:

   cp /scratch/CRM/APPLICATIONS_CONFIG/instance/domains/CRMDomain 
      /scratch/CRM/APPLICATIONS_CONFIG/instance/domains/CRMDomain_backup
   

4. Restore the CRMDomain home from the backup file to a stage directory by restoring the Applications configuration directory:

   mkdir stage
   cd stage
   tar -xvf crm_ApplConfig_backup_030212.tar
   

5. Copy the restored CRMDomain from step 4 to the original location:

   cp /scratch/stage/APPLICATIONS_CONFIG/instance/domains/CRMDomain
      /scratch/CRM/APPLICATIONS_CONFIG/instance/domains/CRMDomain
   

6. Start the Administration Server, as described in 0, "Starting the Administration Servers and Managed Servers" in Section 4.4.3.1.

7. Start the cluster. You can use the Oracle WebLogic Server Administration Console or WLST. For example, to use the WLST start command:

   start('CRMPerformanceCluster', 'Cluster')
   

Note:

You can use this procedure to recover a cluster if the membership is mistakenly deleted.

18.11.9 Recovering Databases for Oracle Fusion Customer Relationship Management

You can recover the databases using RMAN. You can recover the databases at whatever level is appropriate by performing a restore and recovery of either the full databases, a tablespace, or a data file.

Oracle Fusion Customer Relationship Management uses the following databases: Oracle Fusion Applications database, LDAP database, Oracle WebCenter Content, and Oracle Identity Manager database. You must maintain consistency among the databases. If any of these databases are recovered to a different point in time, you may need to reconcile the databases, as described in Section 18.9.9.

For more information, see the following topics:

• For data loss, corruption, or media failure loss, see Section 18.9.8.

• For loss of host, see Section 18.10.4.