2
Middle Tier High Availability

This chapter describes solutions that are available to protect the Oracle Application Server middle tier from failures. It is organized into the following main sections:

• OracleAS Middle Tier Overview

• Features and Components for Middle Tier High Availability

• HTTP Service High Availability

• J2EE High Availability

• Oracle Application Server Portal High Availability

• Oracle Application Server Wireless High Availability

• Business Intelligence High Availability

• Oracle Application Server Forms Services High Availability

• Oracle Application Server Integration High Availability

• Middle Tier Recovery Solutions

OracleAS Middle Tier Overview

OracleAS middle tier consists of components that provide front-end application services to clients. The middle tier can be created with one of three installation types:

• J2EE and Web Cache

  This installation type installs components for Oracle Application Server Containers for J2EE (OC4J) and Oracle Application Server Web Cache (OracleAS Web Cache). These components provide J2EE 1.3 runtime containers, and static and dynamic content caching functionality. Of the three installation types, J2EE and Web Cache installations are the only ones that can be clustered together as OracleAS Clusters.

• Portal and Wireless

  This installation type installs Oracle Application Server Portal and Oracle Application Server Wireless components in addition to the components in the J2EE and Web Cache installation type.

• Business Intelligence and Forms

  This installation installs components for Oracle Application Server Forms Services, Oracle Application Server Reports Services, Oracle Application Server Discoverer, Oracle Application Server Personalization, and components in the Portal and Wireless installation type.

  Note: Oracle Application Server ProcessConnect and Oracle Application Server InterConnect have their own installers that are run from their own CD-ROMs.

  See Also: Oracle Application Server 10g Installation Guide for your platform for complete information on Oracle Application Server installation types.

For running J2EE applications, the J2EE and Web Cache installation type provides the core functionality to do so. This includes HTTP services through Oracle HTTP Server. J2EE applications can be enabled with single sign-on functionality if the middle tier is configured to use the Oracle Identity Management framework in the OracleAS Infrastructure.

Note: The Portal and Wireless, and Business Intelligence and Forms installation types always require the Oracle Application Server Metadata Repository and Oracle Identity Management services, because components in these middle tier types need to access their schemas in the Oracle Application Server Metadata Repository.

OracleAS Middle Tier Terminology

The following terms are useful to review before discussing Oracle Application Server middle tier high availability:

• Oracle Application Server Instance: An Oracle Application Server instance (also called an application server instance and OracleAS instance) is the set of processes required to run the configured components within an application server installation. There can be only one application server instance per application server installation. The terms installation and instance are sometimes used interchangeably; however, note that an installation is the set of files installed into an Oracle home, while an instance is a set of processes associated with those files.

• Component Instance: Component instances include a single Oracle HTTP Server process or multiple Oracle Application Server Containers for J2EE (OC4J) instances (see Figure 2-2).

• Oracle Application Server Cluster: An Oracle Application Server Cluster (OracleAS Cluster) is a collection of application server instances with identical configurations and application deployments. Oracle Application Server Clusters enforce homogeneity between instances that are part of the cluster so that the cluster appears and functions as a single instance. With appropriate front-end load balancing, any instance in an Oracle Application Server Cluster can serve client requests. This simplifies configuration and deployment across multiple instances and provides high availability for applications deployed to Oracle Application Server Clusters. Oracle Application Server Clusters can be managed using a database or file-based repository. They can also be manually configured without a repository. See "Oracle Application Server Clusters" for more details.

• OC4J Island: An Oracle Application Server Containers for J2EE (OC4J) island is a group of OC4J processes that replicate session state, for stateful Web applications, among each OC4J process that is part of the island. OC4J processes in an island can be on multiple nodes. When an OC4J island shares the same name across OracleAS instances, session state is replicated across the OracleAS instances.

• Oracle Application Server Farm: A Farm is a collection of application server instances that share the same Oracle Application Server Infrastructure or that use the same application server instance for their file-based repository host. An Oracle Application Server Farm can include application server instances that are in an OracleAS Cluster, as well as those that are not (Figure 2-2).

  See Also: Oracle Application Server 10g Concepts

Services Available

The three installation types mentioned above provide the following services to application clients:

J2EE

J2EE services are provided by OC4J. OC4J is a J2EE-compliant container providing a JSP, Servlet, and EJB runtime environment. OC4J can be clustered together to provide failover and redundancy to clients. See "Oracle Application Server Clusters" and Chapter 4, "Managing and Operating Middle Tier High Availability".

HTTP

Oracle HTTP Server provides HTTP support for Oracle Application Server. It is based on the open source Apache HTTP Server (version 1.3.27) with several standard Apache and OracleAS-specific modules. Oracle HTTP Server is a component of OracleAS Instances in the middle tier. HTTP support for the Infrastructure tier is also provided by Oracle HTTP Server (refer to the section "Oracle HTTP Server").

Additionally, Oracle Application Server Web Cache (OracleAS Web Cache) provides a cache for requested HTTP objects. See the section called "Caching".

See Also: Oracle HTTP Server Administrator's Guide

Portal

OracleAS provides an out-of-the-box enterprise portal that does not require extensive programming and maintenance. You can use Oracle Application Server Portal (OracleAS Portal) and its associated components to build, deploy, and maintain self-service and integrated enterprise portals. OracleAS Portal allows for self-service content management and publishing, wizard-based development, and Web services deployment and publishing in an extensible framework. Oracle Application Server Portal Developer Kit, Oracle Ultra Search, and Oracle Application Server Syndication Services support OracleAS Portal to provide these functionalities. Refer to "Oracle Application Server Portal High Availability" for high availability details.

See Also: Oracle Application Server Portal Configuration Guide

Business Intelligence

Oracle Application Server provides business intelligence services through several components:

• Oracle Application Server Reports Services

  Oracle Application Server Reports Services publish high quality, dynamically generated reports on a scalable, secure platform. These reports can be delivered through Web-browers or non browser interface.

• Oracle Application Server Discoverer

  Oracle Application Server Discoverer enables you to perform dynamic, ad-hoc query reporting and analysis for Web browser delivery.

• Oracle Application Server Personalization

  Oracle Application Server Personalization dynamically serves personalized content recommendations to both registered and anonymous visitors as they browse your site.

Refer to "Business Intelligence High Availability" for a discussion of high availability for these components.

Oracle Application Server Forms Services

Oracle Application Server Forms Services (OracleAS Forms Services) is a comprehensive application framework optimized to deploy OracleAS Forms Services applications in a multi-tiered environment. It is a middle tier application framework for deploying complex, transactional forms applications.

You can build new applications with Forms Developer and deploy them to the Internet with OracleAS Forms Services. Developers can also take current applications that were previously deployed in the legacy client-server model and move them to a three-tier architecture without changing the application code.

At runtime, Oracle Application Server Forms Services has two components: a servlet and a separate runtime process. Refer to "Oracle Application Server Forms Services" for information on how they can be made highly available.

Single Sign-On

Single sign-on service in Oracle Application Server is provided by Oracle Application Server Single Sign-On (OracleAS Single Sign-On), which is part of the Oracle Identity Management framework. This framework allows all applications deployed on Oracle Application Server and its components, such as OracleAS Portal and OracleAS Reports Services, to have a centralized authentication and authorization system for users. Users need only log in once to access any of the applications and resources they are authorized for. The credentials for users are stored in an LDAP version 3-compliant repository (Oracle Internet Directory).

On the middle tier, the Apache module, mod_osso, allows single sign-on requests to be forwarded to the single sign-on server in the OracleAS Infrastructure where the other components of the framework reside. These components are Oracle Internet Directory, and Oracle Application Server Certificate Authority. They are further discussed in Chapter 3, "Infrastructure High Availability".

See Also: Oracle Application Server 10g Concepts for a discussion on Oracle Identity Management.

Caching

Two main caching mechanisms are available in OracleAS:

• OracleAS Web Cache

  OracleAS Web Cache is a HTTP-level cache deployed in front of Oracle HTTP Server. It caches both static (HTML, GIF, and JPEG) and dynamic (generated by servlets and JSPs) content. OracleAS Web Cache can be configured to perform as a load balancer for Oracle HTTP Server instances. Additionally, they can be clustered together to provide failover, redundancy, and improved scalability for cached content. Refer to "Oracle Application Server Web Cache Clusters" for details.

• OC4J Java Object Cache

  Java Object Cache is an in-process caching service for Java application use. It stores frequently accessed or resource-expensive (to create) objects in memory or on disk. Objects can be distributed across OC4J processes that have the same applications deployed in them (for example, OC4J processes in the same OracleAS Cluster). The distributed objects are coordinated and synchronized. Hence, failure of one process does not reduce the availability of cached objects. Java Object Cache enables Oracle Application Server to retrieve content faster and reduce load on Java applications, thereby increasing application availability. See "Oracle Application Server Clusters" for more information.

Features and Components for Middle Tier High Availability

The middle tier architecture involves several features and constructs that allow for high availability. These are:

• Oracle Application Server Instance High Availability

• Oracle Application Server Clusters

• Oracle Application Server Web Cache Clusters

• OC4J Islands

• Other High Availability Components

• High Availability Through Distributed Configuration

• Process Monitoring and Restart

Oracle Application Server Instance High Availability

The Oracle Application Server architecture supports high availability in the middle tier that in many cases can prevent unplanned down time for deployed applications. This section provides an overview of the architecture of an Oracle Application Server instance and shows some of the mid-tier high availability features.

Within each Oracle Application Server instance, the following features provide high availability within the instance, and for any clusters that the instance is a part of:

• Process Monitoring - Using the Oracle Process Management and Notification system provides for process death detection and process restarting in the event that problems are detected for monitored processes.

• Configuration Cloning - Using the Distributed Configuration Management features that uses a Oracle Application Server Metadata Repository for configuration information provides distributed and managed configuration for Oracle Application Server instances and for Oracle Application Server instances that are part of a cluster.

• Data Replication - Using OC4J instances with OC4J islands that provide Web application level stateful session replication, and using EJB sessions, data is replicated across processes within an Oracle Application Server instance and across different Oracle Application Server instances that may reside on different hosts when using Oracle Application Server Clusters. This allows stateful session based applications to remain available even when processes within an Oracle Application Server instance become unavailable or fail.

• Smart Routing - Oracle Application Server Web Cache and Oracle HTTP Server (mod_oc4j) provide configurable and intelligent routing for incoming requests. Requests are routed only to processes and components that mod_oc4J determines to be alive, through communication with the Oracle Process Management and Notification system.

Figure 2-1 shows the architecture of an Oracle Application Server instance, including the features listed above that provide redundant processes and automatic recovery within an instance.

Figure 2-1 Oracle Application Server Instance Architecture

Text description of the illustration instance.gif

Oracle Application Server Clusters

An Oracle Application Server Cluster (OracleAS Cluster) is a set of application server instances configured to act in concert to deliver greater scalability and availability than a single instance. Using OracleAS Clusters removes the single point of failure that a single host poses. While a single application server instance leverages the operating resources of a single host, a cluster can span multiple hosts, distributing application execution over a greater number of CPUs. A single application server instance is vulnerable to the failure of its host and operating system, but a cluster continues to function despite the loss of an operating system or a host, hiding any such failure from clients.

This section covers the following:

• Types of Oracle Application Server Clusters

• Cluster-Wide Configuration for Oracle Application Server Clusters that are Managed Using a Repository

• Requirements for Oracle Application Server Instances to Join Oracle Application Server Clusters that are Managed Using a Repository

• Properties of Oracle Application Server Instances in Oracle Application Server Clusters that are Managed Using a Repository

Types of Oracle Application Server Clusters

There are two types of Oracle Application Server Clusters, Oracle Application Server Clusters managed using a file-based or database repository and Oracle Application Server Clusters that are manually configured:

Note: Only OracleAS instances of the J2EE and Web Cache installation type can be clustered as an OracleAS Cluster.

• Oracle Application Server Clusters managed using a file-based or database repository contain a collection of application server instances with identical configurations and application deployments. Oracle Application Server Clusters enforce homogeneity between instances that are part of the cluster so that the cluster appears and functions as a single instance. Oracle Application Server Clusters that are managed using a repository propagate configuration information across all application server instances in the cluster, which simplifies configuration and deployment.

  There are two types of Oracle Application Server Clusters managed using a reository: Oracle Application Server Clusters managed using a database repository and Oracle Application Server Clusters managed using a file-based repository:

  • Oracle Application Server Clusters managed using a database repository These clusters use a database to store metadata and configuration information. This type of Oracle Application Server Cluster requires the Oracle Application Server Infrastructure since metadata and configuration information is stored in a Oracle Application Server Metadata Repository that resides on an Infrastructure host.

  • Oracle Application Server Clusters managed using a file-based repository These clusters designate an application server instance as the repository host. The repository host uses its file system to store the Oracle Application Server Metadata Repository that retains the metadata and configuration information for the cluster.

  Figure 2-2 shows an example of an Oracle Application Server Cluster managed using a database repository. Figure 2-2 shows three application server instances. All three application server instances share the same Oracle Application Server Metadata Repository. Thus, all three application server instances in the cluster are part of the same Farm.

  Application server instances 1 and 2 are part of an Oracle Application Server Cluster managed using a database repository. In front of the cluster is a front-end load balancer, this may be Oracle Application Server Web Cache or a hardware load balancer appliance. Included within each application server instance are its manageability features--Oracle Process Management and Notification (OPMN) and Distributed Configuration Management (DCM)--and its installed components--Oracle HTTP Server and Oracle Application Server Containers for J2EE (OC4J).

Figure 2-2 OracleAS Cluster Architecture

Text description of the illustration clusarch.gif

• Manually Configured Oracle Application Server Clusters rely on the administrator to manually configure each instance within the cluster (Figure 2-3). With manually configured Oracle Application Server Clusters, it is the administrator's job to make a group of application server instances function as a cluster. Maintaining the configuration and application deployment information on these Oracle Application Server Clusters can be a difficult task. Manually configured Oracle Application Server Clusters provide scalability and availability, but not manageability. The administrator has the responsibility to synchronize the configuration of the application server instances across the cluster.

Figure 2-3 Manually Configured OracleAS Clusters

Text description of the illustration unmanclu.gif

See Also: "High Availability Through Distributed Configuration" "Process Monitoring and Restart" "Using a File-Based Repository with OracleAS Clusters"

Cluster-Wide Configuration for Oracle Application Server Clusters that are Managed Using a Repository

Oracle Application Server Clusters that are managed using a repository contain a collection of application server instances with identical configuration information. Oracle Application Server propagates configuration information across all application server instances that are in a Oracle Application Server Cluster. Each application server instance in a cluster uses the same base configuration. The base configuration is defined by cluster-wide configuration information. When an application server instance joins an Oracle Application Server Cluster, the Distributed Configuration Management system assures that the base configuration is applied to the new instance so that the new instance uses the same cluster-wide configuration.

Using either Application Server Console or dcmctl to deploy an application on an instance, or to modify an application server instance, cluster-wide configuration modifications are propagated to all other application server instances across Oracle Application Server Clusters.

Cluster-wide configuration excludes certain instance-specific parameters. The instance-specific parameters are not propagated to all of the application server instances across a cluster. If you modify an instance-specific parameter it is not propagated, as it is only applicable to the specified application server instance where the change is made.

See Also: "Cluster-Wide Configuration Changes and Modifying OC4J Instances"

Requirements for Oracle Application Server Instances to Join Oracle Application Server Clusters that are Managed Using a Repository

In order for an application server instance to join Oracle Application Server Clusters, the application server instance must be clusterable. For an application server instance to be clusterable, the following must be true:

1. The application server instance must be part of the Farm where the Oracle Application Server Cluster resides. You can associate application server instances with a OracleAS Metadata Repository either during installation time or after installation using Application Server Console.

2. Each application server instance in a cluster must be installed on the same type of operating system, such as UNIX.

3. Each application server instance can contain only one Oracle HTTP Server.

4. Each application server instance can contain one or more OC4J instances.

   See Also: "Adding an Application Server Instance to an OracleAS Cluster"

Properties of Oracle Application Server Instances in Oracle Application Server Clusters that are Managed Using a Repository

Once application server instances join a cluster, they have the following properties:

• Each application server instance uses the same cluster-wide configuration. That is, if you modify any cluster-wide parameters, the modifications are propagated to all application server instances in the cluster.

• If you deploy an application to one application server instance, it is propagated to all application server instances in the cluster. The application is actually deployed to an OC4J instance in the application server instance and propagated to the same OC4J instance in the other application server instances in the cluster. You can change some of the configuration for the deployed application, and this change is propagated to the same OC4J instance in the other application server instances across the cluster.

• Most clustering management, configuration, and application deployment is handled through Oracle Enterprise Manager. If you want to use a command-line tool, you can use the Distributed Configuration Management command-line tool dcmctl.

• The base configuration is created from the first application server instance to join a cluster.

• You can remove application server instances from the cluster. The application server instance is stopped when removed from the cluster. When the last application server instance is removed, the cluster still remains. You must delete the cluster itself for it to be removed.

  See Also: "Managing Application Server Instances in an OracleAS Cluster" Distributed Configuration Management Reference Guide for information on dcmctl commands

Oracle Application Server Web Cache Clusters

Two or more OracleAS Web Cache instances can be clustered together to create a single logical cache. Physically, the cache can be distributed amongst several nodes. If one node fails, a remaining node in the same cluster can fulfill the requests serviced by the failed node. The failure is detected by the remaining nodes in the cluster who take over ownership of the cacheable content of the failed member. The load balancing mechanism in front of the OracleAS Web Cache cluster, for example, a hardware load balancing appliance, redirects the requests to the live OracleAS Web Cache nodes.

OracleAS Web Cache clusters also add to the availability of OracleAS instances. By caching static and dynamic content in front of the OracleAS instances, requests can be serviced by OracleAS Web Cache reducing the need for the requests to be fulfilled by OracleAS instances, particularly for Oracle HTTP Servers. The load and stress on OracleAS instances is reduced, thereby increasing availability of the components in the instances.

Oracle Application Server Web Cache can also perform a stateless or stateful load balancing role for Oracle HTTP Servers. Load balancing is done based on the percentage of the available capacity of each Oracle HTTP Server, or, in other words, the weighted available capacity of each Oracle HTTP Server. If the weighted available capacity is equal for several Oracle HTTP Servers, OracleAS Web Cache uses round robin to distribute the load. Refer to Oracle Application Server Web Cache Administrator's Guide for the formula to calculate weighted available capacity.

In the case of failure of a Oracle HTTP Server, OracleAS Web Cache redistributes the load to the remaining Oracle HTTP Servers and polls the failed server intermittently until it comes back online. Thereafter, OracleAS Web Cache recalculates the load distribution with the revived Oracle HTTP Server in scope.

See Also: "HTTP Service High Availability" Oracle Application Server Web Cache Administrator's Guide

OC4J Islands

Oracle Application Server provides several strategies for ensuring high availability with OC4J instances, both within an application server instance and across a cluster that includes multiple application server instances.

This section covers the following:

• Web Application Session State Replication with OC4J Islands

• Stateful Session EJB High Availability Using EJB Clustering

• JNDI Namespace Replication

• OC4J Distributed Caching Using Java Object Cache

Besides the high availability features described in this section, other Oracle Application Server features enable OC4J processes to be highly available, including the load balancing feature in Oracle HTTP Server and the Oracle Process Management and Notification system that automatically monitors and restarts processes.

See Also: "HTTP Service High Availability" "Process Monitoring and Restart"

Web Application Session State Replication with OC4J Islands

When a stateful Web application is deployed to OC4J, multiple HTTP requests from the same client may need to access the application. However, if the application running on the OC4J server experiences a problem where the OC4J process fails, the state associated with a client request may be lost. Using Oracle Application Server, there are three ways to guard against such failures:

• State safe applications save their state in a database or other persistent storage system, avoiding the loss of state when the server goes down. Obviously, there is a performance cost for continually writing the application state to persistent storage.

• Stateless applications do not have a state that needs to be carried between requests, and so, stateless applications do not have state integrity considerations when a server goes down. Another active server can handle the request. High availability for stateless applications is easier to achieve than for state safe or stateful applications.

• Stateful applications can use OC4J session state replication, with OC4J islands, to automatically replicate the session state across multiple processes in an application server instance, and in a cluster, across multiple application instances which may run on different nodes.

OC4J processes can be grouped into islands to support session state replication for high availability of Web applications. Using OC4J islands together with Oracle HTTP Server mod_oc4j request routing provides stateful failover in the event of a software or hardware problem. For example, if an OC4J process that is part of an island fails, mod_oc4j is notified of the failure by OPMN and routes requests to another OC4J process in the same island.

Web Application Session State Protecting Against Software Problems

To guard against software problems, such as OC4J process failure or hang, you can configure an OC4J instance to run multiple OC4J processes in the same OC4J island. The processes in the OC4J island communicate their session state between each other. Using this configuration provides failover and high availability by replicating state across multiple OC4J processes running on an application server instance.

In the event of a failure, Oracle HTTP Server forwards requests to active (alive) OC4J process within the OC4J island. In this case, the Web application state for the client is preserved and the client does not notice any loss of service.

Figure 2-4 shows this type of software failure within an application server instance.

Figure 2-4 Web Application Session State Failover Within An OC4J Island in an OC4J Instance

Text description of the illustration failinst.gif

Web Application Session State Replication Protecting Against Hardware Problems

To guard against hardware problems, such as the failure of the node where an application server instance runs, you can configure OC4J islands across application server instances that are in more than one node in an OracleAS Cluster. By configuring an OC4J island that uses the same name across multiple application server instances, the OC4J processes can share session state information across the OracleAS Cluster. When an application server instance fails or is not available, for example, when the node it runs on goes down, Oracle HTTP Server forwards requests to an OC4J process in an application server instance that is available. Thus, Oracle HTTP Server forwards requests only to active (alive) OC4J processes within the cluster.

In this case, the Web application state for the client is preserved and the client does not notice any irregularity.

Figure 2-5 depicts an OC4J island configured within an OracleAS Cluster.

Figure 2-5 Web Application Session State Failover Within An OracleAS Cluster

Text description of the illustration failclus.gif

Configuring OC4J Islands With High Availability

To protect against software or hardware failure while maintaining state with the least number of OC4J processes, you need to configure at least two OC4J processes in the same island on multiple application server instances running on separate nodes. For example, if you have two application server instances, instance 1 and instance 2, you can configure two OC4J processes in the default_island on each application server instance. With this configuration, stateful session applications are protected against hardware and software failures, and the client maintains state if either of the following types of failures occurs:

• If one of the OC4J processes fails, then the client request is redirected to the other OC4J process in the default_island on the same application server instance. State is preserved and the client does not notice any irregularity.

• If application server instance 1 terminates abnormally, then the client is redirected to the OC4J process in the default_island on application server instance 2. The state is preserved and the client does not notice any irregularity.

  See Also: "Configuring OC4J Islands and OC4J Processes"

Stateful Session EJB High Availability Using EJB Clustering

Using OC4J, stateful session EJBs can be configured to provide state replication across OC4J processes running within an application server instance or across an OracleAS Cluster. This EJB replication configuration provides high availability for stateful session EJBs by using multiple OC4J processes to run instances of the same stateful session EJB.

Note: Use of EJB replication (EJB clusters) for high availability is independent of OracleAS Clusters and can involve multiple application server instances installed across nodes that are or are not part of OracleAS Clusters.

EJB clusters provide high availability for stateful session EJBs. They allow for failover of these EJBs across multiple OC4J processes that communicate over the same multicast address. Thus, when stateful session EJBs use replication, this can protect against process and node failures and can provide for high availability of stateful session EJBs running on Oracle Application Server.

See Also: "Configuring EJB Application State Replication for OracleAS Clusters" Oracle Application Server Containers for J2EE User's Guide Oracle Application Server Containers for J2EE Enterprise JavaBeans Developer's Guide

JNDI Namespace Replication

When EJB clustering is enabled, JNDI namespace replication is also enabled between the OC4J instances in an OracleAS Cluster. New bindings to the JNDI namespace in one OC4J instance are propagated to other OC4J instances in the OracleAS Cluster. Re-bindings and unbindings are not replicated.

The replication is done outside the scope of OC4J islands. In other words, multiple islands in an OC4J instance have visibility into the same replicated JNDI namespace.

See Also: Oracle Application Server Containers for J2EE Services Guide

OC4J Distributed Caching Using Java Object Cache

Oracle Application Server Java Object Cache provides a distributed cache that can serve as a high availability solution for applications deployed to OC4J. The Java Object Cache is an in-process cache of Java objects that can be used on any Java platform by any Java application. It allows applications to share objects across requests and across users, and coordinates the life cycle of the objects across processes.

Java Object Cache enables data replication among OC4J processes even if they do not belong to the same OC4J island, application server instance, or Oracle Application Server Cluster.

By using Java Object Cache, performance can be improved since shared Java objects are cached locally, regardless of which application produces the objects. This also improves availability; in the event that the source for an object becomes unavailable, the locally cached version will still be available.

See Also: The Java Object Cache chapter in the Oracle Application Server Web Services Developer's Guide for complete information on using Java Object Cache

Process Monitoring and Restart

In an application server instance and across OracleAS Clusters, Oracle Process Management and Notification (OPMN) monitors Oracle Application Server components, including Oracle HTTP Server, OC4J, OracleAS Web Cache, and Oracle Application Server Reports Services (OracleAS Reports Services).

The OPMN system, which is itself an Oracle Application Server component, assists in making Oracle Application Server highly available by monitoring and automatically restarting Oracle Application Server processes that fail. When a process becomes unavailable, OPMN notifies certain other Oracle Application Server components that the process is unavailable. For example, in an OracleAS Cluster, when an OC4J process fails, the Oracle HTTP Server mod_oc4j modules are notified of the failure and do not send requests to the failed OC4J process until OPMN uses an event to notify the modules that the OC4J process has been restarted.

OPMN consists of the following sub-components:

• Oracle Process Manager

• Oracle Notification System

Oracle Process Manager

The Oracle Process Manager is responsible for starting, restarting, shutting down, and detecting the death of Oracle Application Server processes. The Oracle Process Manager can start or stop processes through one of the following ways:

• directives in the opmn.xml configuration file

• by selecting Application Server Console operations, such as start or stop for components such as OC4J instances

• by using the opmnctl command line utility.

Oracle Notification System

The Oracle Notification System is the communication mechanism for failure, recovery, startup, and other related notifications between components. The notification system operates according to a subscriber-publisher model, wherein any component that wishes to receive an event of a certain type subscribes to the Oracle Notification System. When such an event is published, the Oracle Notification System sends it to all relevant subscribers.

In an Oracle Application Server Cluster, the Oracle HTTP Servers communicate using the Oracle Notfication System and are aware of the active OC4J processes across the Oracle Application Server Cluster. This communication mechanism enables each Oracle HTTP Server to know the live OC4J processes in the Oracle Application Server Cluster so that incoming requests can be load balanced to them.

See Also: Oracle Process Manager and Notification Server Administrator's Guide

High Availability Through Distributed Configuration

Oracle Application Server uses the Distributed Configuration Management (DCM) system to manage the cluster-wide configuration in Oracle Application Server clusters. DCM provides supports the following actions:

• When new application server instances join a cluster, DCM automatically replicates the base configuration to all instances in the cluster.

• DCM propagates application deployments and configuration changes to all application server instances across a cluster.

For Oracle Application Server high availability, when a system in an Oracle Application Server cluster is down, there is no single point of failure for DCM. DCM remains available on all the available nodes in the cluster.

Using DCM helps reduce deployment and configuration errors in a cluster; these errors could, without using DCM, be a significant cause of system downtime.

Enterprise Manager uses DCM commands to perform application server configuration and deployment. You can also issue DCM commands manually using the dcmctl command.

DCM controls the following configuration commands

• Create or remove a cluster

• Add or remove application server instances to or from a cluster

• Synchronize configuration changes across application server instances

Note the following when making configuration changes to a cluster or deploying applications to a cluster:

• If Enterprise Manager is up and managing the cluster, you can invoke the DCM command-line tool from any host where a clustered application server instance is running. DCM informs Enterprise Manager of the requested function and Enterprise Manager then interfaces with the other DCM management features on the other application server instances in the cluster to complete the cluster-wide configuration or application deployment.

• If Enterprise Manager is not up and managing the cluster, if you want configuration changes to by applied dynamically across the cluster, the DCM daemon must be running on each cluster. To start the DCM daemon, run the DCM command-line tool, dcmctl, on each application server instance in the cluster.

  See Also: Distributed Configuration Management Reference Guide

Other High Availability Components

Several external components can be used to improve the availability of Oracle Application Server. These components are discussed in the following sections:

• Improving Availability with an External Load Balancer

• Improving Availability with Operating System Clusters

Improving Availability with an External Load Balancer

You can use an external load balancer to improve the availability of both clustered and non-clustered Oracle Application Server instances.

Clients access the cluster through a load balancer, which hides the cluster configuration. The load balancer can send requests to any application server instance in the cluster, as any instance can service any request. An administrator can raise the capacity of the system by introducing additional application server instances to the cluster. These instances can be installed on multiple nodes to allow for redundancy in case of node failure.

You can also use a load balancer to increase the availability of non-clusterable Oracle Application Server instances, such as Portal and Wireless, when they are installed on multiple nodes. As long as the load balancer is configured to serve a set of nodes, it will route requests accordingly.

Types of External Load Balancers

There are three types of load balancers you can use with Oracle Application Server instances: hardware load balancers and network load balancers. Table 2-4 summaries these:

Table 2-1 Types of External Load Balancers Summary 

Load Balancer Type	Description
Hardware Load Balancer	Hardware load balancing involves placing a hardware load balancer, such as Big-IP or Alteon, in front of a group of Oracle Application Server instances or OracleAS Web Cache. The hardware load balancer routes requests to the Oracle HTTP Server or OracleAS Web Cache instances in a client-transparent fashion.
Windows Network Load Balancer (applicable to Windows version of Oracle Application Server)	With some Windows operating systems, you can use the operating system to perform network load balancing. For example, with Microsoft Advanced Server, the NLB functionality allows you to send requests to different machines that share the same virtual IP or MAC address. The servers themselves to do not need to be clustered at the operating system level.

Note: Check http://metalink.oracle.com for information on supported external load balancers.

High Availablity Benefits of External Load Balancing

There are three main benefits of using clusters: scalability, availability, and manageability. Load balancing improves scalability by providing an access point through which requests are routed to one of many available instances. Instances can be added to the group that the load balancer serves to accomodate additional users.

Load balancing improves availability by routing requests to the most available instances. If one instance goes down, or is particularly busy, the load balancer can send requests to another active instance.

Load balancing improves the system manageability by routing application deployment and system configuration requests to the most available instances. If one instance goes down, or is particularly busy, the load balancer can send requests to another active instance.

Improving Availability with Operating System Clusters

Using operating sytem clusters involves installing Oracle Application Server on a hardware cluster created through the operating system or other clustering system solutions, such as HP MC Service Guard. Operating system clustering is supported in Oracle Application Server 10g (9.0.4).

HTTP Service High Availability

Oracle HTTP Server and Oracle Application Server Web Cache provide HTTP and HTTPS request handling for Oracle Application Server requests. Each HTTP request is met by a response from Oracle HTTP Server or from Web Cache if the content requested is cached.

This section covers the following topics:

• Web Cache and Oracle HTTP Server High Availability Summary

• OC4J Load Balancing Using mod_oc4j

Web Cache and Oracle HTTP Server High Availability Summary

Table 2-2 summarizes some of the Oracle Application Server high availability features for the Oracle HTTP Server and OracleAS Web Cache components.

Table 2-2 Oracle HTTP Server and OracleAS Web Cache high availability characteristics 

Component	Protection from Node Failure	Protection from Service Failure	Protection from Process Failure	Automatic Re-routing	State Replication	Configuation Cloning
OracleAS Web Cache	OracleAS Web Cache cluster protects from single point of failure. An external load balancer should be deployed in front of this cluster to route requests to live OracleAS Web Cache nodes.	In an OracleAS Web Cache cluster, pings are made to a specific URL in each cluster member to ensure that the URL is still serviceable.	OPMN monitors OracleAS Web Cache processes and restarts them upon process failure	OracleAS Web Cache members in a cluster ping each other to verify that peer members are alive or have failed.	OracleAS Web Cache clustering manages replicated objects	OracleAS Web Cache cluster maintains uniform configuration across cluster
Oracle HTTP Server	OracleAS Cluster protects from single point of failure. A load balancer should be deployed in front of Oracle HTTP Server instances. This can be an external load balancer or OracleAS Web Cache.	Load balancer in front of Oracle HTTP Server sends request to another Oracle HTTP Server if first one doesn't respond or is deemed failed through URL pings. Load balancer can be either OracleAS Web Cache or hardware appliance.	OPMN monitors Oracle HTTP Server processes and restarts them upon process failure. Each Oracle HTTP Server is also notified by OPMN when another Oracle HTTP Server process in the OracleAS Cluster fails.	Load balancer in front of Oracle HTTP Server auto re-routes to another Oracle HTTP Server if first does not respond.	None.	OracleAS Cluster allows configuration to be replicated across to other Oracle HTTP Servers in the cluster through DCM.

.

See Also: "Oracle Application Server Web Cache Clusters" "Process Monitoring and Restart" "Types of External Load Balancers"

OC4J Load Balancing Using mod_oc4j

The Oracle HTTP Server module, mod_oc4j provides intelligent routing for HTTP requests that are handled by OC4J. The intelligent routing that mod_oc4j provides is an import Oracle Application Server high availability feature. If an OC4J process fails Oracle Process Management and Notification detects the failure and mod_oc4j does not send requests to the failed OC4J process until the OC4J process is restarted.

Generally, mod_oc4j deals with stateless HTTP requests, since stateful HTTP requests are forwarded to the OC4J process that served the previous request (unless mod_oc4j determines, through communication with Oracle Process Management and Notification that the process is not available, in which case mod_oc4j forwards the request to an available OC4J).

Using mod_oc4j configuration options you can specify different load balancing routing algorithms, depending on the type and complexity of routing you need.

Table 2-3 summarizes the routing styles that mod_oc4j provides. For each routing style, Table 2-3 lists the different algorithms that you can configure to modify the routing behavior. These mod_oc4j configuration options determine the OC4J process where mod_oc4j sends incoming HTTP requests to be handled.

See Also: "mod_oc4j Load Balancing With OracleAS Clusters" Oracle HTTP Server Administrator's Guide for information on using weighted routing and selecting local affinity with mod_oc4j load balancing options.

Table 2-3 mod_oc4j Routing Algorithms Summary 

Routing Method	Description
Round Robin	Using the simple round robin configuration, all OC4J processes, remote and local to the application server instance running the Oracle HTTP Server, are placed in an ordered list. Oracle HTTP Server then chooses an OC4J process at random for the first request. For each subsequent request, Oracle HTTP Server forwards requests to another OC4J process in round robin style. The round robin configuration supports local affinity and weighted routing options.
Random	Using the simple random configuration, all OC4J processes, remote and local to the application server instance running the Oracle HTTP Server, are placed in an ordered list. For every request, Oracle HTTP Server chooses an OC4J process at random and forwards the request to that instance. The random configuration supports local affinity and weighted routing options.
Metric-Based	Using the metric-based configuration OC4J processes, remote and local to the application server instance running the Oracle HTTP Server, are placed into an ordered list. OC4J processes then regularly communicate to Oracle HTTP Server how busy they are and Oracle HTTP Server uses this information to send requests to the OC4J processes that are less busy. The metric-based configuration supports a local affinity option.

OC4J Load Balancing Using Local Afinity and Weighted Routing Options

Using mod_oc4j options, you can select a routing method for routing OC4J requests. If you select either round robin or random routing, you can also use local affinity or weighted routing options. If you select metric-based routing, you can also use the local affinty option.

Using the weighted routing option, a weight is associated with OC4J processes on a node, as configured in mod_oc4j, on a node by node basis. During request routing, mod_oc4j then uses the routing weight to calculate which OC4J process to assign requests to. Thus, OC4J processes running on different nodes can be assigned different weights.

Using the local affinity option, mod_oc4j keeps two lists of available OC4J processes to handle requests, a local list and a remote list. If processes are available from the local list then requests are assigned locally using the random routing method or, for metric-based routing using metric-based routing. If no processes are available in the local list, then mod_oc4j selcts processes randomly from the remote list when random method, using a round robin method for the round robin method, or using metric-based routing with the metric-based method.

Choosing a mod_oc4j Routing Algorithm

Table 2-3 summarizes the available routing options. To select a routing algorithm to configure with mod_oc4j, you need to consider the type of environment where Oracle HTTP Server runs. Use the following guidelines to help determine which configuration options to use with mod_oc4j:

• For a Oracle Application Server cluster setup, with multiple identical machines running Oracle HTTP Server and OC4J, the round robin with local affinity algorithm is preferred. Using this configuration, an external router distributes requests to multiple machines running Oracle HTTP Server and OC4J. In this case Oracle HTTP Server gains little by using mod_oc4j to route requests to other machines, except in the extreme case that all OC4J processes on the same machine are not available.

• For a tiered deployment, where one tier of machines contains Oracle HTTP Server and another contains OC4J instances that handle requests, the preferred algorithms are simple round robin and simple metric-based. To determine which of these two is best in a specific setup, you may need to experiment with each and compare the results. This is required because the results are dependent on system behavior and incoming request distribution.

• For a heterogeneous deployment, where the different application server instances run on nodes that have different characteristics, using the weighted round robin algorithm is preferred. Tune the number of OC4J processes running on each application server instance may allow you to achieve the maximum benefit. For example, a machine with a weight of 4 gets 4 times as many requests as a machine with a weight of 1, but if the system with a weight of 4 may not be running 4 times as many OC4J processes.

  See Also: "mod_oc4j Load Balancing With OracleAS Clusters" Oracle HTTP Server Administrator's Guide for information on using weighted routing and selecting local affinity with mod_oc4j load balancing options.

J2EE High Availability

J2EE requests are fulfilled by OC4J, and involve OracleAS Web Cache and Oracle HTTP Server (mod_oc4j). Hence, the high availability of the J2EE service requires that these components are highly available. The high availability of Oracle HTTP Server and OracleAS Web Cache is discussed in the section "HTTP Service High Availability". The high availability of OC4J is presented in the section "Features and Components for Middle Tier High Availability".

EJB Client Routing

In EJB client routing, EJB classes take on the routing functionality that mod_oc4j provides for Oracle HTTP Server. Using the Active Components for Java (AC4J) architecture, EJBs can interact in a loosely-coupled fashion. This provides support for reliable asynchronous, disconnected, one-way request and response interactions, without the complexity of JMS programming. It automatically routes service requests to the appropriate service provider, and provides automatic security context propagation, authorization and identity impersonation. It also provides automatic exception routing and handling, which is integrated into the EJB framework.

Oracle Application Server Portal High Availability

An OracleAS Portal request's lifecycle is serviced by a number of OracleAS components. These are:

• OracleAS Web Cache

• Oracle HTTP Server and the following modules:
  • mod_oc4j (on middle and Infrastructure tiers)

  • mod_osso (on Infrastructure tier to access OracleAS Single Sign-On)

  • mod_plsql (on middle tier with OracleAS Portal DAD and Infrastructure tier with ORASSO DAD)

  • mod_oradav (on middle tier)

• OC4J (the Portal Page Engine runs as a stateless servlet)

• OracleAS Portal repository (contains OracleAS Portal schemas and also caches group memberships of users after their retrieval from Oracle Internet Directory)

• OracleAS Single Sign-On

• Oracle Internet Directory (including Oracle Delegated Administration Services and Oracle Directory Integration and Provisioning)

• Various web and database portlet providers

In order for OracleAS Portal to be highly available, all these components must be highly available individually. Of particular importance is the availability of Oracle Identity Management because OracleAS Portal uses it for portlet security and management functions.

Reference the following table to find out where you can find high availability information for each of the components mentioned above.

Table 2-4 High availability information for components involved in an OracleAS Portal request 

Component	Where to find information
OracleAS Web Cache	See "Oracle Application Server Web Cache Clusters".
Oracle HTTP Server	See: "Oracle Application Server Clusters" "HTTP Service High Availability"
OC4J	See: "J2EE High Availability" Note: The Portal Page Engine is stateless.
OracleAS Portal repository	See: Chapter 3, "Infrastructure High Availability" and Chapter 5, "Managing Infrastructure High Availability" of this book. Oracle Application Server Portal Configuration Guide
OracleAS Single Sign-On	See: Chapter 3, "Infrastructure High Availability" and Chapter 5, "Managing Infrastructure High Availability" of this book. Oracle Identity Management Concepts and Deployment Planning Guide
Oracle Internet Directory	See: Chapter 3, "Infrastructure High Availability" and Chapter 5, "Managing Infrastructure High Availability" of this book. Oracle Internet Directory Administrator's Guide Oracle Identity Management Concepts and Deployment Planning Guide
Web Provider	See: "Oracle Application Server Clusters" "HTTP Service High Availability" "OC4J Islands" "Stateful Session EJB High Availability Using EJB Clustering" "Oracle Application Server Web Cache Clusters" (OracleAS Web Cache could be providing access to the provider)
Database Providers	For providers using mod_plsql, Oracle HTTP Server high availability is relevant. See "HTTP Service High Availability" and "Oracle Application Server Clusters". For database high availability, see: "Oracle Application Server Cold Failover Cluster" "Oracle Application Server Active Failover Cluster (Limited Release)" See also: "Oracle Application Server Web Cache Clusters" (OracleAS Web Cache could be providing access to the provider)

See Also: Oracle Application Server Portal Configuration Guide

Oracle Application Server Wireless High Availability

Typical Oracle Application Server Wireless (OracleAS Wireless) deployments in the enterprises, and particularly in telecom operator infrastructure, have very high availability and fault tolerance requirements. Oracle Application Server provides a framework and the mechanism to address these requirements.

OracleAS Wireless is integrated with this framework to extend these features to wireless deployments. Since OracleAS Wireless components are deloyed as OC4J applications, Oracle HTTP Server can be configured to provide high availability to OracleAS Wireless applications. In addition, the OracleAS Wireless runtime is designed to handle session state replication so that client sessions failover transparently among multiple OC4J containers. Typical high availability deployments involve several network components, which in turn lead to considerations of configuration topology, performance, and security.

Refer to the "Wireless Gateway Configuration" chapter and the "Load Balancing and Failover" chapters of the Oracle Application Server Wireless Administrator's Guide for details.

Business Intelligence High Availability

The business intelligence components in Oracle Application Server include Oracle Application Server Reports Services and Oracle Application Server Discoverer. The following sections discuss the high availability of each:

• Oracle Application Server Reports Services High Availability

• Oracle Application Server Discoverer High Availability

Oracle Application Server Reports Services High Availability

At runtime, Oracle Application Server Reports Services (OracleAS Reports Services) consists of the following components shown in Table 2-5.

Table 2-5 Oracle Application Server Reports Services runtime components 

Component	Characteristics
Reports Servlet	Translates client requests between HTTP and the Reports Server. It runs in-process in OC4J, and hence, is subject to the failures and high availability solutions for OC4J.
Reports Server	Processes client requests and forwards them to the Reports Engine. It runs as a standalone process and is stateful. Its state is not replicated to other Reports Server processes.
Reports Engine	Fetches requested data from data sources, formats the reports, and notifies the Reports Server that jobs are complete. It runs as a separate process from the Reports Server but is spawned by the latter to service requets. The Reports Engine is stateless. Failure of a Reports Engine process has minimal impact on the overall Reports Services as the Reports Server can spawn new Engine processes.

Of the three components described in the table above, the Reports Server process is the critical process that will adversely affect availability of OracleAS Reports Services if it fails. As it maintains state for client requests and the state is not replicated, its failure will cause client sessions to be lost.

Reports Server processes are monitored by OPMN. When OracleAS Reports Services is installed, it is registered with OPMN by default. Hence, OPMN can monitor and restart a Reports Server process if it fails. If you add more Reports Server processes after installation, you need to add them to the opmn.xml and targets.xml (for Application Server Console). See the book Oracle Application Server Reports Services Publishing Reports to the Web for instructions as well as for more high availability information.

The Reports Server makes database connections to the OracleAS Portal repository in the OracleAS Infrastructure as well as to Oracle Internet Directory. If any of these connections fail, Reports Server retries them before throwing an exception. If a successful connection is made, Reports Server need not be restarted. This also applies to Reports Servlet, which makes connections to Oracle Internet Directory.

High Availability Solution

To eliminate the single point of failure of the Reports Server process, perform multiple installations of OracleAS Reports Services (Business Intelligence and Forms Installation Type). These installations should also be installed on multiple nodes to protect from node failure.

For the OracleAS Infrastructure that is used by the OracleAS Reports Services installations, inclusive of Oracle Identity Management, use the Oracle Application Server Cold Failover Cluster or OracleAS Active Failover Cluster high availability solutions. These are described in Chapter 3, "Infrastructure High Availability".

See Also: Oracle Application Server Reports Services Publishing Reports to the Web ("Clustering Reports Servers" chapter)

Oracle Application Server Discoverer High Availability

Oracle Application Server Discoverer achieves high availability in the following ways:

• Process monitoring and restart

  OPMN is configured to monitor and restart Oracle Application Server Discoverer processes on each middle tier node. See Chapter 4 of Oracle Application Server Discoverer Configuration Guide.

• Load balancing

  OracleAS Web Cache can be set up to perform as a load balancer for Oracle Application Server Discoverer requests. See Chapter 5 of Oracle Application Server Discoverer Configuration Guide.

Oracle Application Server Forms Services High Availability

At runtime, Forms Services consist of the components listed in Table 2-6.

Table 2-6 Runtime Forms Services components 

Component	Function
Forms Servlet	The Forms Servlet handles the initial application request and dynamically generates the start HTML file for the Forms generic Java Applet. If using OracleAS Single Sign-On, the Forms Servlet is also used to verify users' authentication.
Forms Listener Servlet	The Forms Listener Servlet is a dispatcher servlet that handles the communication between the Forms Java client in the client browser and the Forms runtime process in the middle tier server. The Forms Listener Servlet starts a Forms runtime process for each application request and user.
Forms Runtime Engine	The Forms Runtime Engine interprets the Forms application modules (fmx files) and executes the contained business logic. The Forms Runtime Engine also makes the database connection using SQLNet.

Forms Services doesnt exist as a dedicated server process on the middle tier, and therefore, all that is required to request and run a Forms application on the Web is the availability of a servlet container (OC4J) that is configured to run Forms Services.

Because Forms Services launches a dedicated Forms Runtime process for each user there is no transparent application failover. Once a user session is interrupted, the user has to restart the Forms Web application by issuing a new request to the Forms Servlet.

If a middle tier server crashes or a servlet session is interrupted, recover from either failure by restarting the application. To set up high availability for Forms, the following components can be used:

mod_oc4j - Handling the failure of an OC4J instance, Forms can be setup to load balance application requests between different OC4J instances. This ensures that an application request can be routed to the next available OC4J instance if the current OC4J instance fails.

OracleAS Web Cache - Using OracleAS Web Cache as a HTTP load balancer allows you to distribute Forms requests between many Oracle Application Server instances that may or may not share the same Infrastructure installation. If one instance fails, then the next Forms application request gets routed to the next available instance. Each instance can also use mod_oc4j to load balance Forms sessions between OC4J instances.

Hardware load balancers - A hardware load balancer can be deployed in front of OracleAS Web Cache, thereby adding one more layer of load balancing for Forms requests. Or, they can also replace OracleAS Web Cache and load balance requests directly to Oracle HTTP Servers.

For the OracleAS Infrastructure that is used by Forms Services installations, inclusive of Oracle Identity Management, use the Oracle Application Server Cold Failover Cluster or Oracle Application Server Active Failover Cluster high availability solutions. These are described in Chapter 3, "Infrastructure High Availability".

For more information about Forms Services architecture and setup, refer to Oracle Application Server Forms Services Deployment Guide.

Oracle Application Server Integration High Availability

High availability for the Oracle Application Server e-business integration products, Oracle Application Server InterConnect and Oracle Application Server ProcessConnect, are dependent on the various high availability solutions for the Infrastructure (see the section "High Availability Configurations for Infrastructure"). This is because InterConnect and ProcessConnect utilize the database in the Infrastructure to store and queue information. However, not all high availability solutions for the Infrastructure can be used by InterConnect or ProcessConnect. The following points elaborate further:

• Oracle Application Server InterConnect

  High availability is supported by the Oracle Application Server Cold Failover Cluster and Oracle Application Server Active Failover Cluster solutions. The adapter.ini and hub.ini files are populated with the host, port, and instance information of all the nodes in the cluster. When a node or database instance failure occurs, the InterConnect adapters are able to reconnect and continue message delivery without the need to republish the message and without losing the message at any stage of processing and delivery.

  Detailed information on the adapter.ini and hub.ini contents can be found in chapter 8, under the section "RAC Support," in the Oracle Application Server InterConnect User's Guide.

• Oracle Application Server ProcessConnect

  ProcessConnect uses the database in the Infrastructure to store connection information instead of storing the information in files as for InterConnect. Hence, both the Oracle Application Server Cold Failover Cluster and Oracle Application Server Active Failover Cluster solutions for the Infrastructure enable high availability for ProcessConnect.

Middle Tier Recovery Solutions

Once a failure has occurred in your system, it is important to recover from that failure as quickly as possible. There are four main types of recovery solutions that you can use, depending on the type and severity of the failure.

Restarting Processes

Recovering from almost all types of failures requires restarting one or more failed processes in your system. There are three process restart scenarios:

• Automatic restart of processes: The failed processes are automatically restarted by OPMN upon detected failure. No manual intevention is required.

• Manually restart an individual process: This implies that the process failure does not affect any other middle tier or Infrastructure processes, and can be restarted individually.

• Manually restart all processes.

Most types of failures in both the middle tier and Infrastructure only require a process restart solution. Such failures include the death of OPMN, an Oracle Application Server Metadata Repository failure, or an Application Server Console crash.

Restoring from Cold Backup

Some failures require more involved recovery scenarios than simply restarting processes. In some cases, you will have to perform restoration operations based on cold backup procedures that you had previously implemented. These cold backups include installed OracleAS binaries. Cold backup restoration operations can be done for both the middle tier and the Infrastructure.

• Middle tier restoration from cold backup - Restoration of the entire Oracle Application Server middle tier, including the Oracle Home, configuration files, and database files, which were backed up after completing a clean and normal shutdown of all Oracle Application Server Infrastructure processes and the Oracle Application Server Metadata Repository.

• Infrastructure restoration from cold backup - Restoration of the entire Oracle Application Server Infrastructure instance, including the Oracle Home, configuration files, and data base files, which were backed up after completing a clean and normal shutdown of all Oracle Application Server Infrastructure processes and the Oracle Application Server Metadata Repository.

Failures that require the restoring from cold backup solution for recovery include node failure where the node needs to be completely replaced, and the deletion or corruption of Oracle software or binary files. Failures that require this type of recovery solution also then require the manual restart of all processes. For details about specific failure types and how to recover, see the Oracle Application Server 10g Administrator's Guide

Restoring from Online Backup

Depending on the type of failure your system is experiencing, you may need to restore your system from an online backup. There are four types of online backup restoration scenarios:

• Middle tier restoration from online backup - Restoration of the Oracle Application Server configuration files, which were backed up while processes were up and running on the middle tier. This also includes restoring a stamped image, which may require additional steps to complete the restoration.

• Infrastructure restoration from online backup - Restoration of the Oracle Application Server Infrastructure configuration files, which were backed up after completing a proper online backup of the Oracle Application Server instance and Oracle Application Server Metadata Repository.

• Oracle Application Server Metadata Repository restoration from online backup - Restoration of the Oracle Application Server Metadata Repository taken from a proper online backup. Complete recovery is required of the database component.

• Infrastructure configuration files restoration from online backup - Restoration of the Infrastructure configuration files taken from an online backup.

Failures that require restoration from online backup solutions for recovery include data failure in the Oracle Application Server Metadata Repository, and deletion or corruption of Oracle Application Server component runtime configuration files. Failures that require this type of solutions also then require one or more processes to be restarted. For details about specific failure types and how to recover, see the Oracle Application Server 10g Administrator's Guide.

Disaster Recovery

Disaster recovery (DR) refers to how a system recovers after a catastrophic site failure. Catastrophic failures include earthquakes, tornadoes, floods, and fires. On the most basic level, DR involves replicating an entire site, not just pieces of hardware or subcomponents. The service-level requirements for DR depend on the business applications. Some applications may not have any disaster recovery requirements. Others may simply have backup data tapes from which they would rebuild a new working site over a period of time. Still others may have requirements to begin operations with a few days or hours after the disaster. The most stringent requirement is to keep the services running despite the disaster.

The Oracle Application Server disaster recovery solution consists of two identically configured sites. Both sites may be dispersed geographically, and if so, they are connected via a network. When the primary site becomes unavailable due to disaster, the secondary site can become operational within a reasonable amount of time. Client requests are always routed to the site in the production role. After a failover or switchover operation occurs due to an outage, client requests are routed to another site that assumes the production role. Each site contains identical middle tier servers, which are also identical between the two sites. The site that is in the production role contains a production backend customer database and production Oracle Application Server Metadata Repository configured using the cold failover cluster Infrastructure high availability solution to protect from host failure. The site in the standby role contains a physical standby of the Oracle Application Server Metadata Repository. Database switchover and failover functions allow the roles to be traded between sites.

See Also: Chapter 6, "Oracle Application Server Disaster Recovery"

DCM Archive/Recover

The DCM archive and recovery feature allows you to take a snapshot of your system configuration. Taken at a time when everything is working properly and optimally, you can restore the system to this previous configuration in the event of a failure. In response to a catastrophic failure, the snapshot can be restored to a system in a remote location.

Cloning

Using the DCM tool, you can also clone an existing Oracle Application Server instance, creating an additional instance with the exact same configuration. With cloning, you reduce the possibility of introducing configuration errors during installation and setup of the new instance.

Note: Currently, only an Oracle Application Server instance installed using the J2EE and Web Cache installation type can be cloned by the DCM tool.