| Oracle® Application Server High Availability Guide 10g (10.1.4.0.1) Part Number B28186-01 |
|
|
View PDF |
| Oracle® Application Server High Availability Guide 10g (10.1.4.0.1) Part Number B28186-01 |
|
|
View PDF |
This release of Oracle Application Server extends and improves upon the high availability solutions that were available in earlier releases. New, flexible, and automated high availability solutions for Oracle Application Server have been tested and are described in this guide. All of these solutions seek to ensure that applications that you deploy on Oracle Application Server meet the required availability to achieve your business goals. The solutions and procedures described in this book seek to eliminate single points of failure of any Oracle Application Server components with no or minimal outage in service.
This chapter explains high availability and its importance from the perspective of Oracle Application Server.
This chapter contains the following sections:
Section 1.2, "Oracle Application Server High Availability Concepts"
Section 1.3, "Oracle Application Server Features Related to High Availability"
Section 1.4, "High Availability Information in Other Documentation"
This section provides an overview of high availability from a problem-solution perspective. It contains these sections:
Mission critical computer systems need to be available 24 hours a day, 7 days a week, and 365 days a year. However, part or all of the system may be down during planned or unplanned downtime. A system's availability is measured by the percentage of time that it is providing service in the total time since it is deployed. Table 1-1 provides an example.
Table 1-1 Availability Percentages and Corresponding Downtime Values
| Availability Percentage | Approximate Downtime Per Year |
|---|---|
|
95% |
18 days |
|
99% |
4 days |
|
99.9% |
9 hours |
|
99.99% |
1 hour |
|
99.999% |
5 minutes |
Table 1-2 depicts the various types of failures that are possible with a computer system.
Table 1-2 System Downtime and Failure Types
| Downtime Type | Failure Type |
|---|---|
|
Unplanned downtime |
System failure |
|
Data failure |
|
|
Disasters |
|
|
Human error |
|
|
Planned downtime |
System maintenanceFoot 1 |
|
Data maintenance |
Footnote 1 Includes hardware and/or software changes (operating system, application server, configuration, application changes).
These two types of downtimes (planned and unplanned) are usually considered separately when designing a system's availability requirements. A system's needs may be very restrictive regarding its unplanned downtimes, but very flexible for planned downtimes. This is the typical case for applications with high peak loads during working hours, but that remain practically inactive at night and during weekends.
Oracle Application Server provides both local high availability and disaster recovery solutions for maximum protection against any kind of failure with flexible installation, deployment, and security options. The redundancy of Oracle Application Server local high availability and disaster recovery originates from its redundant high availability architectures.
High availability solutions can be categorized into:
local high availability solutions, which provide high availability in a single data center deployment. Local high availability solutions protect against failures such as process, node, and media failures as well as human errors.
Oracle Application Server provides many local high availability solutions. See Chapter 2, "Overview of Oracle Application Server High Availability Topologies" for an overview.
disaster recovery solutions, which are usually geographically distributed deployments that protect your applications from disasters such as floods or regional network outages.
To solve the high availability problem, a number of technologies and best practices are needed. The most important mechanism is redundancy. High availability comes from redundant systems and components. Local high availability solutions can be categorized, by how they provide redundancy, into active-active solutions and active-passive solutions (see Figure 1-1).
Figure 1-1 Active-Active and Active-Passive High Availability Solutions
Active-active solutions deploy two or more active system instances and can be used to improve scalability as well as provide high availability. All instances handle requests concurrently.
Active-passive solutions deploy an active instance which handles requests and a passive instance which is on standby. A heartbeat mechanism is set up between these two instances. This mechanism is provided and managed through operating system vendor-specific clusterware. Generally, vendor-specific cluster agents are also available to automatically monitor and failover between cluster nodes, so that when the active instance fails, an agent shuts down the active instance completely, brings up the passive instance, and application services can successfully resume processing. As a result, the active-passive roles are now switched. The same procedure can be done manually for planned or unplanned downtime. Active-passive solutions are also generally referred to as cold failover clusters.
From the entry point of an Oracle Application Server system (content cache) to the back end layer (data sources), all the tiers that are crossed by a client request can be configured in a redundant manner either in an active-active topology using OracleAS Clusters or in an active-passive topology using OracleAS Cold Failover Clusters.
In addition to architectural redundancies, the following local high availability technologies are also necessary in a comprehensive high availability system:
Process death detection and automatic restart
Processes may die unexpectedly due to configuration or software problems. A proper process monitoring and restart system should monitor all system processes constantly and restart them should problems appear.
A system process should also maintain the number of restarts within a specified time interval. This is also important since continually restarting within short time periods may lead to additional faults or failures. Therefore a maximum number of restarts or retries within a specified time interval should also be designed as well.
Oracle Application Server performs process death detection and automatic restart through Oracle Process Manager and Notification Server (OPMN). For details, see Section 1.3.1, "Process Death Detection and Automatic Restart".
Clustering
Clustering components of a system together enables the components to be viewed functionally as a single entity from the perspective of a client for runtime processing and manageability. A cluster is a set of processes running on single or multiple nodes that share the same workload. There is a close correlation between clustering and redundancy. A cluster provides redundancy for a system.
Oracle Application Server provides clustering at different levels (for example, instance clustering and component clustering). For creating high availability topologies, you can cluster at the instance level with OracleAS Clusters. See Chapter 2, "Overview of Oracle Application Server High Availability Topologies" for details.
Configuration management
A cluster of components often need to share common configuration. Proper configuration management ensures that components provide the same reply to the same incoming request, enables the components to synchronize their configurations, and provides highly available configuration management for less administration downtime.
In Oracle Application Server, the Distributed Configuration Management (DCM) component synchronizes configuration information for the members of a cluster.
State replication and routing
For stateful applications, client state can be replicated to enable stateful failover of requests in the event that processes servicing these requests fail.
Server load balancing and failover
When multiple instances of identical server components are available, client requests to these components can be load balanced to ensure that the instances have roughly the same workload. With a load balancing mechanism in place, the instances are redundant. If any of the instances fail, requests to the failed instance can be sent to the surviving instances.
Oracle Application Server components perform load balancing as necessary when communicating with other components. See Section 1.3.2, "Internal Load Balancing in Oracle Application Server" for details.
Backup and recovery
User errors may cause a system to malfunction. In certain circumstances, a component or system failure may not be repairable. A backup and recovery facility should be available to back up the system at certain intervals and restore a backup when an unrepairable failure occurs.
Oracle Application Server provides the OracleAS Backup and Recovery Tool to assist you in backing up and restoring Oracle Application Server files. See Section 1.3.3, "Backup and Recovery" for details.
The Oracle Application Server Disaster Recovery solution builds on top of the local high availability solutions and includes Oracle Application Server Guard. This unique solution combines the proven Oracle Data Guard technology in the Oracle Database with advanced disaster recovery technologies in the application realm to create a comprehensive disaster recovery solution. This solution requires homogenous production and standby sites, but other Oracle Application Server instances can also be installed in either site as long as they do not interfere with the instances in the disaster recovery setup. Configurations and data must be synchronized regularly between the two sites to maintain homogeneity.
Disaster recovery solutions typically set up two homogeneous sites, one active and one passive. Each site is a self-contained system. The active site is generally called the production site, and the passive site is called the standby site. During normal operation, the production site services requests; in the event of a site failover or switchover, the standby site takes over the production role and all requests are routed to that site. To maintain the standby site for failover, not only must the standby site contain homogeneous installations and applications, data and configurations must also be synchronized constantly from the production site to the standby site.
See Part IV, "Disaster Recovery" for complete details on OracleAS Disaster Recovery solutions.
Figure 1-2 Geographically Distributed Disaster Recovery
The following sections provide an overview of high availability in Oracle Application Server:
Section 1.2.2, "Oracle Application Server Base Architecture"
Section 1.2.3, "Oracle Application Server Base Architecture with Oracle Access Manager"
Section 1.2.4, "Oracle Application Server Base Architecture with Oracle Identity Federation"
This book uses the terms below to describe high availability concepts and topologies:
active-active: In an active-active high availability system, the equivalent members in that system can service requests concurrently. Under normal operation where none of the members have failed, all members are active and none are on standby.
active-passive: In an active-passive high availability system, some members of the system can be actively servicing requests and performing work, while other members are inactive (passive). They become active only if one or more of the active nodes have failed. Consumers of services provided by the system may or may not notice the failure. An active-active system generally provides more transparency and options for scalability to consumers than an active-passive system.
failover: When a member of a highly available system fails unexpectedly (unplanned downtime), in order to continue offering services to its consumers, the system undergoes a failover operation. If the system is an active-passive system, the passive member is activated during the failover operation and consumers are directed to it instead of the failed member. The failover process can be performed manually, or it can be automated by setting up hardware cluster services to detect failures and move cluster resources from the failed node to the standby node. If the system is an active-active system, the failover is performed by the load balancer entity serving requests to the active members. If an active member fails, the load balancer detects the failure and automatically redirects requests for the failed member to the surviving active members.
failback: After a system undergoes a successful failover operation, the original failed member can be repaired over time and be re-introduced into the system as a standby member. If desired, a failback process can be initiated to activate this member and deactivate the other. This process reverts the system back to its pre-failure configuration.
hardware cluster: A hardware cluster is a collection of nodes that provides a single view of network services (for example: an IP address) or application services (for example: databases, Web servers) to clients of these services. Each node in a hardware cluster is a standalone server that runs its own processes. These processes can communicate with one another to form what looks like a single system that cooperatively provides applications, system resources, and data to users.
A hardware cluster achieves high availability and scalability through the use of specialized hardware (cluster interconnect, shared storage) and software (health monitors, resource monitors). (The cluster interconnect is a private link used by the hardware cluster for heartbeat information to detect node death.) Due to the need for specialized hardware and software, hardware clusters are commonly provided by hardware vendors such as Sun, Hewlett-Packard, IBM, and Dell. While the number of nodes that can be configured in a hardware cluster is vendor dependent, for the purpose of Oracle Application Server high availability, only two nodes are required. Hence, this document assumes a two-node hardware cluster for high availability solutions employing a hardware cluster.
cluster agent: A cluster agent is the software that runs on a node member of a hardware cluster that coordinates availability and performance operations with other nodes. Clusterware provides resource grouping, monitoring, and the ability to move services. A cluster agent can automate the service failover.
clusterware: Clusterware is the software that manages the operations of the members of a cluster as a system. It enables you to define resources and services to monitor via a heartbeat mechanism between cluster members and to move these resources and services to a different member in the cluster as efficiently and transparently as possible.
shared storage: Although each hardware cluster node is a standalone server that runs its own set of processes, the storage subsystem required for any cluster-aware purpose is usually shared. Shared storage refers to the ability of the cluster to be able to access the same storage, usually disks, from both the nodes. While the nodes have equal access to the storage, only one node, the primary node, has active access to the storage at any given time. The hardware cluster's software grants the secondary node access to this storage if the primary node fails. For the OracleAS Infrastructure in the OracleAS Cold Failover Cluster environment, its ORACLE_HOME is on such a shared storage file system. This file system is mounted by the active node; if that node fails, the standby node takes over and mounts the file system. In some cases, the active node may relinquish control of the shared storage, such as when the hardware cluster's software deems the OracleAS Infrastructure as unusable from the active node and decides to move it to the standby node.
primary node: The primary node is the node that is actively running one or more Oracle Application Server installations at any given time. If this node fails, Oracle Application Server is failed over to the secondary node. Because the primary node runs the active Oracle Application Server installation(s), it is considered the "hot" node. See the definition for "secondary node" in this section.
secondary node: The secondary node is the node that takes over the execution of Oracle Application Server if the primary node fails. Because the secondary node does not originally run Oracle Application Server, it is considered the "cold" node. Because Oracle Application Server fails over from a hot node (primary) to a cold node (secondary), this type of failover is called cold failover. See the definition for "primary node" in this section.
network hostname: The network hostname is a name assigned to an IP address through the /etc/hosts file (in UNIX), C:\WINDOWS\system32\drivers\etc\hosts file (in Windows), or through DNS resolution. This name is visible in the network to which the machine is connected. The network hostname and the physical hostname are usually the same. However, each machine has only one physical hostname but may have multiple network hostnames. Thus, a machine's network hostname may not always be its physical hostname.
physical hostname: This guide differentiates between physical hostname and network hostname. This guide uses physical hostname to refer to the "internal name" of the current machine. In UNIX, this is the name returned by the hostname command.
Physical hostname is used by Oracle Application Server middle-tier installation types to reference the local host. During installation, the installer automatically retrieves the physical hostname from the current machine and stores it in the Oracle Application Server configuration metadata on disk.
switchover: During normal operation, active members of a system may require maintenance or upgrading. A switchover process can be initiated to enable a substitute member to take over the workload performed by the member that requires maintenance or upgrading, which undergoes planned downtime. The switchover operation ensures continued service to consumers of the system.
switchback: When a switchover operation is performed, a member of the system is deactivated for maintenance or upgrading. When the maintenance or upgrading is completed, the system can undergo a switchback operation to activate the upgraded member and bring the system back to the pre-switchover configuration.
virtual hostname: Virtual hostname is a network addressable hostname that maps to one or more physical machines via a load balancer or a hardware cluster. For load balancers, the "virtual server name" is used interchangeably with virtual hostname in this book. A load balancer can hold a virtual hostname on behalf of a set of servers, and clients communicate indirectly with the machines using the virtual hostname. A virtual hostname in a hardware cluster is a network hostname assigned to a cluster virtual IP. Because the cluster virtual IP is not permanently attached to any particular node of a cluster, the virtual hostname is not permanently attached to any particular node either.
|
Note: Whenever the phrase "virtual hostname" is used in this guide, it is assumed to be associated with a virtual IP address. In cases where just the IP address is needed or used, it will be explicitly stated. |
virtual IP, cluster virtual IP: Generally, a virtual IP can be assigned to a hardware cluster or load balancer. To present a single system view of a cluster to network clients, a virtual IP serves as an entry point IP address to the group of servers which are members of the cluster. A virtual IP can be assigned to a server load balancer or a hardware cluster.
A hardware cluster uses a cluster virtual IP to present to the outside world the entry point into the cluster (it can also be set up on a standalone machine). The hardware cluster's software manages the movement of this IP address between the two physical nodes of the cluster while clients connect to this IP address without knowing which physical node this IP address is currently active on. In a typical two-node hardware cluster topology, each machine has its own physical IP address and physical hostname, while there could be several cluster IP addresses. These cluster IP addresses float or migrate between the two nodes. The node with current ownership of a cluster IP address is active for that address.
A load balancer also uses a virtual IP as the entry point to a set of servers. These servers tend to be active at the same time. This virtual IP address is not assigned to any individual server but to the load balancer which acts as a proxy between servers and their clients.
Before building Oracle Application Server high availability topologies, you should understand Oracle Application Server's base architecture because to create a highly available topology, you add redundancy to the base architecture. This means that you add redundancy to all Oracle Application Server components and also to connections between components.
Figure 1-3 illustrates the base architecture of Oracle Application Server.
Figure 1-3 Oracle Application Server Base Architecture
At a high level, Oracle Application Server consists of the Oracle Application Server middle-tier business applications, Oracle Identity Management, and OracleAS Metadata Repository. The latter two are part of the OracleAS Infrastructure.
Oracle Identity Management manages user authentication, authorization, and identity information. It includes the following components:
OracleAS Single Sign-On
Oracle Delegated Administration Services
Oracle Internet Directory
Oracle Directory Integration Platform
Architecturally, Oracle Identity Management can be broken down into a Web server tier of Oracle HTTP Server, an OracleAS Single Sign-On/Oracle Delegated Administration Services middle-tier composed of an Oracle Containers for J2EE (OC4J) instance for these security applications, and an Oracle Internet Directory/Oracle Directory Integration Platform tier at the back end. OracleAS Metadata Repository is an Oracle database that manages configuration, management, and product metadata for middle-tier and Oracle Identity Management components.
The middle tier hosts most of Oracle Application Server business applications, such as:
Oracle Application Server Portal
Oracle Application Server Wireless
Oracle Application Server Integration
These applications rely on Oracle Identity Management and OracleAS Metadata Repository for security and metadata support. The middle tier also includes a Web caching sub-tier (Oracle Application Server Web Cache), a Web server sub-tier (Oracle HTTP Server), and OC4J instances. Behind the middle tier, the OracleAS Metadata Repository serves as the data tier. In actual deployments, other databases may also exist in the data tier (for example, a customer database for OC4J applications deployed on the middle tier).
Figure 1-4 shows the various tiers that are traversed by client requests to the Oracle Application Server business applications and the Oracle Application Server Infrastructure services. Figure 1-5 provides an overall view of Oracle Identity Management, OracleAS Metadata Repository, and LDAP services.
Although Oracle Application Server provides many features that make it easy to build high availability topologies (such as automatic process monitoring and restart, and backup and recovery), they do not provide complete high availability. Several single points of failure exist. To eliminate them, you need to provide redundancy for each component. This can be achieved by extending the base architecture with additional high availability architectures.
Figure 1-4 Base Architecture of Oracle Application Server
Figure 1-5 Overview of Infrastructure Services
Oracle Access Manager provides single sign-on, authorization, and authentication services for your applications. It is an alternative to using OracleAS Single Sign-On and Oracle Delegated Administration Services.
Oracle Access Manager stores its information on users in an LDAP directory server (for example, Oracle Internet Directory).
Oracle Access Manager includes WebGate and WebPass, which are plug-ins for the Oracle HTTP Server. WebGate directs requests from the Oracle HTTP Server to the Access Server, and WebPass directs requests from the Oracle HTTP Server to the Identity Server.
Figure 1-6 shows a simple topology that involves Oracle Access Manager.
Figure 1-6 Simple Topology with Oracle Access Manager
Oracle Identity Federation goes beyond simple single sign-on in that it enables you to share identity information with your business partners. In a federated environment (that is, an environment where services and information are shared between trusted partners), users have to sign on only once to access all the entities in the federation. Without Oracle Identity Federation, users have to sign on at each entity.
Oracle Application Server includes the following features that are helpful in high availability topologies. These features are used in all Oracle Application Server topologies, not just high availability topologies.
Section 1.3.1, "Process Death Detection and Automatic Restart"
Section 1.3.2, "Internal Load Balancing in Oracle Application Server"
An Oracle Application Server instance runs many different processes to serve client requests. Ensuring high availability means ensuring that all these processes run smoothly, fulfill requests, and do not experience any unexpected hangs or failures.
The interdependency of these processes must also be managed so that they are started in the proper sequence, with processes starting up only after the processes that they are dependent on have started successfully.
Oracle Application Server provides management services at the process level with Oracle Process Manager and Notification Server (OPMN). OPMN has the following capabilities:
Provides automatic death detection of Oracle Application Server processes.
Provides automatic restart of Oracle Application Server processes when they become unresponsive, terminate unexpectedly, or become unreachable as determined by ping and notification operations.
Channels events from different Oracle Application Server component instances to all Oracle Application Server components that can utilize them.
Enables gathering of host and Oracle Application Server process statistics and tasks.
Does not depend on any other Oracle Application Server component being up and running before it can be started and used.
OPMN manages the following Oracle Application Server components:
Oracle HTTP Server
Oracle Containers for J2EE (OC4J)
Distributed Configuration Management daemon
OracleAS Log Loader
OracleAS Guard (for disaster recovery)
Oracle Internet Directory
OracleAS Port Tunnelling
In addition, OPMN implicitly manages any applications that rely on the above components. For example, J2EE applications, which run under OC4J, are managed by OPMN.
|
Note: OPMN does not manage Oracle Access Manager or Oracle Identity Federation. |
OPMN running on different Oracle Application Server instances can also work together to provide distributed process management and control. For example, you can issue an OPMN command on one machine to start all processes or a specific process type across all local and remote Oracle Application Server instances.
OPMN consists of two major components:
Oracle Notification Server (ONS)
The ONS is the transport mechanism for failure, recovery, startup, and other related notifications between components in Oracle Application Server. It operates according to a publish-subscribe model: an Oracle Application Server component receives a notification of a certain type per its subscription to ONS. When such a notification is published, ONS sends it to the appropriate subscribers.
Oracle Process Manager (PM)
The PM is the centralized process management mechanism in Oracle Application Server and is used to manage Oracle Application Server processes. It is responsible for starting, restarting, stopping, and monitoring every process it manages. The PM handles all requests sent to OPMN associated with controlling a process or obtaining status about a process. The PM is also responsible for performing death-detection and automatic restart of the processes it manages. The Oracle Application Server processes that the PM is configured to manage are specified in a file named opmn.xml. The PM waits for a user command to start processes. When a specific process or all processes are to be stopped, the PM receives a request as specified by the request parameters.
Load balancing involves the ability to distribute requests among two or more processes. A software or hardware load balancer typically includes the following features:
load balancing algorithm
A set of rules for distributing requests across the different instances. Common load balancing algorithms include simple round-robin or assignment based on some weighted property of the instance, such as the response time or capacity of that instance relative to other instances.
death detection
The ability to recognize failed requests to one or more instances, and additionally, the ability to mark those instances as inactive so that no further requests will be forwarded to them.
Oracle Application Server uses different load balancing mechanisms to communicate requests between components in an Oracle Application Server system. Load balancing takes place:
from OracleAS Web Cache to Oracle HTTP Servers
from Oracle HTTP Servers to OC4J processes for J2EE applications
from Oracle HTTP Servers to the database for PL/SQL applications
intra OC4J processes from the presentation layer components (servlets and JSPs) to the business layer components (EJBs)
from OC4J processes to databases
from WebGate to Access Servers
from WebPass to Identity Servers
All tiers in Oracle Application Server are enabled to manage failures in the connections that they establish with the next tier as follows:
Connections established from OracleAS Web Cache to Oracle HTTP Servers: OracleAS Web Cache detects failures in the replies returned by Oracle HTTP Servers and routes the new requests to the available Oracle HTTP Servers.
Connections established from Oracle HTTP Servers to OC4J processes: Oracle HTTP Server maintains a routing table of available OC4J processes and routes new requests only to those OC4J processes that are up an running.
Connections established from Oracle HTTP Servers to databases: mod_plsql detects failures in the database and routes requests to the available database nodes.
Connections established between OC4J processes: OC4J detects failures in the RMI invocations to the EJB tier and fails communication over to available EJB nodes.
Connections established between OC4J processes and databases: OC4J drivers are enabled to detect failures of database nodes and re-route requests to available nodes.
Backing up your files to protect against data loss is critical to maintaining a highly available environment. Oracle Application Server includes the OracleAS Backup and Recovery Tool to help you back up configuration and data files.
You can use the OracleAS Backup and Recovery Tool to back up and recover the following types of files:
configuration files in the middle-tier and OracleAS Infrastructure homes
OracleAS Metadata Repository files
The OracleAS Backup and Recovery Tool is installed whenever you install Oracle Application Server. The tool is installed in the ORACLE_HOME/backup_restore directory.
The OracleAS Backup and Recovery Tool supports the following installation types:
OracleAS Infrastructure (Identity Management and Metadata Repository)
OracleAS Infrastructure (Identity Management only)
OracleAS Infrastructure (Metadata Repository only)
|
Note: OracleAS Backup and Recovery Tool does not back up files in the Oracle Access Manager or Oracle Identity Federation homes. |
A complete Oracle Application Server environment backup includes:
A full backup of all files in the middle-tier Oracle homes (this includes Oracle software files and configuration files).
A full backup of all files in the OracleAS Infrastructure home (this includes Oracle software files and configuration files).
A complete cold backup of the OracleAS Metadata Repository.
A full backup of the Oracle system files on each host in your environment.
See the "Backup and Recovery" chapters in the Oracle Application Server Administrator's Guide for details.
Table 1-3 provides a list of cross-references to high availability information in other documents in the Oracle library. This information mostly pertains to high availability of various Oracle Application Server components.
Table 1-3 High Availability Information in Oracle Documentation
| Component | Location of Information |
|---|---|
|
Oracle installer |
In the chapters for installing in high availability environments in Oracle Application Server Installation Guide. |
|
Oracle Application Server Backup and Recovery Tool |
In the backup and restore part of Oracle Application Server Administrator's Guide. |
|
Identity Management service replication |
In the "Advanced Configurations" chapter of Oracle Application Server Single Sign-On Administrator's Guide. |
|
Identity Management high availability deployment |
In the "Identity Management Deployment Planning" chapter of Oracle Identity Management Infrastructure Administrator's Guide. |
|
Database high availability |
Oracle High Availability Architecture and Best Practices |
|
Oracle Process Manager and Notification Server commands |
Oracle Process Manager and Notification Server Administrator's Guide |
|
Load balancing to OC4J processes |
Oracle HTTP Server Administrator's Guide |
In addition, references to these and other documentation are noted in the text of this guide, where applicable.
