10 High Availability

This chapter describes issues related to highly available topologies. This chapter contains the following issues:

Section 10.1, "Configuration Issues and Workarounds"
Section 10.2, "Documentation Errata"

10.1 Configuration Issues and Workarounds

This section describes configuration issues and their workarounds. It includes the following topics:

Section 10.1.1, "Compatible ASG Releases for OracleAS Instances from Different Oracle Application Server Releases"
Section 10.1.2, "Compatible ASG Releases in an OracleAS Disaster Recovery Topology"
Section 10.1.3, "Configuring Portlet Producers and WebCenter Applications in an OracleAS Cluster Topology"
Section 10.1.4, "Supported Java Object Types for Session Attributes in Distributed Applications"
Section 10.1.5, "ASG to Catch Array Overflow Exceptions in Queries to Primary"
Section 10.1.6, "Startup of Database on the Standby2 Fails after a Switchover"
Section 10.1.7, "The create standby Command Fails if the Redo Log Files Directories do not Exist at the Standby"
Section 10.1.8, "Corrupt Index Blocks in Metadata Repository Databases"
Section 10.1.9, "Database SIDs Must be the Same for Database Peers at Primary and Standby Sites"
Section 10.1.10, "Use All Uppercase Characters for Database Initialization Parameters to Avoid Instantiate and Sync Problems"
Section 10.1.11, "Workaround for ASG_DUF-3800 "Failed trying to connect to the OPMN Manager" Error"
Section 10.1.12, "Use the Same Port for ASG on the Production and Standby Sites to Avoid clone instance Operation Problems"
Section 10.1.13, "Use Fully Qualified Path Names with the add instance Command"
Section 10.1.14, "ASG Cloning is Not Supported when the Number of Oracle Homes is Different at the Primary and Standby Hosts"
Section 10.1.15, "Steps to Add a Database with OMF or ASM to ASG Topology"

10.1.1 Compatible ASG Releases for OracleAS Instances from Different Oracle Application Server Releases

By default, when you install an Oracle Application Server instance using a particular release of Oracle Application Server, a particular release of Application Server Guard (ASG) is installed into the Oracle home for the instance. You also install ASG on standalone hosts on which external resources (such as an Oracle database) are located that you want to include in your OracleAS Disaster Recovery topology.

Multiple releases of ASG are available. It is possible (recommended) in some cases to upgrade the ASG release that was installed in an Oracle Application Server instance home when you installed that instance. To upgrade the ASG release in an Oracle Application Server instance home, download the ASG standalone kit for the recommended ASG release from the Oracle Technology Network (OTN), and then use that ASG standalone kit to install the recommended ASG release into the home. You also use the ASG standalone kit to install ASG on standalone hosts that you want to include in your OracleAS Disaster Recovery topology.

Use Table 10-1 and Table 10-2 to determine whether a particular ASG release is compatible when installed into an Application Server instance home for a particular Oracle Application Server release. The left column of the table shows the different ASG releases for which an ASG standalone installation kit is available. The remaining columns show different Oracle Application Server releases for which an Oracle Application Server instance can be created.

This list describes the meaning of the entries in Table 10-1 and Table 10-2:

N: This ASG release is not compatible with an instance from this Oracle Application Server release.
X: This ASG release cannot be installed into the Oracle home for an instance from this Oracle Application Server release.
Y-NR: This ASG release is compatible with an instance from this Oracle Application Server release, but Oracle recommends that you do not install this ASG release into the instance's Oracle home because another ASG release is recommended.
Y: This ASG release is compatible with an instance from this Oracle Application Server release. Oracle recommends you install this ASG release into the instance's Oracle home.

Table 10-1 shows the compatible ASG releases for Oracle Application Server instances from Oracle Application Server 10.1.2.0.2 through 10.1.3.3.

Table 10-1 Compatible ASG Releases for OracleAS Instances from Releases 10.1.2.0.2 Through 10.1.3.3

ASG Release	10.1.2.0.2 OracleAS Instance	10.1.2.1 OracleAS Instance	10.1.2.2 OracleAS Instance	10.1.3.0 OracleAS Instance	10.1.3.1 OracleAS Instance	10.1.3.2 OracleAS Instance	10.1.3.3 OracleAS Instance
10.1.2.0.2	Y-NR	X	X	N	N	N	N
10.1.2.2	Y	Y	Y	N	N	N	N
10.1.2.2.1 (ASG-only release)^Foot 1	Y	Y	Y	N	N	N	N
10.1.3.0	N	N	N	Y-NR	X	N	X
10.1.3.1	N	N	N	Y-NR	Y-NR	Y-NR	X
10.1.3.3	N	N	N	Y	Y	Y	Y

^Footnote 1This is the ASG release that was provided (installed by default) with the OracleAS 10.1.4.2 release. It is compatible with the OracleAS 10.1.2.x releases. There is no OracleAS 10.1.2.2.1 release.

For example, if you have an Oracle Application Server 10.1.3.1 instance and you want to know which ASG release to install in the instance home, you can use Table 10-1 to determine the following:

No ASG 10.1.2.x release is compatible with an Oracle Application Server 10.1.3.1 instance.
The ASG 10.1.3.0 release cannot be installed into the Oracle home for an Oracle Application Server 10.1.3.1 instance.
The ASG 10.1.3.1 release is compatible with an Oracle Application Server 10.1.3.1 instance, but Oracle recommends that you do not install the ASG 10.1.3.1 release into the Oracle home for an Oracle Application Server 10.1.3.1 instance.
The ASG 10.1.3.3 release is compatible with an Oracle Application Server 10.1.3.1 instance and Oracle recommends that you install the ASG 10.1.3.3 release into the Oracle home for an Oracle Application Server 10.1.3.1 instance.

Table 10-2 shows the compatible ASG releases for Oracle Application Server instances from Oracle Application Server 10.1.4.0 through 10.1.4.2.

Table 10-2 Compatible ASG Releases for OracleAS Instances from Releases 10.1.4.0 Through 10.1.4.2

ASG Release	10.1.4.0 OracleAS Instance^Foot 1	10.1.4.1 OracleAS Instance^Foot 2	10.1.4.2 OracleAS Instance^Foot 3
10.1.2.0.2	Y-NR	Y-NR	X
10.1.2.2	Y-NR	Y-NR	Y-NR
10.1.2.2.1 (ASG-only release)^Foot 4	Y	Y	Y
10.1.3.0	N	N	N
10.1.3.1	N	N	N
10.1.3.3	N	N	N

^Footnote 1ASG 10.1.2.0.2 is installed by default.

^Footnote 2ASG 10.1.2.0.2 is installed by default.

^Footnote 3ASG 10.1.2.2.1 is installed by default.

^Footnote 4This is the ASG release that was provided (installed by default) with the OracleAS 10.1.4.2 release. It is compatible with the OracleAS 10.1.2.x releases. There is no OracleAS 10.1.2.2.1 release.

10.1.2 Compatible ASG Releases in an OracleAS Disaster Recovery Topology

This chart shows which ASG release combinations are compatible in in an OracleAS Disaster Recovery topology. A topology is a collection of Oracle Application Server instance homes and standalone host homes that combine to comprise the OracleAS Disaster Recovery production site and standby site. Each Oracle Application Server home has a specific ASG release installed, either by default or by an ASG standalone installation. Since OracleAS Disaster Recovery operations are distributed, there is a collection of recommended ASG releases that should be installed across the topology. Example for the EDG deployment, the collection of Oracle homes should be upgraded to ASG version 10.1.3.3 or 10.1.2.2.x.

Use Table 10-3 to determine whether two ASG releases are compatible in an OracleAS Disaster Recovery topology. Find the first ASG release in the left column of the table and then find the second ASG release in one of the other columns of the table.

This list describes the meaning of the entries in Table 10-3:

Y-NR: The first ASG release is compatible with the second ASG release, but Oracle recommends that you do not use this ASG release combination in your topology.
Y: The first ASG release is compatible with the second ASG release. Oracle recommends that you use this ASG release combination in your topology.

Table 10-3 shows which ASG releases are compatible with other ASG releases.

Table 10-3 Compatible ASG Releases in a Topology

ASG Release	10.1.2.0.2	10.1.2.2	10.1.2.2.1	10.1.3.0	10.1.3.1	10.1.3.3
10.1.2.0.2	Y-NR	Y-NR	Y-NR	Y-NR	Y-NR	Y-NR
10.1.2.2	Y-NR	Y	Y	Y-NR	Y-NR	Y
10.1.2.2.1	Y-NR	Y	Y	Y-NR	Y-NR	Y
10.1.3.0	Y-NR	Y-NR	Y-NR	Y-NR	Y-NR	Y-NR
10.1.3.1	Y-NR	Y-NR	Y-NR	Y-NR	Y-NR	Y-NR
10.1.3.3	Y-NR	Y	Y	Y-NR	Y-NR	Y

10.1.3 Configuring Portlet Producers and WebCenter Applications in an OracleAS Cluster Topology

To run portlet producers or WebCenter applications in an OracleAS Cluster (or active-active) topology, you need to configure Java Object Cache on all the nodes in the OracleAS Cluster with a list of all the cluster nodes' IP addresses and port numbers so that it can share objects and coordinate across the cluster. You specify this list in the javacache.xml files.

There are two javacache.xml files:

Portlet producers use their own Java Object Cache, which can be configured in ORACLE_HOME/portal/conf/javacache.xml.
WebCenter applications, unless otherwise specified, use the global Java Object Cache, which can be configured in ORACLE_HOME/javacache/admin/javacache.xml.

To configure portlet producers and WebCenter applications in an OracleAS Cluster topology:

Edit both javacache.xml files on each node in the OracleAS Cluster:
- ORACLE_HOME/portal/conf/javacache.xml
- ORACLE_HOME/javacache/admin/javacache.xml
Make these edits to the files:
- Set the isDistributed element to true. You may need to create this element, if it does not already exist in the file.
- In the discoverer element, specify the IP addresses of all the nodes in the OracleAS Cluster, and specify also the discovery port. You may need to create this element, if it does not already exist in the file.
Example:
```
<communication>
   <isDistributed>true</isDistributed>
   <discoverer ip="apphost1_ip_address">
               discovery-port="apphost1_discovery_port"/>
   <discoverer ip="apphost2_ip_address"
               discovery-port="apphost2_discovery_port"/>
</communication>
```
Note:
All caches cooperating in the same cache system must specify exactly the same set of IP addresses and port numbers in the same order.
Access the Application Server Control Console at http://hostname:port/em/ and log in as the oc4jadmin user. The Cluster Topology page appears.
Restart the home and OC4J_WebCenter instances and any other instances that are running WebCenter applications.

For information on Java Object Cache, see the "Java Object Cache" chapter in the Oracle Containers for J2EE Services Guide.

10.1.4 Supported Java Object Types for Session Attributes in Distributed Applications

For applications that you want to deploy and run in a clustered environment (that is, you explicitly added the <distributable/> tag to the application's web.xml file and you have configured the application for session replication), be aware that you can bind only the following Java types as attributes in a session:

Java primitives
Java objects of type java.io.Serializable, javax.ejb.EJBObject, and javax.ejb.EJBHome

The types listed above reflect the Java objects that can be serialized and deserialized using the Java object serialization mechanism.

If you use other types, you will get an error that looks something like this:

<MSG_TEXT>ServletException cause</MSG_TEXT>
<SUPPL_DETAIL><![CDATA[java.lang.IllegalArgumentException: Only
java.io.Serializable, javax.ejb.EJBObject and javax.ejb.EJBHome instances can
be bound to a session in a distributable web-application, not:
oracle.adfdemo.view.faces.portal.SkinsDemo@2acfa2 (class
oracle.adfdemo.view.faces.portal.SkinsDemo)
  at com.evermind.server.http.EvermindHttpSession.setAttribute
                                    (EvermindHttpSession.java:143)
  at com.sun.faces.context.SessionMap.put(ExternalContextImpl.java:580)
  at com.sun.faces.application.ApplicationAssociate.
                    createAndMaybeStoreManagedBeans(ApplicationAssociate.java:299)
  at com.sun.faces.el.VariableResolverImpl.resolveVariable
                                    (VariableResolverImpl.java:97)
  at oracle.adfinternal.view.faces.el.AdfFacesVariableResolver.resolveVariable
                                    (AdfFacesVariableResolver.java:40)
  at oracle.adfinternal.view.faces.model.VariableResolverUtils$JspResolver.
                                    resolveVariable(VariableResolverUtils.java:79)
  at com.sun.faces.el.impl.NamedValue.evaluate(NamedValue.java:145)
  at com.sun.faces.el.impl.ComplexValue.evaluate(ComplexValue.java:166)
  at com.sun.faces.el.impl.ExpressionEvaluatorImpl.evaluate
                                    (ExpressionEvaluatorImpl.java:263)
  at com.sun.faces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:160)

10.1.5 ASG to Catch Array Overflow Exceptions in Queries to Primary

The asgctl create standby database command may fail with an ASG_DUF-4950 error and an error message stack on the console similar to the following:

ASG_DUF-4950: An error occurred on host "myhost" with IP "1.1.1.1" and port "7890"
ASG_SYSTEM-100: 10
ASG_DUF-4900: An exception occurred on the server.
ASG_DGA-13009: Error during Create Physical Standby

In the duf_client.log file on the primary site DB node, the following messages would be found:

java.lang.ArrayIndexOutOfBoundsException: 10
at oracle.duf.DufJdbc.queryRedoLogInfo(DufJdbc.java:535)
at oracle.duf.DufDb$jdbc.queryRedoLogInfo(DufDb.java:4966)
at oracle.duf.DufDb$jdbc.access$2000(DufDb.java:4884)
at oracle.duf.DufDb.queryRedoLogInfo(DufDb.java:3439)

This problem occurs because ASG does not properly handle a database configured with 10 or more redo log files.

To avoid this problem, reduce the number of database redo log files to fewer than 10.

This problem is fixed in release 10.1.3.3.

10.1.6 Startup of Database on the Standby2 Fails after a Switchover

Sometimes after an ASG switchover operation has been performed, a database has the following problem:

SQL> startup;
ORA-09925: Unable to create audit trail file
Linux Error: 2: No such file or directory
Additional information: 9925

When the asgctl create standby database command is executed on Linux, ASG creates the initfile with audit_file_dest set to $ohome/admin/<SID>/admin instead of $ohome/admin/<SID>/adump on the standby site. For RAC consisting of multiple nodes, one of the nodes has the admin directory created, while the other one does not.

To avoid this problem, issue the following command on the standby RAC node #2 before attempting any manual startups:

mkdir $ohome/admin/<SID>/admin

This problem is fixed in release 10.1.3.3.

10.1.7 The create standby Command Fails if the Redo Log Files Directories do not Exist at the Standby

The asgctl create standby database command fails if the redo log files directories on the production site do not exist at the standby.ASG expects the target redo log directory structure to be symmetrical with the redo log directory structure on the production site. If they do not exist in the same directories, a failure will result with the following output:

ASG_ORACLE-300: ORA-00301: error in adding log file '/PATH/redo010.log' - file cannot be created
ASG_ORACLE-300: ORA-27040: file create error, unable to create file
ASG_DUF-3700: Failed in SQL*Plus executing SQL statement:  ALTER DATABASE ADD STANDBY LOGFILE GROUP 10
'/PATH/redo010.log' SIZE 52428800 /* ASG_DGA */;.
ASG_DUF-3535: Failed to create standby redo log.
ASG_DUF-3535: Failed to create standby redo log.
ASG_DGA-13011: Error during Create Physical Standby: Finish-configure standby.
ASG_DUF-3027: Error while executing Creating physical standby database - finish phase at step - finish step.

To avoid this problem, create the target redo log file directories on the standby site in the same directories as the production site.

10.1.8 Corrupt Index Blocks in Metadata Repository Databases

After an ASG switchover or failover operation, metadata index block corruption may occur in a Disaster Recovery metadata repository database.

For a full description of the problem and an explanation of how to deal with it, refer to OracleMetaLink note 386830.1 at:

http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=386830.1

10.1.9 Database SIDs Must be the Same for Database Peers at Primary and Standby Sites

The SIDs must be the same for database peers at a primary site and standby site(s) in a Disaster Recovery topology.

10.1.10 Use All Uppercase Characters for Database Initialization Parameters to Avoid Instantiate and Sync Problems

Use all uppercase characters for database initialization parameters.

In the following example, the database initialization parameter, service, that is used in the archive log destination parameter is in all uppercase characters (SERVICE):

log_archive_dest_2="SERVICE=SIDM valid_for=(online_logfiles,primary_role) 
db_unique_name=SIDM"

But in the following example, the database initialization parameter, service, that is used in the archive log destination parameter is in lowercase characters (service):

log_archive_dest_2="service=SIDM valid_for=(online_logfiles,primary_role) 
db_unique_name="SIDM"

When the database initialization parameter is not in all uppercase characters, error messages similar to the following can occur during an instantiate topology or sync topology operation:

stajo05: -->ASG_DUF-4950: An error occurred on host "stajo05" with IP
"140.87.25.33" and port "7890"
stajo05: -->ASG_SYSTEM-100: String index out of range: -9
stajo05: -->ASG_DUF-3760: Failed to query archive log destination
information.
stajo05: -->ASG_IAS-15753: Error preparing to instantiate the topology on
host "stajo05"
stajo05: -->ASG_DUF-3027: Error while executing Instantiating each instance
in the topology to standby topology at step - prepare step.

10.1.11 Workaround for ASG_DUF-3800 "Failed trying to connect to the OPMN Manager" Error

When using the 10.1.2.2 ASG standalone kit to update a 10.1.4.x Oracle home, ASG installed a copy of optic.jar in its component path that is not compatible with the optic.jar of the Oracle home. Although this will not affect the operation of the runtime system, it could potentially cause problems with some future ASG operations in certain configurations.

To prevent these potential problems:

Rename optic.jar in the $ORACLE_HOME/dsa/jlib directory to optic.jar.orig on all nodes of the topology.
Restart the ASG component, if started.

If the above workaround is not performed, you may receive the ASG_DUF-3800 error "Failed trying to connect to the OPMN Manager" error message during a switchover topology operation if your Disaster Recovery configuration includes Oracle Application Server release 10.1.4.0.1 Identity Management and the standalone ASG kit for release 10.1.2.2.

The error messages received may be similar to:

"4-May 12:51:20  stamx12 152.68.64.214:7890(home/opt/maa/oracle/product/10.1.4/im)
4-May 12:51:20      Running opmnctl reload command:
"/opt/maa/oracle/product/10.1.4/im/opmn/bin/opmnctl reload".
4-May 12:51:25  stamx11: -->ASG_DUF-4950: An error occurred on host "stamx11"
with IP "140.87.21.201" and port "7890"
stamx11: -->ASG_DUF-4950: An error occurred on host "stamx11" with IP
"152.68.64.213" and port "7890"
stamx11: -->ASG_DUF-3800: Failed trying to connect to the OPMN Manager.
stamx11: -->ASG_DUF-3027: Error while executing Starting each instance in the
topology at step - starting instance step.
stamx11: -->ASG_DUF-3027: Error while executing Switchover each instance in
the topology to standby topology at step - starting and resyncing instance
step."

10.1.12 Use the Same Port for ASG on the Production and Standby Sites to Avoid clone instance Operation Problems

Use the same port for ASG on the primary site and standby site(s) to avoid error messages such as the following during a clone instance operation:

3-May 15:45:43  >>clone instance prodsso1 to stbyinfra1
3-May 15:45:43  stamx11: -->ASG_DUF-4950: An error occurred on host
"stamx11" with IP "140.87.21.201" and port "7890"
stamx11: -->ASG_DUF-3601: Error connecting to server host 152.68.64.213
on port 7890
stamx11: -->ASG_DUF-3512: Error creating remote worker on node 152.68.64.213:7890.

The dsa.conf file contains ASG configuration information, and it is configured into the Application Server instance's backup/restore IP configuration. The dsa.conf file configuration is handled symmetrically between Application Server instances. Due to this, the dsa.conf file from a production site's instance will be synchronized to the corresponding standby site's instance.

The port numbers between the production and standby instance pairings should match for ASG.

10.1.13 Use Fully Qualified Path Names with the add instance Command

As a best practice, use fully qualified path names with the add instance command.

10.1.14 ASG Cloning is Not Supported when the Number of Oracle Homes is Different at the Primary and Standby Hosts

The ASG clone topology and clone instance commands are not supported by DR configurations if there are a different number of Oracle Homes at the primary and standby hosts.

As part of the cloning operation, the Oracle Inventory for each host is cloned. Therefore, the assumption is that the Oracle Home configuration is symmetrical for any host that is being cloned.

For a full description of supported Disaster Recovery asymmetric topologies, refer to Section 5.1.3.2 of the Application Server High Availability Guide for release 10.1.3.2.0.

10.1.15 Steps to Add a Database with OMF or ASM to ASG Topology

The asgctl create standby database command is designed to automate the creation of simple standby databases. It does not support some database options, such as the OMF (Oracle Managed Files) or ASM (Automatic Storage Management) storage options. If you plan to use the create standby database command to create a database at the standby site, create a new database instance on the primary site using DBCA (Database Configuration Assistant) without specifying the OMF or ASM storage options.

To create a new database instance with the OMF or ASM storage options, follow the instructions in the "Creating a Standby Database the Uses OMF or ASM" section of the Oracle Data Guard Concepts and Administration manual at:

http://www.oracle.com/technology/documentation/index.html

Then, after creating the database with the OMF or ASM storage options, use the asgctl add instance command to add the instance to the Disaster Recovery topology, so that it can be included in Disaster Recovery operations, such as failover and instantiate topology operations. See Section A.1.23 "OracleAS Guard Add Instance Command Fails When Attempting to Add an Oracle RAC Database to the Topology" of the Oracle Application Server High Availability Guide for release 10.1.3.2 for more information about using the asgctl add instance command.

10.2 Documentation Errata

The section describes documentation errata. It includes the following topics:

Section 10.2.1, "Additional Step in Switchover Procedure"
Section 10.2.2, "Additional Site Switchover Information for RAC Deployments"
Section 10.2.3, "Correction to Configuration Instructions for a Primary Site with RAC Database and Standby Site with Non-RAC Database"

10.2.1 Additional Step in Switchover Procedure

In section 5.1.4.1.3 of the Oracle Application Server High Availability Guide, there should be a step 4.f after step 4.e for Unix platforms. Step 4.f is as follows:

Create an admin directory under the default ADMIN directory of the database (ADMINDIR/<dbname>/admin) on standbynode2. For example:

mkdir ORACLE_HOME/admin/<dbname>/admin

If this directory already exists, skip this step.

10.2.2 Additional Site Switchover Information for RAC Deployments

The Oracle Application Server High Availability Guide for 10.1.3.2 includes the "Site Switchover Operations" subsection in Section 5.12.1.1 "Scheduled Outages."

In step 5 in the numbered list in the "Site Switchover Operations" subsection, the following two list elements regarding RAC deployments should be added as list elements c and d:

For Oracle RAC Disaster Recovery deployments, shut down all the RAC instances and start up a single RAC node prior to the switchover.
After setting up the ORACLE_HOME, ORACLE_SID, and PATH variables on the database node, start up the database node:
```
DBHOME/bin/sqlplus /as sysdba
SQL> startup
```

10.2.3 Correction to Configuration Instructions for a Primary Site with RAC Database and Standby Site with Non-RAC Database

Section 5.14.2.2 "Configuration Procedure" of the Oracle Application Server High Availability Guide for release 10.1.3.2.0 includes incorrect information in Step 1.

The introductory paragraph before Step 1 says that the standby site uses a non-Real Application Clusters database.

However, Step 1 says to "Stop the Real Application Clusters database on the standby site."

The database on the standby site is a non-Real Applications Cluster database. Step 1 should read "Stop the non-Real Application Cluster database on the standby site."