Skip Headers
Oracle® Application Server Release Notes
10g Release 3 (10.1.3.1.0) for HP-UX PA-RISC

Part Number B32220-06
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Master Index
Master Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
View PDF

15 High Availability

This chapter describes issues related to highly available topologies using the OracleAS High Availability. This chapter contains the following issues:

15.1 General Issues and Workarounds

This section describes general issues and workarounds. It includes the following topic:

15.1.1 Compatible ASG Releases for OracleAS Instances from Different Oracle Application Server Releases

By default, when you install an Oracle Application Server instance using a particular release of Oracle Application Server, a particular release of Application Server Guard (ASG) is installed into the Oracle home for the instance. You also install ASG on standalone hosts on which external resources (such as an Oracle database) are located that you want to include in your OracleAS Disaster Recovery topology.

Multiple releases of ASG are available. It is possible (recommended) in some cases to upgrade the ASG release that was installed in an Oracle Application Server instance home when you installed that instance. To upgrade the ASG release in an Oracle Application Server instance home, download the ASG standalone kit for the recommended ASG release from the Oracle Technology Network (OTN), and then use that ASG standalone kit to install the recommended ASG release into the home. You also use the ASG standalone kit to install ASG on standalone hosts that you want to include in your OracleAS Disaster Recovery topology.

Use Table 15-1 and Table 15-2 to determine whether a particular ASG release is compatible when installed into an Application Server instance home for a particular Oracle Application Server release. The left column of the table shows the different ASG releases for which an ASG standalone installation kit is available. The remaining columns show different Oracle Application Server releases for which an Oracle Application Server instance can be created.

This list describes the meaning of the entries in Table 15-1 and Table 15-2:

  • N: This ASG release is not compatible with an instance from this Oracle Application Server release.

  • X: This ASG release cannot be installed into the Oracle home for an instance from this Oracle Application Server release.

  • Y-NR: This ASG release is compatible with an instance from this Oracle Application Server release, but Oracle recommends that you do not install this ASG release into the instance's Oracle home because another ASG release is recommended.

  • Y: This ASG release is compatible with an instance from this Oracle Application Server release. Oracle recommends you install this ASG release into the instance's Oracle home.

Table 15-1 shows the compatible ASG releases for Oracle Application Server instances from Oracle Application Server 10.1.2.0.2 through 10.1.3.3.

Table 15-1 Compatible ASG Releases for OracleAS Instances from Releases 10.1.2.0.2 Through 10.1.3.3

ASG Release 10.1.2.0.2 OracleAS Instance 10.1.2.1 OracleAS Instance 10.1.2.2 OracleAS Instance 10.1.3.0 OracleAS Instance 10.1.3.1 OracleAS Instance 10.1.3.2 OracleAS Instance 10.1.3.3 OracleAS Instance

10.1.2.0.2

Y-NR

X

X

N

N

N

N

10.1.2.2

Y

Y

Y

N

N

N

N

10.1.2.2.1 (ASG-only release)Foot 1 

Y

Y

Y

N

N

N

N

10.1.3.0

N

N

N

Y-NR

X

N

X

10.1.3.1

N

N

N

Y-NR

Y-NR

Y-NR

X

10.1.3.3

N

N

N

Y

Y

Y

Y


Footnote 1 This is the ASG release that was provided (installed by default) with the OracleAS 10.1.4.2 release. It is compatible with the OracleAS 10.1.2.x releases. There is no OracleAS 10.1.2.2.1 release.

For example, if you have an Oracle Application Server 10.1.3.1 instance and you want to know which ASG release to install in the instance home, you can use Table 15-1 to determine the following:

  • No ASG 10.1.2.x release is compatible with an Oracle Application Server 10.1.3.1 instance.

  • The ASG 10.1.3.0 release cannot be installed into the Oracle home for an Oracle Application Server 10.1.3.1 instance.

  • The ASG 10.1.3.1 release is compatible with an Oracle Application Server 10.1.3.1 instance, but Oracle recommends that you do not install the ASG 10.1.3.1 release into the Oracle home for an Oracle Application Server 10.1.3.1 instance.

  • The ASG 10.1.3.3 release is compatible with an Oracle Application Server 10.1.3.1 instance and Oracle recommends that you install the ASG 10.1.3.3 release into the Oracle home for an Oracle Application Server 10.1.3.1 instance.

Table 15-2 shows the compatible ASG releases for Oracle Application Server instances from Oracle Application Server 10.1.4.0 through 10.1.4.2.

Table 15-2 Compatible ASG Releases for OracleAS Instances from Releases 10.1.4.0 Through 10.1.4.2

ASG Release 10.1.4.0 OracleAS InstanceFoot 1  10.1.4.1 OracleAS InstanceFoot 2  10.1.4.2 OracleAS InstanceFoot 3 

10.1.2.0.2

Y-NR

Y-NR

X

10.1.2.2

Y-NR

Y-NR

Y-NR

10.1.2.2.1 (ASG-only release)Foot 4 

Y

Y

Y

10.1.3.0

N

N

N

10.1.3.1

N

N

N

10.1.3.3

N

N

N


Footnote 1 ASG 10.1.2.0.2 is installed by default.

Footnote 2 ASG 10.1.2.0.2 is installed by default.

Footnote 3 ASG 10.1.2.2.1 is installed by default.

Footnote 4 This is the ASG release that was provided (installed by default) with the OracleAS 10.1.4.2 release. It is compatible with the OracleAS 10.1.2.x releases. There is no OracleAS 10.1.2.2.1 release.

15.1.2 Compatible ASG Releases in an OracleAS Disaster Recovery Topology

This chart shows which ASG release combinations are compatible in in an OracleAS Disaster Recovery topology. A topology is a collection of Oracle Application Server instance homes and standalone host homes that combine to comprise the OracleAS Disaster Recovery production site and standby site. Each Oracle Application Server home has a specific ASG release installed, either by default or by an ASG standalone installation. Since OracleAS Disaster Recovery operations are distributed, there is a collection of recommended ASG releases that should be installed across the topology. Example for the EDG deployment, the collection of Oracle homes should be upgraded to ASG version 10.1.3.3 or 10.1.2.2.x.

Use Table 15-3 to determine whether two ASG releases are compatible in an OracleAS Disaster Recovery topology. Find the first ASG release in the left column of the table and then find the second ASG release in one of the other columns of the table.

This list describes the meaning of the entries in Table 15-3:

  • Y-NR: The first ASG release is compatible with the second ASG release, but Oracle recommends that you do not use this ASG release combination in your topology.

  • Y: The first ASG release is compatible with the second ASG release. Oracle recommends that you use this ASG release combination in your topology.

Table 15-3 shows which ASG releases are compatible with other ASG releases.

Table 15-3 Compatible ASG Releases in a Topology

ASG Release 10.1.2.0.2 10.1.2.2 10.1.2.2.1 10.1.3.0 10.1.3.1 10.1.3.3

10.1.2.0.2

Y-NR

Y-NR

Y-NR

Y-NR

Y-NR

Y-NR

10.1.2.2

Y-NR

Y

Y

Y-NR

Y-NR

Y

10.1.2.2.1

Y-NR

Y

Y

Y-NR

Y-NR

Y

10.1.3.0

Y-NR

Y-NR

Y-NR

Y-NR

Y-NR

Y-NR

10.1.3.1

Y-NR

Y-NR

Y-NR

Y-NR

Y-NR

Y-NR

10.1.3.3

Y-NR

Y

Y

Y-NR

Y-NR

Y


15.1.3 Adding an Instance from a Remote Client Adds an Instance on the Local Instance and Not on the Remote Instance

When using the asgctl add instance command, the OracleAS Guard client must be run from a system that is already included in the topology.

For example, when an OracleAS Guard client is connected to the OracleAS Guard server that is to be added to an existing topology, the following error is returned:

ASG_IAS-15785: ERROR: The topology is missing the instance that exists in the home 
where the ASG server is running. 
You must first discover or add the instance in home

The workaround to this problem is to use an OracleAS Guard client from a system that is already included in the topology to perform the asgctl add instance command to add an instance to the topology.

15.1.4 Switchover Operation in an Asymmetric Topology Requires All Components to be Shutdown on Instances on the Primary Site that Do Not Have a Standby Peer

Prior to performing an asgctl switchover operation in an asymmetric topology for instances that do not have a standby peer, you must perform an opmnctl stopall command to shutdown all components on each of these ignored instances on the primary site.

When an XML policy file is in use for an asymmetric topology and has the <instanceList successRequirement ="Ignore" set for an instance, for example, as shown in the following example, then in a switchover operation OracleAS Guard ignores that instance:

.
.
.
<instanceList successRequirement = "Ignore">
  <instance>instance B</instance>
</instanceList>
.
.
.

OracleAS Guard, on a switchover operation, shuts down all components on the old primary site except for OracleAS Guard and OPMN and ignores instance B because the policy file specifies to do so. The switchover operation fails because all components are not shut down on the primary site, in this case instance B because the policy file specifies to ignore instance B on the primary site, which has no standby peer.

To workaround this problem, the OracleAS Disaster Recovery Administrator must perform an opmnctl stopall operation for all components on instance B prior to the switchover operation in order for the switchover operation to succeed in this asymmetric topology.

15.1.5 HTTP Server Configuration When Using a Server Load Balancer

If you are using a Server Load Balancer to direct HTTP requests to multiple Oracle HTTP Server instances, Web access to some applications (such as the Application Server Control console and Oracle Web Services Manager) may be redirected to the physical HTTP Server hosts.

To ensure that redirected requests are always sent to the load balancer, configure an Oracle HTTP Server virtual host for the load balancer.

For example, if Oracle HTTP Server is listening on port 7777 and a load balancer called bigip.acme.com is listening on port 80, then consider the following entry in the httpd.conf file:

NameVirtualHost *:7777 
<VirtualHost *:7777> 
ServerName bigip.us.oracle.com 
Port 80 
ServerAdmin youyour.address 
RewriteEngine On 
RewriteOptions inherit 
</VirtualHost> 

15.1.6 Problem Performing a Clone Instance or Clone Topology Operation

At the current time, the semantics of an asgctl clone topology operation will not clone databases that are outside of the Oracle Application Server home, thus only the default database installed into the Oracle Application Server home by some infrastructure installation types will be cloned. The asgctl create standby database command should be used by users not familiar with Oracle Data Guard.

15.1.7 OracleAS Guard Release 10.1.2.1.1 Cannot Be Used with Oracle RAC Databases

OracleAS Guard version shipped with this release is 10.1.2.1.1. This version of OracleAS Guard cannot be used with Oracle RAC Databases. For all other purposes, this OracleAS Guard version is completely supported by Oracle.

To use OracleAS Guard with an Oracle RAC database, it is recommended to use Release 10.1.2.2 stand alone version of OracleAS Guard with this release. OracleAS Guard 10.1.2.2 version (with instructions) is available for download from Oracle OTN as an OracleAS Guard stand alone install, or please contact Oracle Support for further instructions.

15.1.8 OracleAS Guard Returned an Inappropriate Message When It Could Not Find the User Specified Database Identifier

When adding an Oracle RAC instance to the topology using the OracleAS Guard add instance command and OracleAS Guard could not find the user specified identifier, an inappropriate error message was returned. If the user had entered the database name rather that the Oracle instance SID, there was no indication that this was the problem.

Now if OracleAS Guard is unable to locate the oratab entry (on UNIX) or the system registry service (on Windows) for the user specified database identifier, the following ASG_SYSTEM-100 message now precedes the existing ASG_DUF-3554 message and both messages will be displayed to the console:

ASG_SYSTEM-100: An Oracle database is identified by its database unique name (db_name)
ASG_DUF-3554: The Oracle home that contains SID <user specified identifier> cannot be found

15.1.9 Database Instance on Standby Site Must Be Shutdown Before Issuing an asgctl create standby database Command

Oracle recommends that you shut down the database on the standby site if it is up and running before issuing the asgctl create standby database command; otherwise, the following error is returned:

ASG_DGA-12500: Standby database instance "<instance_name>" already exists on host "<hostname>"

15.1.10 Problem in an Oracle RAC-non Oracle RAC Environment with Naming Conventions

There is a problem with the naming conventions used in the Oracle RAC/non Oracle RAC environment. The asgctl set primary database command must be issued for both the primary and standby site within asgctl to define the service name mapping within OracleAS Guard before attempting an asgctl create standby database command; otherwise, the following error message is returned.

ASG_DUF-4902: Object not found in clipboard for key "orcl1keySourceDb". 

15.1.11 In an Oracle RAC-non Oracle RAC Environment, an asgctl create standby database Operation Returns an Inappropriate Error When the Database Is Already in a Physical Standby State

An error ora-01671 will occur, when attempting to perform an asgctl create standby database operation from a database that is already in 'physical standby' state. An appropriate error message should be echoed indicating that a standby database is already running, rather than returning this error. This is a known issue.

15.1.12 GNU Tar is Required for ASG Clone Topology or Clone Instance Operations

When using the ASG clone topology or clone instance operations, the tar utility is utilized. The target system(s) of these operations must have a version of GNU tar in the default PATH of the system user account in which the standalone ASG install runs.

GNU tar can be obtained at the following location:

http://www.gnu.org/software/tar

15.1.13 ASG Operations Fail if Multiple DB ORACLE_HOMEs Exist on the Same System

If you have multiple ORACLE_HOME directories on the same system in a disaster recovery setup, the set primary database command fails with the following error:

prodnode1: -->ASG_DUF-4950: An error occurred on host "stama03v1" with IP  "XXX.XX.XX.XXX" and port "7890" 
prodnode1: -->ASG_ORACLE-300: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor 
prodnode1: -->ASG_DUF-3700: Failed in SQL*Plus executing SQL statement:
connect sys/******@rac as sysdba;.
prodnode1: -->ASG_DUF-3502: Failed to connect to database orcl.
prodnode1: -->ASG_DUF-3027: Error while executing  at step - Default Step.
.
The database credentials have been set successfully, but they have not been
validated 

To work around this issue, make sure you have only one database on the system sharing the same inventory for a disaster recovery setup.

15.1.14 Placing the Oracle BPEL Process Manager Dehydration Store in a Database Managed by OracleAS Guard

If you are running Oracle BPEL Process Manager in an OracleAS Disaster Recovery topology, you want to ensure that:

  • the Oracle BPEL Process Manager dehydration data stored in databases at the primary and standby sites are continuously synchronized

  • when a switchover operation occurs, Oracle BPEL Process Manager uses the database at the standby site.

To achieve this:

  • Store the dehydration data in a database.

  • Set up log archiving and set Oracle Data Guard protection of the standby database at the maximum availability mode (instead of at the maximum protection mode). This allow logs to be applied continuously at the standby site without shutting down the primary database if the standby database is taken offline.

    Note:

    You should not configure Oracle Data Guard to maximum protection mode because, by definition, this mode takes the primary database offline if the standby database is taken offline.

    By default, the ASG "create standby database" command configures the standby database in maximum performance mode. You need to change the mode to maximum availability, as shown in step 2 below.

  • Include the database in the OracleAS Disaster Recovery topology. You want this database to be monitored by OracleAS Guard, even if Oracle Application Server was not utilized to set up Oracle Data Guard. This ensures that the database and related services are failed over in coordination with the Oracle Application Server services.

To create a standby database and put it in maximum availability mode using the ASG "create standby database" command, perform these steps:

  1. Create a database on the standby site and include it in the OracleAS Disaster Recovery topology. Here are two ways in which you can achieve this:

    • Install a database manually on the standby site and then run the ASG "instantiate topology" command on the primary site.

      ASGCTL> instantiate topology to standbynode1
      

      standbynode1 refers to the physical name of the standby node. The standby node has a corresponding node on the primary site.

      For details on ASG commands and OracleAS Disaster Recovery, see the Oracle Application Server High Availability Guide.

    OR

    • Run the ASG "create standby database" command on the primary node to create a standby database on the standby node. The command configures the primary database for maximum performance and defines log_archive_dest_n with the "LGRW SYNC AFFIRM" archive attributes for the standby database.

      ASGCTL> create standby database orcl1 on standbynode1
      

      orcl1 refers to the database name, and standbynode1 refers to the physical name of the standby node.

  2. Upgrade the configuration to maximum availability mode.

    1. On the primary database, run the following SQL command:

      SQL> alter database set standby database to maximize availability;
      
    2. On the standby database, run the following SQL command to place the standby database in managed recovery mode. (This places the standby database in a constant state of media recovery.)

      Note:

      Configuring the standby database for managed recovery is not a requirement of maximum availability, but it provides for shorter failover times.
      SQL> alter database recovery managed standby database
            [disconnect from session];
      

      Add the optional "disconnect from session" part if you want to end the session after the command.

Notes:

  • In the steps above, the Oracle Data Guard protection mode of the primary database changes from maximum performance to maximum availability.

    For details on the different Oracle Data Guard protection modes (maximum protection, maximum availability, and maximum performance), see the Oracle Data Guard Concepts and Administration guide.

  • Running the primary database in maximum availability mode may cause a "hang" waiting for an available online log file. A maximum availability primary database will not reuse an online log file until it has been archived to the standby database. This could happen if the standby database is taken offline for a long time.

  • Only data with same synchronization requirements should be stored in the same database.

    For example, the Oracle BPEL Process Manager dehydration store and the OracleAS Portal data should be stored in separate databases because the synchronization objectives of Oracle BPEL Process Manager and OracleAS Portal are different. The synchronization objective of Oracle BPEL Process Manager dehydration store is to maintain consistency between the dehydration store and the BPEL process, while the synchronization objective of OracleAS Portal is to ensure that data and configuration maintained within the middle tier and database do not diverge.

Actions Performed by the "sync topology" Command

When you run the ASG "sync topology" command in this configuration, it performs the following:

  • Performs a log switch at the primary and ensures that the log is shipped and archived

  • Performs process management at the primary and standby sites

  • Encapsulates the incremental changes for all the data in the Oracle homes

  • Restores the standby peers to the configuration level of the primary

  • Propagates the changes to all standby instances

  • For standby databases:

    • With managed recovery running, the "sync topology" command simply reports the sync-scn and the current database-scn of the standby database. For this configuration, the standby database-scn is guaranteed to be beyond the sync-scn. ASG logs the sync scn level as it corresponds to the current scn level of the standby database.

    • Without managed recovery running, the "sync topology" command recovers the standby database to the sync-scn. It is equivalent to running the following command:

      alter database recover managed standby database until change <sync-scn>
      

15.1.15 Application Server Components Tested and Certified with OracleAS Guard

OracleAS Guard has been tested and certified with most of the components in the Oracle Application Server 10g SOA Release (the only components not verified with ASG are BAM and Registry.

For some specific recommendations for BPEL PM and ESB configuration with OracleAS Guard see the OracleAS Guard Release Notes.

15.1.16 ASG to Catch Array Overflow Exceptions in Queries to Primary

The asgctl create standby database command may fail with an ASG_DUF-4950 error and an error message stack on the console similar to the following:

ASG_DUF-4950: An error occurred on host "myhost" with IP "1.1.1.1" and port "7890"
ASG_SYSTEM-100: 10
ASG_DUF-4900: An exception occurred on the server.
ASG_DGA-13009: Error during Create Physical Standby

In the duf_client.log file on the primary site DB node, the following messages would be found:

java.lang.ArrayIndexOutOfBoundsException: 10
at oracle.duf.DufJdbc.queryRedoLogInfo(DufJdbc.java:535)
at oracle.duf.DufDb$jdbc.queryRedoLogInfo(DufDb.java:4966)
at oracle.duf.DufDb$jdbc.access$2000(DufDb.java:4884)
at oracle.duf.DufDb.queryRedoLogInfo(DufDb.java:3439)

This problem occurs because ASG does not properly handle a database configured with 10 or more redo log files.

To avoid this problem, reduce the number of database redo log files to fewer than 10.

This problem is fixed in release 10.1.3.3.

15.1.17 The create standby Command Fails if the Redo Log Files Directories do not Exist at the Standby

The asgctl create standby database command fails if the redo log files directories on the production site do not exist at the standby.ASG expects the target redo log directory structure to be symmetrical with the redo log directory structure on the production site. If they do not exist in the same directories, a failure will result with the following output:

ASG_ORACLE-300: ORA-00301: error in adding log file '/PATH/redo010.log' - file cannot be created
ASG_ORACLE-300: ORA-27040: file create error, unable to create file
ASG_DUF-3700: Failed in SQL*Plus executing SQL statement:  ALTER DATABASE ADD STANDBY LOGFILE GROUP 10
'/PATH/redo010.log' SIZE 52428800 /* ASG_DGA */;.
ASG_DUF-3535: Failed to create standby redo log.
ASG_DUF-3535: Failed to create standby redo log.
ASG_DGA-13011: Error during Create Physical Standby: Finish-configure standby.
ASG_DUF-3027: Error while executing Creating physical standby database - finish phase at step - finish step.

To avoid this problem, create the target redo log file directories on the standby site in the same directories as the production site.

15.1.18 Corrupt Index Blocks in Metadata Repository Databases

After an ASG switchover or failover operation, metadata index block corruption may occur in a Disaster Recovery metadata repository database.

For a full description of the problem and an explanation of how to deal with it, refer to OracleMetLink note 386830.1 at:

https://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=386830

15.1.19 Database SIDs Must be the Same for Database Peers at Primary and Standby Sites

The SIDs must be the same for database peers at a primary site and standby site(s) in a Disaster Recovery topology.

15.1.20 Use All Uppercase Characters for Database Initialization Parameters to Avoid Instantiate and Sync Problems

Use all uppercase characters for database initialization parameters.

In the following example, the database initialization parameter, service, that is used in the archive log destination parameter is in all uppercase characters (SERVICE):

log_archive_dest_2="SERVICE=SIDM valid_for=(online_logfiles,primary_role) 
db_unique_name=SIDM"

But in the following example, the database initialization parameter, service, that is used in the archive log destination parameter is in lowercase characters (service):

log_archive_dest_2="service=SIDM valid_for=(online_logfiles,primary_role) 
db_unique_name="SIDM"

When the database initialization parameter is not in all uppercase characters, error messages similar to the following can occur during an instantiate topology or sync topology operation:

stajo05: -->ASG_DUF-4950: An error occurred on host "stajo05" with IP
"140.87.25.33" and port "7890"
stajo05: -->ASG_SYSTEM-100: String index out of range: -9
stajo05: -->ASG_DUF-3760: Failed to query archive log destination
information.
stajo05: -->ASG_IAS-15753: Error preparing to instantiate the topology on
host "stajo05"
stajo05: -->ASG_DUF-3027: Error while executing Instantiating each instance
in the topology to standby topology at step - prepare step.

15.1.21 Workaround for ASG_DUF-3800 "Failed trying to connect to the OPMN Manager" Error

When using the 10.1.2.2 ASG standalone kit to update a 10.1.4.x Oracle home, ASG installed a copy of optic.jar in its component path that is not compatible with the optic.jar of the Oracle home. Although this will not affect the operation of the runtime system, it could potentially cause problems with some future ASG operations in certain configurations.

To prevent these potential problems:

  1. Rename optic.jar in the $ORACLE_HOME/dsa/jlib directory to optic.jar.orig on all nodes of the topology.

  2. Restart the ASG component, if started.

If the above workaround is not performed, you may receive the ASG_DUF-3800 error "Failed trying to connect to the OPMN Manager" error message during a switchover topology operation if your Disaster Recovery configuration includes Oracle Application Server release 10.1.4.0.1 Identity Management and the standalone ASG kit for release 10.1.2.2.

The error messages received may be similar to:

"4-May 12:51:20  stamx12 152.68.64.214:7890(home/opt/maa/oracle/product/10.1.4/im)
4-May 12:51:20      Running opmnctl reload command:
"/opt/maa/oracle/product/10.1.4/im/opmn/bin/opmnctl reload".
4-May 12:51:25  stamx11: -->ASG_DUF-4950: An error occurred on host "stamx11"
with IP "140.87.21.201" and port "7890"
stamx11: -->ASG_DUF-4950: An error occurred on host "stamx11" with IP
"152.68.64.213" and port "7890"
stamx11: -->ASG_DUF-3800: Failed trying to connect to the OPMN Manager.
stamx11: -->ASG_DUF-3027: Error while executing Starting each instance in the
topology at step - starting instance step.
stamx11: -->ASG_DUF-3027: Error while executing Switchover each instance in
the topology to standby topology at step - starting and resyncing instance
step."

15.1.22 Use the Same Port for ASG on the Production and Standby Sites to Avoid clone instance Operation Problems

Use the same port for ASG on the primary site and standby site(s) to avoid error messages such as the following during a clone instance operation:

3-May 15:45:43  >>clone instance prodsso1 to stbyinfra1
3-May 15:45:43  stamx11: -->ASG_DUF-4950: An error occurred on host
"stamx11" with IP "140.87.21.201" and port "7890"
stamx11: -->ASG_DUF-3601: Error connecting to server host 152.68.64.213
on port 7890
stamx11: -->ASG_DUF-3512: Error creating remote worker on node 152.68.64.213:7890.

The dsa.conf file contains ASG configuration information, and it is configured into the Application Server instance's backup/restore IP configuration. The dsa.conf file configuration is handled symmetrically between Application Server instances. Due to this, the dsa.conf file from a production site's instance will be syncronized to the corresponding standby site's instance.

The port numbers between the production and standby instance pairings should match for ASG.

15.1.23 Use Fully Qualified Path Names with the add instance Command

As a best practice, use fully qualified path names with the add instance command.

15.1.24 ASG Cloning is Not Supported when the Number of Oracle Homes is Different at the Primary and Standby Hosts

The ASG clone topology and clone instance commands are not supported by DR configurations if there are a different number of Oracle Homes at the primary and standby hosts.

As part of the cloning operation, the Oracle Inventory for each host is cloned. Therefore, the assumption is that the Oracle Home configuration is symmetrical for any host that is being cloned.

For a full description of supported Disaster Recovery asymmetric topologies, refer to Section 5.1.3.2 of the Application Server High Availability Guide for release 10.1.3.2.0.

15.1.25 Start ESB Repository before Starting ESB Runtime Instances

In a distributed ESB environment where ESB Repository Server and ESB Runtime Server are in different Oracle homes, you need to start the ESB Repository before starting the ESB runtime instances.

15.1.26 Use Static Routing for ESB Servers

The URL of an Oracle Enterprise Service Bus (ESB) service endpoint looks similar to the following:

http://esb-runtime-server:7777/event/ESBSystemName/ESB_ServiceName

The /event is the context root ESB runtime engine server, which is a J2EE application. Behind the ESB server, there can be one or more ESB systems, and each system can contain one or more ESB services. Furthermore, you can configure one ESB Server in one OC4J container to house ESBSystem1 and ESBSystem2, and another ESB Server in another OC4J container to house ESBSystem3 and ESBSystem4, and so on.

A problem arises in that these ESB servers are on the same OPMN cluster and behind the same HTTP Servers. A call to http://virtualurl:80/event/ESBSystem1/ESBServer1 may be routed by Oracle HTTP Server to an ESB Server that is not associated with the specified ESB system and an error may occur.

Workaround

You can use static routing to work around this problem. Edit the Oracle HTTP Server mod_oc4j.conf file and add static mount points pointing to each specific OC4J that provides a service. For example, if OC4J instances 1 and 3 provide service 1, and OC4J instance 2 provides service 2, you would have mount directives like these:

Oc4jMount /event/ESBSystemName/ESB_Service1/* instance://ias_instance_1:home
Oc4jMount /event/ESBSystemName/ESB_Service1/* instance://ias_instance_3:home
Oc4jMount /event/ESBSystemName/ESB_Service2/* instance://ias_instance_2:home

Note:

For more information on mount point configurations, refer to the section on the Oc4jMount directive in Oracle HTTP Server Administrator's Guide 10g (10.1.3.1.0), Chapter 7, "Understanding Modules".

15.2 Configuration Issues and Workarounds

This section describes configuration issues and their workarounds. It includes the following topics:

15.2.1 The asgctl shutdown topology Command Does Not Shut Down an MRCA Database That is Detected To Be of a repCa Type Database

The asgctl shutdown topology command only handles non-database instances. Thus, in a repCA environment when OracleAS Guard detects an instance and determines it to be a repCa type database, its instance is ignored in a shutdown topology operation. Any repCA type database is considered to be managed outside of OracleAS Guard.

Therefore, within an environment where an MRCA database has been added to the topology, the database will not be handled by the asgctl shutdown topology command.

15.2.2 Only One Oracle RAC Node with an Instance on the New Primary Site Is Started Up Following an asgctl switchover Operation

In a Disaster Recovery environment that involves Oracle RAC databases, after a switchback operation(switchover topology to <primary site>), the database will be started up on only one of the Oracle RAC nodes by OracleAS Guard; however, the remaining Oracle RAC instances on the primary site must be started up manually.

15.2.3 An asgctl add instance Operation from a Remote Client Adds an Instance on the Local System Rather than on the Intended Remote System

After performing an asgctl add instance operation from a remote client, it was noted that the instance was being added to the local system rather than to the intended remote system.

As a workaround to this problem, the Disaster Recovery Administrator must first perform an asgctl discover topology operation on the local client system before attempting to perform an asgctl add instance operation to add an instance to the remote system.

15.2.4 Connecting to an OracleAS Guard Server May Return an Authentication Error

When a user connects to an OracleAS Guard server and gets an authentication error even though the correct user name and password were entered, the user should try to put the following flag in the dsa.conf file in the <ORACLE_HOME>/dsa directory and try the operation again: dsa_realm_override=1.

Note that this DSA configuration file parameter is not documented in the "OracleAS Guard Configuration File Parameters" section of the OracleAS Guard Release Information readme.txt file.

15.2.5 All emagents Must Be Shut Down Before Performing OracleAS Guard Operations

Before performing any OracleAS Guard operations, you must shut down the emagents. This operation is required for OracleAS Guard commands that recycle OracleAS services. You can issue the asgctl run command in a script to perform this operation from within OracleAS Guard. See the OracleAS Disaster Recovery chapters in the Oracle Application Server High Availability Guide for more information.

Otherwise, for example you may get an ORA-01093: ALTER DATABASE CLOSE only permitted with no sessions connected error message.

Shutting down emagents is only described for performing a switchover operation. However, it applies to all OracleAS Guard operations. The documentation will be updated in a future release.

15.2.6 Procedure to Patch a 10.1.2.0.0 Disaster Recovery Setup with a 10.1.2.1.0 Patchset

Assuming you already have an existing Disaster Recovery Setup for a 10.1.2.0.0 production database, follow these conceptual steps to apply a 10.1.2.1.0 Disaster Recovery Patchset:

  1. Break the Disaster Recovery setup. Perform an asgctl failover command.

  2. Apply the patch 10.1.2.1.0.

  3. Recreate the Disaster Recovery setup. Perform an asgctl create standby database command followed by an asgctl instantiate topology command. Alternatively, see the Oracle Data Guard documentation for more information about how to reestablish the standby database.

15.2.7 Running Instantiate Topology Across Nodes After Executing a Failover Operation Results in an ORA-01665 Error

If you attempt to perform an asgctl instantiate topology operation immediately following an asgctl failover operation, an ORA-01665: control file is not a standby control file error message is returned.

To work around this problem, you must first perform an asgctl create standby database command to create the standby database on the remote host. See Section 15.3.1, "Availability of a Previously Undocumented asgctl Command: create standby database" for more information about this previously undocumented asgctl command. Also see Section 15.2.6, "Procedure to Patch a 10.1.2.0.0 Disaster Recovery Setup with a 10.1.2.1.0 Patchset" for more information.

15.2.8 OracleAS Guard Is Unable to Shutdown the Database Because More Than One Instance of Oracle RAC is Running

When you are running OracleAS Guard in an Oracle RAC environment, you should have only one Oracle RAC instance running while performing OracleAS Guard operations. Otherwise, an error will occur where the primary database will complain that it is mounted by more than one instance, which will prevent a shutdown.

For example, when performing an OracleAS Guard create standby database operation in an Oracle RAC environment with more than one Oracle RAC instance running, the following error will be seen:

ASGCTL> create standby database orcl1 on stanb06v3 
.
.
.
      This operation requires the database to be shutdown. Do you want to
continue? Yes or No 
y 
      Database must be mounted exclusive 
stanb06v1: -->ASG_DUF-4950: An error occurred on host "stanb06v1" with IP
 "141.86.22.32" and port "7890" 
stanb06v1: -->ASG_DUF-3514: Failed to stop database orcl1.us.oracle.com. 
stanb06v1: -->ASG_DGA-13002: Error during Create Physical Standby: 
Prepare-primary processing. 
stanb06v1: -->ASG_DUF-3027: Error while executing Creating physical standby
 database - prepare phase at step - primary processing step. 

15.2.9 Add Instance Adds an Instance to Topology with Empty Instancename

When you create a new database instance using the Database Configuration Assistant (DBCA), the SID defaults to the database name. If you enter a name in the SID field other than the database name, and then later add this database to the Disaster Recovery topology, the instancename in the topology added is empty.

To avoid this problem, make sure the DBName (without domain) and DBSID are the same when creating the database for a Disaster Recovery setup on primary or standby sites.

15.2.10 Create Standby Fails with if Initiated on a Different ASGCTL Shell

The create standby database command fails if initiated by ASG clients from any node other than the source primary node where the database resides. To avoid the problem, run the create standby command from the same primary (source) node, where the database for the primary site resides.

For example, if you ran the create standby command from the production database to the standby database where prodnode1 is the primary site database nodename and standbynode1 is its standby database nodename. The ASGCTL shell should always be invoked and connected to prodnode1. If you try to run ASGCTL shell from standbynode1 and connect to prodnode1, the create standby command fails.

15.2.11 Heartbeat Failure After Failover in Alert Logs

The following warning appears in the alert logs of the database after a failover scenario, where the new primary database fails to tnsping its remote database instance.

Errors in file c:\oracle\product\10.2.0\admin\orcl\udump\orcl1_rfs_1816.trc:
 ORA-16009: remote archive log destination must be a STANDBY database
 .
 Fri Sep 08 09:11:13 2006
 Errors in file c:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_arc1_496.trc:
 ORA-16009: remote archive log destination must be a STANDBY database
 .
 Fri Sep 08 09:11:13 2006
 PING[ARC1]: Heartbeat failed to connect to standby 'orcl1_remote1'. Error is 16009.
 Fri Sep 08 09:11:50 2006
 Redo Shipping Client Connected as PUBLIC
 -- Connected User is Valid
 RFS[67]: Assigned to RFS process 628 RFS[67]: Database mount ID mismatch [0x4342404d:0x4341ffb0]
 Fri Sep 08 09:11:50 2006
 Errors in file c:\oracle\product\10.2.0\admin\orcl\udump\orcl1_rfs_628.trc:
 ORA-16009: remote archive log destination must be a STANDBY database
 .
 Redo Shipping Client Connected as PUBLIC
 -- Connected User is Valid
 RFS[68]: Assigned to RFS process 2488
 RFS[68]: Database mount ID mismatch [0x4342404d:0x4341ffb0]
 Fri Sep 08 09:12:05 2006
 Errors in file c:\oracle\product\10.2.0\admin\orcl\udump\orcl1_rfs_2488.trc:
 ORA-16009: remote archive log destination must be a STANDBY database
 .
 Fri Sep 08 09:12:14 2006
 Errors in file c:\oracle\product\10.2.0\admin\orcl\bdump\orcl1_arc1_496.trc:
 ORA-16009: remote archive log destination must be a STANDBY database 

To avoid these error messages in the alert logs, null the log_archive_dest_2 parameter using the following commands:

alter system set log_archive_dest_2='SERVICE=null LGWR ASYNC REOPEN=60';
alter system set log_archive_dest_state_2='defer';

15.2.12 Create Standby Database Fails If Database Uses OMF Storage or ASM storage

The create standby database command fails with ASG_ORACLE-300: ORA-01276 errors if the database storage option uses OMF (Oracle Managed Files) or ASM (Automatic Storage Management).

To work around the problem, create a new database instance using DBCA on the primary site with alternate storage options before running thecreate standby database command.

15.2.13 Database Already Exists Errors During Create Standby

If you run a create standby command to overwrite an existing database, you get the following error messages:

Checking whether standby instance already exists
proddnode1: -->ASG_DUF-4950: An error occurred on host "proddnode1" with IP
"a.b.c.d" and port "7891"
standbynode1: -->ASG_DUF-4950: An error occurred on host "standbynode1" with IP
"e.f.g.h" and port "7891"
standbynode1: -->ASG_DGA-12500: Standby database instance "db102" already exists
on host "standbynode1".
standbynode1: -->ASG_DGA-13001: Error during Create Physical Standby:
Prepare-check standby.
standbynode1: -->ASG_DUF-3027: Error while executing Creating physical standby
database - prepare phase at step - check standby step.

Use the oradim -delete -sid <DBSID> command on Windows, or remove the database entry from oratab on UNIX platforms to make sure entries in the standby site for the database do not exist. Now rerun the create standby database to overwrite any existing database successfully.

15.2.14 Steps to Add a Database with OMF or ASM to ASG Topology

The asgctl create standby database command is designed to automate the creation of simple standby databases. It does not support some database options, such as the OMF (Oracle Managed Files) or ASM (Automatic Storage Management) storage options. If you plan to use the create standby database command to create a database at the standby site, create a new database instance on the primary site using DBCA (Database Configuration Assistant) without specifying the OMF or ASM storage options.

To create a new database instance with the OMF or ASM storage options, follow the instructions in the "Creating a Standby Database the Uses OMF or ASM" section of the Oracle Data Guard Concepts and Administration manual at:

http://www.oracle.com/technology/documentation/index.html

Then, after creating the database with the OMF or ASM storage options, use the asgctl add instance command to add the instance to the Disaster Recovery topology, so that it can be included in Disaster Recovery operations, such as failover and instantiate topology operations. See Section A.1.23 "OracleAS Guard Add Instance Command Fails When Attempting to Add an Oracle RAC Database to the Topology" of the Oracle Application Server High Availability Guide for release 10.1.3.2 for more information about using the asgctl add instance command.

15.3 Documentation Errata

This section describes documentation errata and omissions. It includes the following topics:

15.3.1 Availability of a Previously Undocumented asgctl Command: create standby database

The asgctl create standby database command is not documented. The following information describes this command in more detail.

The syntax for the asgctl create standby database command is as follows:

create standby database <database_name> on <remote_host>

<database_name> is the primary database unique name used to create the standby database on the remote host system.

<remote_host> is the name of the host system on which the standby database is to be created.

Oracle software and OracleAS Guard software are required to be installed on the node designated as <remote_host>.The init.ora parameter file generated for the standby database is configured assuming a non Oracle RAC enabled standby database. If the standby database is to be Oracle RAC enabled, the following initialization parameters must be defined appropriately:

  • cluster_database

  • cluster_database_instances

  • remote_listener

Users should use this command sparingly and only as needed.

15.3.2 "Oracle BPEL Process Manager Clustering" Chapter Can Be Found in the Oracle BPEL Process Manager Installation Guide

In the Oracle Application Server High Availability Guide, section 5.2.2, "Oracle BPEL Process Manager in an Active-Active Topology", contains a reference to the "Oracle BPEL Process Manager Clustering" chapter in the Oracle BPEL Process Manager Administrator's Guide. This chapter is actually located in the Oracle BPEL Process Manager Installation Guide.

15.3.3 Additional Site Switchover Information for RAC Deployments

The Oracle Application Server High Availability Guide for 10.1.3.2 includes the "Site Switchover Operations" subsection in Section 5.12.1.1 "Scheduled Outages."

In step 5 in the numbered list in the "Site Switchover Operations" subsection, the following two list elements regarding RAC deployments should be added as list elements c and d:

  1. For Oracle RAC Disaster Recovery deployments, shut down all the RAC instances and start up a single RAC node prior to the switchover.

  2. After setting up the ORACLE_HOME, ORACLE_SID, and PATH variables on the database node, start up the database node:

    DBHOME/bin/sqlplus /as sysdba
    SQL> startup
    

15.3.4 Configuring Jgroups and OC4J Groups in BPEL Active-Active Topology

Section 5.2.2, "Oracle BPEL Process Manager in an Active-Active Topology," in Oracle Application Server High Availability Guide 10g Release 3 (10.1.3.1.0) does not include the required Jgroups and OC4J groups configuration recommendations. Refer to Oracle Application Server Enterprise Deployment Guide 10g Release 3 (10.1.3.3.0) for complete information on configuring Jgroups and OC4J groups.