This chapter describes issues associated with Oracle Fusion Middleware high availability and enterprise deployment. It includes the following topics:
Note:
This chapter contains issues you might encounter while configuring any of the any of the Oracle Fusion Middleware products for high availability or an enterprise deployment.
Be sure to review the product-specific release note chapters elsewhere in this document for any additional issues specific to the products you are using.
This section describes general issue and workarounds. It includes the following topics:
Section 6.1.2, "mod_wl Not Supported for OHS Routing to Managed Server Cluster"
Section 6.1.4, "SOA Composer Generates Error During Failover"
Section 6.1.5, "Accessing Web Services Policies Page in Cold Failover Environment"
Section 6.1.6, "Considerations for Oracle Identity Federation HA in SSL Mode"
Section 6.1.9, "Changes to Rulesets May Not be Persisted During an Oracle RAC Instance Failover"
Section 6.1.13, "Harmless SQLIntegrityConstraintViolationException Can be Received in a SOA Cluster"
Section 6.1.14, "WebLogic Cluster WS-AT Recovery Can Put a Server into a 'Warning' State"
Section 6.1.17, "ClassCastExceptions in a SOA Cluster for the SOA Worklist Application"
Section 6.1.20, "Failover Is Not Seamless When Creating Reports in Oracle BI Publisher"
Section 6.1.23, "Cannot Save Agent When Oracle Business Intelligence Managed Server Fails Over"
Section 6.1.25, "JMS Instance Fails In a BI Publisher Cluster"
Section 6.1.27, "Undelivered Records not Recovered During RAC Failover of Singleton SOA Server"
It is highly recommended that the application tier in the SOA Enterprise Deployment topology and the WebCenter Enterprise Deployment topology is protected against anonymous RMI connections. To prevent RMI access to the middle tier from outside the subset configured, follow the steps in "Configure connection filtering" in the Oracle WebLogic Server Administration Console Online Help. Execute all of the steps, except as noted in the following:
Do not execute the substep for configuring the default connection filter. Execute the substep for configuring a custom connection filter.
In the Connection Filter Rules field, add the rules that will allow all protocol access to servers from the middle tier subnet while allowing only http(s) access from outside the subnet, as shown in the following example:
nnn.nnn.0.0/nnn.nnn.0.0 * * allow 0.0.0.0/0 * * allow t3 t3s
Oracle Fusion Middleware supports only mod_wls_ohs and does not support mod_wl for Oracle HTTP Server routing to a cluster of managed servers.
For Oracle Fusion Middleware high availability deployments, Oracle strongly recommends following only the configuration procedures documented in the Oracle Fusion Middleware High Availability Guide and the Oracle Fusion Middleware Enterprise Deployment Guides.
During failover, if you are in a SOA Composer dialog box and the connected server is down, you will receive an error, such as Target Unreachable, 'messageData' returned null.
To continue working in the SOA Composer, open a new browser window and navigate to the SOA Composer.
In a Cold Failover Cluster (CFC) environment, the following exception is displayed when Web Services policies page is accessed in Fusion Middleware Control:
Unable to connect to Oracle WSM Policy Manager. Cannot locate policy manager query/update service. Policy manager service look up did not find a valid service.
To avoid this, implement one the following options:
Create virtual hostname aliased SSL certificate and add to the key store.
Add "-Dweblogic.security.SSL.ignoreHostnameVerification=true" to the JAVA_OPTIONS parameter in the startWeblogic.sh or startWeblogic.cmd files
In a high availability environment with two (or more) Oracle Identity Federation servers mirroring one another and a load balancer at the front-end, there are two ways to set up SSL:
Configure SSL on the load balancer, so that the SSL connection is between the user and the load balancer. In that case, the keystore/certificate used by the load balancer has a CN referencing the address of the load balancer.
The communication between the load balancer and the WLS/Oracle Identity Federation can be clear or SSL (and in the latter case, Oracle WebLogic Server can use any keystore/certificates, as long as these are trusted by the load balancer).
SSL is configured on the Oracle Identity Federation servers, so that the SSL connection is between the user and the Oracle Identity Federation server. In this case, the CN of the keystore/certificate from the Oracle WebLogic Server/Oracle Identity Federation installation needs to reference the address of the load balancer, as the user will connect using the hostname of the load balancer, and the Certificate CN needs to match the load balancer's address.
In short, the keystore/certificate of the SSL endpoint connected to the user (load balancer or Oracle WebLogic Server/Oracle Identity Federation) needs to have its CN set to the hostname of the load balancer, since it is the address that the user will use to connect to Oracle Identity Federation.
In a high availability environment, if you are using online help and a failover occurs on one of the machines in your environment, your context in online help may be lost when the application is failed over.
For example, the online help table of contents may not remember the topic that was selected prior to the failover, or the last online help search results may be lost.
No data is lost, and your next online help request after the failover will be handled properly.
In Patch Set 2 of the Oracle Fusion Middleware 11g Release 1 (11.1.1) release, a new feature was added to Application Server Cluster Ready Services (ASCRS) to enable users to create an ASCRS database resource for the Oracle Database Console service. Using ASCRS to create an ASCRS database resource is described in the "Creating an Oracle Database Resource" section of the "Using Cluster Ready Services" chapter in the Oracle Fusion Middleware High Availability Guide.
This feature works on UNIX, because the Oracle Database Console can be CFC enabled on UNIX.
However, on Windows, there is no CFC support for the Oracle Database Console service. Therefore, you cannot use ASCRS to create a database resource for the Oracle Database Console service on Windows.
When you update rulesets (used in Human Workflow or BPEL) through the Worklist configuration UI or the SOA Composer application during an Oracle RAC instance failover, the new rule metadata may not get persisted to the database. In this case, you will need to perform a manual retry. However, you can continue to use the older version of metadata without any errors.
When redeploying tasks with large number of rules during an Oracle RAC instance failover, a manual retry may be needed by the end user occasionally.
In an active-active Oracle SOA cluster, when a node failure occurs, the timeout settings for request-response operations in receive activities are not propagated from one node to the other node or nodes. If a failure occurs in the server that scheduled these activities, they must be rescheduled with the scheduler upon server restart.
The scale out and scale up operations performed on your environment after re-associating the local file based WLS LDAP store with an external LDAP store will fail. To avoid this failure, follow the steps below before performing a scale up or scale out operation.
Edit the setDomainEnv.sh file located under the DOMAIN_HOME/bin directory and add the "-Dcommon.components.home=${COMMON_COMPONENTS_HOME}" and "-Djrf.version=11.1.1" variables to the file.
These variables should be added to the "EXTRA_JAVA_PROPERTIES". For example:
EXTRA_JAVA_PROPERTIES="-Ddomain.home=${DOMAIN_HOME}
-Dcommon.components.home=${COMMON_COMPONENTS_HOME} -Djrf.version=11.1.1
.
.
.
Save the file and proceed with the scale out or scale up operation.
The following SQLIntegrityConstraintViolationException can be received in a SOA cluster:
[TopLink Warning]: 2010.04.11 14:26:53.941--UnitOfWork(275924841)--Exception [TOPLINK-4002] (Oracle TopLink - 11g Release 1 (11.1.1.3.0): Internal Exception: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (JYIPS2RC4B49_SOAINFRA.SYS_C0035333) violated . . .
This is not a bug. In a cluster environment, when the messages for the same group arrive on both the nodes, one node is bound to experience this exception for the first message. The application is aware of this exception and handles it properly. It does not break any functionality.
This exception can also come on a single node after you restart the server and send the message for the existing group. Again, this exception will be experienced on the very first message.
In summary, this exception is within the application design and does not impact any functionality. It is for this reason that you do not see this exception logged as severe in the soa-diagnostic logs.
Toplink does, however, log it in its server logs.
In certain WebLogic cluster process crash scenarios, WS-AT recovery results in stuck threads that put the server into a warning state. WS-AT data recovery is successful in these cases despite the fact that the logs show failed state messages, due to the fact that commit acks are not being processed correctly for this scenario (this issue does not occur when the scenario involves the rollback of the transaction). While the server may continue to function in this warning state, the threads remain stuck until the transaction abandonment timeout, which defaults to 24 hours, is reached. The workaround is to restart the server, which removes the stuck threads and warning state. A patch for this issue can be obtained from Oracle Support.
The "Adding the I/PM Server Listen Addresses to the List of Allowed Hosts in UCM" section in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Enterprise Content Management Suite and the "Adding the I/PM Server Listen Addresses to the List of Allowed Hosts in UCM" section in the Oracle Fusion Middleware High Availability Guide describe how to add hostname-based filters for Oracle I/PM managed server listen addresses to the list of allowed hosts in Oracle UCM.
When using hostname-based filters in Oracle UCM (config.cfg file) a high latency/performance impact may be observed in the system for very intensive uploads of documents from Oracle I/PM to Oracle UCM. This is caused by the reverse DNS lookup that is required in Oracle UCM to allow the connections from Oracle I/PM servers. Using hostname-based filters is recommended in preparation for configuring the system for Disaster Protection and to restore to a different host (since the configuration used is IP-agnostic when using hostname-based filters). However if the performance of the uploads needs to be improved, users can use instead IP-based filters. To do this:
Edit the file /u01/app/oracle/admin/domainName/ucm_cluster/config/config.cfg and remove or comment out:
SocketHostNameSecurityFilter=localhost|localhost.mydomain.com|ecmhost1vhn1|ecmhost2vhn1 AlwaysReverseLookupForHost=Yes
Add the IP addresses (listen address) of the WLS_IPM1 and WLS_IPM2 managed servers (ECMHOST1VHN1 and ECMHOST2VHN1, respectively) to the SocketHostAddressSecurityFilter parameter list as follows:
SocketHostAddressSecurityFilter=127.0.0.1|0:0:0:0:0:0:0:1|X.X.X.X|Y.Y.Y.
where X.X.X.X and Y.Y.Y.Y are the listen addresses of WLS_IPM1 and WLS_IPM2 respectively. Notice that 127.0.0.1 also needs to be added as shown above.
Restart the UCM servers.
If you use the Oracle Business Process Management Suite Worklist application Actions dropdown menu to take action on a task while a failover is in progress, an exception similar to the following may be thrown:
<oracle.adf.view.rich.component.fragment.UIXInclude> <ADF_FACES-10020> <Tear
down of include component context failed due to an unhandled e
xception.
java.util.NoSuchElementException
at java.util.ArrayDeque.removeFirst(ArrayDeque.java:251)
at java.util.ArrayDeque.pop(ArrayDeque.java:480)
at
oracle.adfinternal.view.faces.context.ApplicationContextManagerImpl.popContext
Change(ApplicationContextManagerImpl.java:66)
.
.
.
In this case, the approval or rejection of the task does not go through.
To work around this problem, use either of these approaches:
Instead of using the Actions dropdown menu to take action on the task, use the TaskForm to take action.
Do a refresh after the error message. Then take the action again using the Actions dropdown menu.
ClassCastExceptions may arise in a SOA cluster for the Oracle SOA Worklist application (java.lang.ClassCastException: oracle.adf.model.dcframe.DataControlFrameImpl is reported in the logs). As a result, the Worklist application state may not be replicated to other managed servers in the cluster. The Worklist application and the corresponding user sessions will be usable after the exception is thrown, but any failovers to other servers in the cluster will not succeed.
There is no workaround to this problem.
To solve this problem, download the patch for bug 9561444, which solves the problem. Follow these steps:
To obtain the patch, log into My Oracle Support (formerly OracleMetaLink) at the following URL:
Click the Patches & Updates tab.
In the Patch Search section, enter 9561444 in the Patch ID or number is field, and enter your platform in the field after the and Platform is field.
Click Search.
On the Patch Search page, click the patch number in the Patch ID column. This causes the page content to change to display detailed information about the patch.
Click Download to download the patch.
Because of a known issue in 11.2 Oracle RAC databases, it is required to use srvctl to set up AQ notification and server-side TAF. Using DBMS_SQL packages will not work as expected.
Here is an example use of srvctl:
srvctl modify service -d orcl -s orclSVC -e SELECT -m BASIC -w 5 -z 5 -q TRUE
In the example:
orcl - Database Name
orclSVC - Service Name used by middleware component
SELECT - Failover type
BASIC - Failover method
5 - Failover delay
5 - Failover retry
TRUE - AQ HA notifications set to TRUE
Please refer to the Oracle 11.2 Oracle database documentation for detailed information about this command usage.
With Oracle I/PM and Oracle UCM file processing, some files may not get loaded in UCM properly during an Oracle RAC instance failover.
The incoming files to be processed by Oracle I/PM are put into an input folder. Oracle I/PM processes the files in the input folder and then puts them into Oracle UCM, which is backed by an Oracle RAC database. Sometimes when an Oracle RAC instance failure occurs, the retry may not happen correctly and the incoming files do not get processed. These unprocessed files show up in an error folder. These unprocessed files can manually be put back into the input folder and processed.
If you create a report in Oracle BI Publisher, and a Managed Server is failed over before the report is saved, the failover might not be seamless. For example, when you attempt to save the report, the system might not be responsive.
If this occurs, click one of the header links, such as Home or Catalog, to be redirected to the Oracle BI Publisher login page. Then, log in and create and save the report again.
In the Oracle BI Publisher layout editor, when a Managed Server is failed over, opening or creating a Web-based layout can cause the following error to appear:
Failed to load: object_name
Please contact the system administrator.
To work around this issue, close the message and click one of the header links, such as Home or Catalog, to be redirected to the login page.
When scheduling a job in Oracle BI Publisher, after a Managed Server fails over, a large popup window appears when you click Submit that shows the HTML source for the login page.
To work around this issue, close the message window and click one of the header links, such as Home or Catalog, to be redirected to the login page. You will need to re-create the report job again.
If you create an agent in the Oracle Business Intelligence Web interface, and a Managed Server fails over before you save the agent, an error occurs when you try to save the agent.
To work around this issue, log out, then log back in to Oracle Business Intelligence and create the agent again.
This issue occurs with Oracle Portal, Forms, Reports, and Discoverer 11g environments that have been upgraded from using Oracle Single-Sign On 10g to Oracle Access Manager 11g for authentication.
When performing subsequent Oracle Portal, Forms, Reports, and Discoverer 11g installations against the same environment where the initial Oracle Portal, Forms, Reports, and Discoverer 10g installation was upgraded to Oracle Access Manager, there are some requirements that must be met.
For each subsequent Oracle Portal, Forms, Reports, and Discoverer 11g installation, you must maintain the original Oracle Single Sign-On 10g instance and keep it actively running--in addition to new Oracle Access Manager 11g instance--while the additional Oracle Portal, Forms, Reports, and Discoverer 11g installations are performed.
This is necessary because Oracle Portal, Forms, Reports, and Discoverer 11g cannot be installed directly against Oracle Access Manager 11g.
After the subsequent classic installs are completed, the Oracle Single Sign-On 10g to Oracle Access Manager 11g upgrade procedure must be performed again. For more information, see "Upgrading Your Oracle Single Sign-On Environment" in the Oracle Fusion Middleware Upgrade Guide for Oracle Identity Management.
This procedure upgrades the new Oracle Portal, Forms, Reports, and Discoverer 11g instance to Oracle Access Manager 11g.
Note that these considerations apply only in an environment with Multiple Oracle Portal, Forms, Reports, and Discoverer 11g middle tiers that are installed or added to a your environment after the initial upgrade from Oracle Single Sign-On 10g to Oracle Access Manager 11g.
On rare occasions, a JMS instance is missing from a BI Publisher Scheduler cluster.
To resolve this issue, restart the BI Publisher application from the WebLogic Server Administration Console.
To restart your BI Publisher application:
Log in to the Administration Console.
Click Deployments in the Domain Structure window.
Select bipublisher(11.1.1).
Click Stop.
After the application stops, click Start.
When failover occurs, a Null Pointer Exception error window may open the second time you click Approve task during the operational approval task. (The Null Pointer Exception error window always opens during failover.) The Null Pointer Exception window does not interrupt any processes and approval succeeds.
If there is a RAC failover of a singleton server in an SOA RAC environment, you cannot use the Enterprise Manager console to recover undelivered records that are recoverable.
This section describes configuration issues and their workarounds. It includes the following topics:
Section 6.2.1, "jca.retry.count Doubled in a Clustered Environment"
Section 6.2.3, "Fusion Middleware Control May Display Incorrect Status"
Section 6.2.4, "Accumulated BPEL Instances Cause Performance Decrease"
Section 6.2.5, "Extra Message Enqueue when One a Cluster Server is Brought Down and Back Up"
Section 6.2.6, "Duplicate Unrecoverable Human Workflow Instance Created with Oracle RAC Failover"
Section 6.2.8, "No High Availability Support for SOA B2B TCP/IP"
Section 6.2.9, "WebLogic Administration Server on Machines with Multiple Network Cards"
Section 6.2.10, "Additional Parameters for SOA and Oracle RAC Data Sources"
Section 6.2.11, "Message Sequencing and MLLP Not Supported in Oracle B2B HA Environments"
Section 6.2.12, "Credentials not Propagated for Transport Protocols in B2B"
Section 6.2.13, "Create a Protected Resource for Oracle Identity Navigator"
Section 6.2.15, "Managed Server goes into Suspended Status After Oracle RAC Failover"
Section 6.2.16, "Primary/Secondary Configuration Section of the Availability Tab is Not Visible"
Section 6.2.18, "Ensuring the Oracle HTTP Server Lock File is on a Local Drive"
Section 6.2.19, "Recreating OSSO Agents that Point to the Load Balancer URL"
Section 6.2.20, "Use Lower-Case Letters for GridLink Data Source RAC Service Name"
Section 6.2.21, "Additional Steps Needed for Oracle RTD Request Forwarding to Work Correctly"
Section 6.2.22, "Error INST-08075 Occurs When Scaling Out the BI System"
In a clustered environment, each node maintains its own in-memory Hasmap for inbound retry. The jca.retry.count property is specified as 3 for the inbound retry feature. However, each node tries three times. As a result, the total retry count becomes 6 if the clustered environment has two nodes.
All the machines in a cluster must be in the same time zone. WAN clusters are not supported by Oracle Fusion Middleware high availability. Even machines in the same time zone may have issues when started by command line. Oracle recommends using Node Manager to start the servers.
In some instances, Oracle WebLogic Fusion Middleware Control may display the incorrect status of a component immediately after the component has been restarted or failed over.
In a scaled out clustered environment, if a large number of BPEL instances are accumulated in the database, it causes the database's performance to decrease, and the following error is generated: MANY THREADS STUCK FOR 600+ SECONDS.
To avoid this error, remove old BPEL instances from the database.
In a non-XA environment, MQSeries Adapters do not guarantee the only once delivery of the messages from inbound adapters to the endpoint in case of local transaction. In this scenario, if an inbound message is published to the endpoint, and before committing the transaction, the SOA server is brought down, inbound message are rolled back and the same message is again dequeued and published to the endpoint. This creates an extra message in outbound queue.
In an XA environment, MQ Messages are actually not lost but held by Queue Manager due to an inconsistent state. To retrieve the held messages, restart the Queue Manager.
As soon as Oracle Human Workflow commits its transaction, the control passes back to BPEL, which almost instantaneously commits its transaction. Between this window, if the Oracle RAC instance goes down, on failover, the message is retried and can cause duplicate tasks. The duplicate task can show up in two ways - either a duplicate task appears in worklistapp, or an unrecoverable BPEL instance is created. This BPEL instance appears in BPEL Recovery. It is not possible to recover this BPEL instance as consumer, because this task has already completed.
The following information refers to Chapter 10, "Managing the Topology," of the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.
When performing a planned stop of the Administration Server's node (rebooting or shutting down the Admin Server's machine), it may occur that the OS NFS service is disabled before the Administration Server itself is stopped. This (depending on the configuration of services at the OS level) can cause the detection of missing files in the Administration Server's domain directory and trigger their deletion in the domain directories in other nodes. This can result in the framework deleting some of the files under domain_dir/fmwconfig/. This behavior is typically not observed for unplanned downtimes, such as machine panic, power loss, or machine crash. To avoid this behavior, shutdown the Administration Server before performing reboots or, alternatively, use the appropriate OS configuration to set the order of services in such a way that NFS service is disabled with later precedence than the Administration Server's process. See your OS administration documentation for the corresponding required configuration for the services' order.
High availability failover support is not available for SOA B2B TCP/IP protocol. This effects primarily deployments using HL7 over MLLP. For inbound communication in a clustered environment, all B2B servers are active and the address exposed for inbound traffic is a load balancer virtual server. Also, in an outage scenario where an active managed server is no longer available, the persistent TCP/IP connection is lost and the client is expected to reestablish the connection.
When installing Oracle WebLogic Server on a server with multiple network cards, always specify a Listen Address for the Administration Server. The address used should be the DNS Name/IP Address of the network card you wish to use for Administration Server communication.
To set the Listen Address:
In the Oracle WebLogic Server Administration Console, select Environment, and then Servers from the domain structure menu.
Click the Administration Server.
Click Lock and Edit from the Change Center to allow editing.
Enter a Listen Address.
Click Save.
Click Activate Changes in the Change Center.
In some deployments of SOA with Oracle RAC, you may need to set parameters in addition to the out-of-the-box configuration of the individual data sources in an Oracle RAC configuration. The additional parameters are:
Add property oracle.jdbc.ReadTimeout=300000 (300000 milliseconds) for each data source.
The actual value of the ReadTimeout parameter may differ based on additional considerations.
If the network is not reliable, then it is difficult for a client to detect the frequent disconnections when the server is abruptly disconnected. By default, a client running on Linux takes 7200 seconds (2 hours) to sense the abrupt disconnections. This value is equal to the value of the tcp_keepalive_time property. To configure the application to detect the disconnections faster, set the value of the tcp_keepalive_time, tcp_keepalive_interval, and tcp_keepalive_probes properties to a lower value at the operating system level.
Note:
Setting a low value for the tcp_keepalive_interval property leads to frequent probe packets on the network, which can make the system slower. Therefore, the value of this property should be set appropriately based on system requirements.
For example, set tcp_keepalive_time=600 at the system running the WebLogic Server managed server.
Also, you must specify the ENABLE=BROKEN parameter in the DESCRIPTION clause in the connection descriptor. For example:
dbc:oracle:thin:@(DESCRIPTION=(enable=broken)(ADDRESS_LIST=(ADDRESS=(PRO TOCOL=TCP)(HOST=node1-vip.mycompany.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_ NAME=example.com)(INSTANCE_NAME=orcl1)))
As a result, the data source configuration appears as follows:
<url>jdbc:oracle:thin:@(DESCRIPTION=(enable=broken)(ADDRESS_LIST=(ADDRESS=(PRO
TOCOL=TCP)(HOST=node1-vip.us.example.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=example.com)(INSTANCE_NAME=orcl1)))</url>
<driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name>
<properties>
<property>
<name>oracle.jdbc.ReadTimeout</name>
<value>300000</value>
</property>
<property>
<name>user</name>
<value>jmsuser</value>
</property>
<property>
<name>oracle.net.CONNECT_TIMEOUT</name>
<value>10000</value>
</property>
</properties>
Message sequencing and MLLP are not supported in Oracle B2B high availability (HA) environments.
The Oracle FMW credential store maintains usernames and passwords that you define for Transport protocols. If you use the default file store for these credentials, changes you make to usernames and passwords do not propagate across nodes. You must use a central LDAP for these credentials to be synchronized across nodes in a cluster, as described in, and required by, the Oracle Fusion Middleware High Availability Guide and Enterprise Deployment Guides.
To create a protected resource for Oracle Identity Navigator, log in to the Oracle Access Manager console at http://admin.mycompany.com/oamconsole using the oamadmin account. Then proceed as follows:
From the Navigation window expand: Application Domains > IDMDomainAgent.
Click Resources.
Click Create on the tool bar below the Browse tab).
Enter the following information:
Type: http
Host Identifier: IDMDomain
Resource URL: /oinav
Click Apply.
From the Navigation window expand: Application Domains > IDMDomainAgent >Authentication Policies.
Click Protected HigherLevel Policy.
Click Edit on the tool bar below the Browse tab.
In the Resources box, click +.
From the list, select the resource /oinav.
Click Apply.
From the Navigation window expand: Application Domains > IDMDomainAgent >Authorization Policies.
Click Protected Resource Policy.
Click Edit on the tool bar below the Browse tab.
In the Resources box, click +.
From the list, select the resource /oinav
Click Apply.
Oracle recommends using the full name of the host, including the domain name, when configuring front-end hosts in Oracle Fusion Middleware high availability configurations. Use the host's full name instead of using only the host name.
For example, if myhost is the name of a frontend host in a high availability configuration, set the frontend host URL to the fully-qualified hostname, such as myhost.mycompany.com as DNS or local host name resolution files (for example, /etc/hosts) define.
The Managed Server wls_ods(x) can enter a suspended status in the following situations:
A database connection in the data source is wrong or not complete.
The host is not a fully-qualified host for the database.
To correct the status of the Managed Server wls_ods(x):
Under the data source, verify that the database connection is correct and complete with the domain.
Under the data source, verify that the host name for the database is a fully- qualified hostname with the domain.
Verify the connection by selecting the Test button.
During the system component scale out process, the Primary/Secondary Configuration section in the Availability tab of the Capacity Management page in Fusion Middleware Control may not be visible in the browser. This issue occurs when you perform the scale out process using Microsoft Internet Explorer version 7.0.5730.11.
To avoid this issue, do not use the browser Microsoft Internet Explorer version 7.0.5730.11 to scale out; use another browser such as Google Chrome.
After scaling out Oracle Business Intelligence, Server Start parameters are not getting set correctly. To work around this issue, update the Server Start parameters for the scaled out BI Managed Server to include the following:
-Dserver.group=obi arguments
If you configure an Oracle instance for Oracle HTTP Server 11g on shared storage, such as NAS, NFS, or SAN storage, you must ensure that the lock file is created on a local drive instead of the shared drive. If you do not do this, Oracle HTTP Server might experience performance problems. Perform these steps to point the LockFile directive at a local file system:
Stop the OHS instances on WEBHOST1 and WEBHOST2.
Open the file ORACLE_INSTANCE/config/OHS/ohs_name/httpd.conf in a text editor.
Find the LockFile directive, configured under both the prefork and worker MPM configuration blocks in the httpd.conf file. It looks like this:
LockFile ORACLE_INSTANCE/diagnostics/logs/COMPONENT_TYPE/COMPONENT_NAME/http_lock
Change the LockFile directive under the appropriate MPM configuration to point to a local file system, for example:
LockFile /local_disk/path/http_lock
Restart Oracle HTTP Server.
Verify that the http_lock file exists in the directory specified by the LockFile directive.
A high availability Classic environment typically has a load balancer in front of the Classic OHS instances. When you configure a classic instance with OAM 11g, the Configuration Wizard automatically configures an OSSO agent. The OSSO agent contains the individual Classic OHS instance URL. In a high availability cluster consisting of two Classic instances, the Configuration Wizard automatically configures two OSSO agents. Each OSSO agent contains the URL information of one Classic Webtier instance URL.
In a high availability cluster, you must recreate an OSSO agent that points to the load balancer URL.
To recreate an OSSO agent that points to the load balancer URL:
From the OAM console, click New OSSO Agent to open the OSSO Wizard application.
Enter the following information:
Name: Enter any name
Token Version: Use the default setting, v3.0
Base URL: Enter the load balancer URL, for example http://haqaedg04.us.example.com:7788
Admin ID: Leave blank
Admin Inf: Leave blank
Host Identifier: Keep default value from the Name field.
Auto Create Policies: Check this setting to enable it.
Copy the osso.conf file of the new OSSO agent from the OAM server to the Classic Web Instances config directory.
When you create a GridLink data source in the Configuration Wizard, you must verify that the service name on the database uses lowercase letters only and is a qualified domain name. For example, <mydbservice>.us.example.com. The Service Name field is in the Configure GridLink RAC Component Schema screen.
Note:
The Oracle RAC Service name is defined on the database; it is not a fixed name. Oracle recommends that you register/add the RAC service name with the database domain name, for example, us.example.com
Due to an Oracle RTD issue related to request forwarding, the frontend URL must be the same as the backend URL for deployments that include Oracle RTD. To set the frontend URL for Oracle RTD, perform the steps listed in the following procedures at the point indicated in the Oracle Business Intelligence EDG task flow.
After performing the steps listed in Section 5.7, "Setting the Listen Address for bi_server1 Managed Server," set the frontend URL for the bi_server1 Managed Server, as follows:
Log in to the Administration Console.
In the Change Center, click Lock & Edit.
Expand the Environment node in the Domain Structure window.
Click Servers. The Summary of Servers page is displayed.
Select bi_server1 in the Names column of the table. The settings page for bi_server1 is displayed.
Click the Protocols tab.
Click the HTTP tab.
Set the Frontend Host field to APPHOST1VHN1 (your bi_server1 Listen address).
Click Save, then click Activate Changes.
After performing the steps listed in Section 6.4.1, "Setting the Listen Address for the bi_server2 Managed Server," set the frontend URL for the bi_server2 Managed Server, as follows:
Log in to the Administration Console.
In the Change Center, click Lock & Edit.
Expand the Environment node in the Domain Structure window.
Click Servers. The Summary of Servers page is displayed.
Select bi_server2 in the Names column of the table. The settings page for bi_server2 is displayed.
Click the Protocols tab.
Click the HTTP tab.
Set the Frontend Host field to APPHOST2VHN1 (your bi_server2 Listen address).
Click Save, then click Activate Changes.
When you are scaling out the BI System using the Oracle Business Intelligence Configuration Assistant, the following error occurs:
INST-08075: Weblogic Server 10.3.6.0 is installed but Weblogic Server Temporary is used in the BI Domain.
To work around this error, perform the following steps:
Open MW_HOME/registry.xml for editing.
Locate the following line:
<component name="WebLogic Server" version="10.3.6.0" InstallDir="ORACLE_BASE/fmw/wlserver_10.3">
Change the line to the following:
<component name="WebLogic Server" version="Temporary" InstallDir="ORACLE_BASE/fmw/wlserver_10.3"
Save and close the file.
Return to the Oracle Business Intelligence Configuration Assistant and proceed past the Scale Out BI System Details screen.
Revert the entry in registry.xml back to version="10.3.6.0".
When you configure the RAC data source for OPSS, Oracle recommends using an Oracle GridLink data source type. If you decide to use a RAC multi data source, you must ensure that the first RAC instance listed in the multi data source definition is available during the first domain startup. If you do not use the first RAC instance listed, configuration fails.
If JMS messages and transaction logs are stored on an NFS-mounted directory, Oracle strongly recommends that you verify the behavior of a server restart after abrupt machine failures. Depending on the NFS implementation, different issues can arise post failover/restart.
To verify server restart behavior, abruptly shut down the node that hosts WebLogic servers while the servers are running.
If you configured the server for server migration, it should start automatically in failover node after the failover period.
If you did not configure the server for server migration, you can manually restart the WebLogic Server on the same host after the node completely reboots.
If Oracle WebLogic Server does not restart after abrupt machine failure, the following error entry may appear in server log files:
<MMM dd, yyyy hh:mm:ss a z> <Error> <Store> <BEA-280061> <The persistent
store "_WLS_server_soa1" could not be deployed:
weblogic.store.PersistentStoreException: java.io.IOException:
[Store:280021]There was an error while opening the file store file
"_WLS_SERVER_SOA1000000.DAT"
weblogic.store.PersistentStoreException: java.io.IOException:
[Store:280021]There was an error while opening the file store file
"_WLS_SERVER_SOA1000000.DAT"
at weblogic.store.io.file.Heap.open(Heap.java:168)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:88)
...
java.io.IOException: Error from fcntl() for file locking, Resource
temporarily unavailable, errno=11
This error occurs when the NFSv3 system does not release locks on the file stores. WebLogic Server maintains locks on files that store JMS data and transaction logs to prevent data corruption that can occur if you accidentally start two instances of the same managed server. Because the NFSv3 storage device doesn't track lock owners, NFS holds the lock indefinitely if a lock owner crashes. As a result, after abrupt machine failure followed by a restart, subsequent attempts by WebLogic Server to acquire locks may fail.
If it is not reasonably possible to tune locking behavior in your NFS environment, use one of the following solutions to unlock the logs and data files:
Use the WebLogic Server Administration Console to disable WebLogic file locking mechanisms for the default file store, a custom file store, a JMS paging file store, and a Diagnostics file store. To do this, see Considerations for Using File Stores on NFS in the Oracle Fusion Middleware High Availability Guide.
Manually unlock the logs and JMS data files and start the servers by creating a copy of the locked persistence store file and using the copy for subsequent operations. See the following section Unlocking Logs and Data Files Manually.
Unlocking Logs and Data Files Manually
Manually unlock the logs and JMS data files and start the servers by creating a copy of the locked persistence store file and using the copy for subsequent operations. To create a copy of the locked persistence store file, rename the file then copy it back to its original name. The following sample steps assume that transaction logs are stored in the /shared/tlogs directory and JMS data is stored in the /shared/jms directory.
cd /shared/tlogs mv _WLS_SOA_SERVER1000000.DAT _WLS_SOA_SERVER1000000.DAT.old cp _WLS_SOA_SERVER1000000.DAT.old _WLS_SOA_SERVER1000000.DAT cd /shared/jms mv SOAJMSFILESTORE_AUTO_1000000.DAT SOAJMSFILESTORE_AUTO_1000000.DAT.old cp SOAJMSFILESTORE_AUTO_1000000.DAT.old SOAJMSFILESTORE_AUTO_1000000.DAT mv UMSJMSFILESTORE_AUTO_1000000.DAT UMSJMSFILESTORE_AUTO_1000000.DAT.old cp UMSJMSFILESTORE_AUTO_1000000.DAT.old UMSJMSFILESTORE_AUTO_1000000.DAT
With this solution, the WebLogic file locking mechanism continues to protect against accidental data corruption if multiple instances of the same servers are accidently started. However, you must restart the servers manually after abrupt machine failures. File stores create multiple consecutively numbered.DAT files when they store large amounts of data. You may need to copy and rename all files when this occurs.
This section describes documentation errata. It includes the following topics:
Section 6.4.1, "Documentation Errata for the Fusion Middleware High Availability Guide"
Section 6.4.5, "Documentation Errata Affecting Multiple Enterprise Deployment Guides"
This section contains Documentation Errata for Oracle Fusion Middleware High Availability Guide and includes the topic Section 6.4.1.1, "Latest Requirements and Certification Information."
Several manuals in the Oracle Fusion Middleware 11g documentation set have information on Oracle Fusion Middleware system requirements, prerequisites, specifications, and certification information. For the latest informationon these topics, see the following documents on Oracle Technology Network:
http://www.oracle.com/technology/software/products/ias/files/fusion_certification.html
This document contains information related to hardware and software requirements, minimum disk space and memory requirements, and required system libraries, packages, or patches. It also includes information on supported installation types, platforms, operating systems, databases, JDKs, and third-party products.
This section contains Documentation Errata for Oracle Fusion Middleware Enterprise Deployment Guide for Oracle WebCenter.
It includes the following topics:
Section 6.4.2.2, "Additional Information for Discussions Forum Mulitcast to Unicast Conversion"
Section 6.4.2.3, "Additional Discussion Connection Properties Explained in Administration Guide"
In Section 8.1, "Configuring the Discussion Forum Connection" of the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle WebCenter, the link to section 8.1.3, "Creating a Discussions Server Connection for WebCenter From EM" is missing.
In section 6.14, "Converting Discussions Forum from Multicast to Unicast" of the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle WebCenter, the following information is missing from Step 3:
Step 3: Repeat steps 1 and 2 for WLS_Services2, swapping WCHost1 for WCHost2, and WCHost2 for WCHost1 as follows:
-Dtangosol.coherence.wka1=WCHost2 -Dtangosol.coherence.wka2=WCHost1 -Dtangosol.coherence.localhost=WCHost2 -Dtangosol.coherence.wka1.port=8089 -Dtangosol.coherence.wka2.port=8089
For additional Discussions Server connection properties associated with the procedure in Section 8.1.3 "Creating a Discussions Server Connection for WebCenter From EM" of the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle WebCenter, refer to section 12.3.1, "Registering Discussions Servers Using Fusion Middleware Control," in the Oracle Fusion Middleware Administrator's Guide for Oracle WebCenter.
This section contains documentation errata for Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Identity Management.
It includes the following topics:
Section 6.4.3.1, "Set -DDomainRegistrationEnabled=true when Starting Node Manager"
Section 6.4.3.2, "Ignore Empty Section in the Oracle Virtual Directory Chapter"
Section 6.4.3.3, "Installing Identity Management Sections Are Incorrectly Organized"
Section 6.4.3.4, "Errors in Instructions for Using the Guide"
Section 6.4.3.7, "Errors in Section 2.4, Shared Storage and Recommended Directory Structure"
The November 2010 edition of Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Identity Management failed to mention that, prior to starting the Node Manager that controls the WebLogic Administration Server, you must set -DDomainRegistrationEnabled=true. For example:
export JAVA_OPTIONS=-DDomainRegistrationEnabled=true
In the November 2010 edition of Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Identity Management, Section 8.1.1 in Chapter 11, "Extending the Domain with Oracle Virtual Directory is an empty section." Please ignore it.
In Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Identity Management 11g Release 1 (11.1.1.5), Part Number E12035-07, Section 4.5.5, "Installing Oracle Identity Management," should be reorganized as follows:
The content beginning with "Start the Oracle Fusion Middleware 11g Oracle Identity Management Installer" should be in a subsection, Section 4.5.5.1, entitled "Installing Oracle Identity Management 11.1.1.2."
Section 4.5.6, "Upgrading the Oracle Homes for Oracle Identity Management from 11.1.1.2 to 11.1.1.5" should be Section 4.5.5.2.
Errors exist in Section 1.6, "Using This Guide." They should be corrected as follows:
Step 11 should be:
If you are using Oracle Access Manager, follow the steps in Chapter 12, "Extending the Domain with Oracle Access Manager 11g."
Steps 11 through 18 should refer to chapters, not sections.
The LDIF file in Step 2a of Section 11.4.4, "Creating Users and Groups for Oracle WebLogic Server," is missing some line breaks. It should appear as follows:
dn: cn=orclFAUserReadPrivilegeGroup,cn=Groups,dc=mycompany,dc=com changetype: modify add: uniquemember uniquemember: cn=IDROUser,cn=Users,dc=mycompany,dc=com
In the chapters "Extending the Domain with Oracle Internet Directory" and "Extending the Domain with Oracle Virtual Directory," you are instructed run
./emctl switchOMS ReposURL
to enable the local emagent to communicate with the WebLogic Administration Server using the virtual IP address. After you have run that command, you must also perform the following tasks:
Force the agent to reload its configuration by issuing the command:
./emctl reload
Check that the agent is using the correct Upload URL using the command:
./emctl status agent
Table 2-3, Recommended Directory Structure, is missing some values in the Shared Storage column. The following table entries should have the value "Yes" in the Shared Storage column, indicating that these directories should be on shared storage:
IAM_ORACLE_HOME
ASERVER_DOMAIN_HOME
ASERVER_APP_HOME
This section contains documentation errata for Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Business Intelligence.
It includes the following topics:
Section 6.4.4.4, "Replace 10.1.1.1 with the Source IP Address of the Load Balancer"
Section 6.4.4.5, "Additional Steps Required When Configuring BI Composer"
After restarting Oracle BI Publisher when specifying the location of the configuration folder, as described in Section 6.5.3.1, "Setting the Location of the Shared Oracle BI Publisher Configuration Folder," you must copy the XML configuration file for Oracle BI Publisher from the Managed Server to the Administration Server location. Oracle BI Publisher reads its configuration from the Administration Server central location rather than from the Managed Server's configuration directory when the Managed Servers are restarted.
To do this, on APPHOST1, copy the file xmlp-server-config.xml from:
ORACLE_BASE/admin/domain_name/mserver/domain_name/config/bipublisher
to:
ORACLE_BASE/admin/domain_name/aserver/domain_name/config/bipublisher
The "Setting the Location of the Shared Oracle BI Presentation Catalog" section of the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Business Intelligence should be replaced by the following section:
Each Presentation Services instance loads the Oracle BI Presentation Catalog from the catalog location specified in Fusion Middleware Control.
Perform the following steps:
Copy your existing (locally published) Oracle BI Presentation Catalog to the shared location. An example of a locally published catalog is:
ORACLE_INSTANCE/bifoundation/OracleBIPresentationServicesComponent/ coreapplication_obipsn/catalog/SampleAppLite
You must perform this step before designating the Catalog Location from Fusion Middleware Control.
If you plan to use the SampleAppLite catalog mentioned as an example in this section as the shared catalog, make sure to copy it from APPHOST1.
Log in to Fusion Middleware Control.
Expand the Business Intelligence node in the Farm_domain_name window.
Click coreapplication.
Click Deployment, then click Repository.
Click Lock and Edit Configuration.
Specify the Catalog Location for the shared Oracle BI Presentation Catalog.
In a Windows environment, specify a UNC path name.
Click Apply.
Click Activate Changes.
The "Setting the Location of the Shared Oracle BI Repository" section of the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Business Intelligence should be replaced by the following section:
Specifying the RPD Publishing Directory
Specify a repository publishing directory for the Oracle BI repository. This location is used for propagating online repository changes in a cluster.
Perform the following steps in Oracle Enterprise Manager Fusion Middleware Control:
Log in to Fusion Middleware Control.
Expand the Business Intelligence node in the Farm_domain_name window.
Click coreapplication.
Click Deployment, then click Repository.
Click Lock and Edit Configuration.
Select Share Repository and specify the RPD Publishing Directory for the Oracle BI Repository.
In a Windows environment, you must specify a UNC path name.
Click Apply.
Click Activate Changes.
In the example file given in Section 12.3.4.2, Updating the OAM11gRequest File, replace the IP address listed for the ValListMember parameter (10.1.1.1) with the load balancer source IP.
After completing the steps in Section 8.3.9, "Configuring Oracle BI Composer," the following additional steps are required:
Log in to Oracle Enterprise Manager Fusion Middleware Control.
Expand Application Deployments in the left-hand navigation pane.
Under bicomposer(11.1.1) (bi_cluster), right-click bicomposer(11.1.1) (bi_server1) and select System MBean Browser.
Go to the following MBean:
Application Defined MBeans > oracle.adf.share.connections > Server: bi_server1 > Application: bicomposer > ADFConnections > BISoapConnection > bi-default
Set the Protocol attribute to https.
Set the Port attribute to the load balancer SSL port (443).
Click Apply.
Go to the ADFConnections MBean and select the Operations tab.
Click save, then click Invoke.
Restart the BI Composer application using Fusion Middleware Control or the Administration Console.
This section describes documentation errata that affects multiple Enterprise Deployment Guides. Any Enterprise Deployment Guide that have the documentation errata issue discussed in the release notes below should be updated as specified in that release note.
It includes these topics:
Section 6.4.5.1, "Sections on Configuring Oracle Coherence for SOA Composites Need Fixes"
Section 6.4.5.2, "Updates are Needed to Steps for Testing Server Migration"
Section 6.4.5.3, "Steps for Updating Data Sources for Server Migration Need Updates"
Section 6.4.5.4, "Clarification of the Procedure for Configuring the Analytics Collectors"
Section 6.4.5.6, "Note Missing from Configure Cluster Screens Step in Extend Domain for SOA Chapter"
Several Enterprise Deployment Guide manuals have a "Configuring Oracle Coherence for Deploying Composites" section that includes a Note like the following:
Note:
The Coherence cluster used for deployment uses port 8088 by default. This port can be changed by specifying the -Dtangosol.coherence.wkan.port startup parameter.
This Note should read as follows:
Note:
The Coherence cluster used for deployment uses port 8088 by default. This port can be changed by specifying a different port (for example, 8089) with the -Dtangosol.coherence.wkan.port and -Dtangosol.coherence.localport startup parameters. For example:
WLS_SOA1 (enter the following into the Arguments field on a single line, without a carriage return):
-Dtangosol.coherence.wka1=soahost1vhn1 -Dtangosol.coherence.wka2=soahost2vhn1 -Dtangosol.coherence.localhost=soahost1vhn1 -Dtangosol.coherence.localport=8089 -Dtangosol.coherence.wka1.port=8089 -Dtangosol.coherence.wka2.port=8089
WLS_SOA2 (enter the following into the Arguments field on a single line, without a carriage return):
-Dtangosol.coherence.wka1=soahost1vhn1 -Dtangosol.coherence.wka2=soahost2vhn1 -Dtangosol.coherence.localhost=soahost2vhn1 -Dtangosol.coherence.localport=8089 -Dtangosol.coherence.wka1.port=8089 -Dtangosol.coherence.wka2.port=8089
Several Enterprise Deployment Guide manuals have one or more subsections that describe how to test server migration.
The following Note should appear at the end of every section on testing server migration:
Note:
After a server is migrated, to fail it back to its original node/machine, stop the managed server from the Oracle WebLogic Administration Console and then start it again. The appropriate Node Manager will start the managed server on the machine to which it was originally assigned.
Several Enterprise Deployment Guide manuals have one or more subsections that describe how to update the data sources used for leasing when you configure server migration.
The following text appears in the instructions on how to update data sources for leasing as part of server migration configuration:
Use Supports Global Transactions, One-Phase Commit, and specify a service name for your database
That text should appear as follows:
Data sources do not require support for global transactions. Therefore, do not use any type of distributed transaction emulation/participation algorithm for the data source (do not choose the Supports Global Transactions option, or the Logging Last Resource, Emulate Two-Phase Commit, or One-Phase Commit options of the Supports Global Transactions option), and specify a service name for your database.
Section 6.4.16, "Configuring the Analytics" in the Oracle Fusion Middleware High Availability Guide contains content that indicates that you must configure an analytic collector cluster. In fact, there is no need to configure the collectors themselves. Instead, the procedure in this section explains how to configure the Oracle WebCenter Spaces servers to communicate with the analytic collectors.
Further, for Oracle Fusion Middleware 11g Release 1 (11.1.1.4.0), clustered analytics collectors are not supported for collecting WebCenter events.
In Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Business Intelligence, Chapter 2, "Database and Environment Preconfiguration," Table 2-2 lists the ports used in the Oracle Business Intelligence topology. The following additional information should be included in the table above the row containing "Database Access:"
Type: Database access for BI Server and BI Publisher JDBC Data Sources
Firewall: FW1
Port and Port Range: Listening port for client connections to the listener.
Protocol/Application: SQL*Net
Inbound/Outbound: Both
Other Considerations and Timeout Guidelines: Timeout depends on all database content and on the type of process model used for BI.
Note:
This issue was fixed in the E15722-03 revision of Oracle Fusion Middleware Enterprise Deployment Guide for Oracle Business Intelligence.
The following note is missing from Step 13 of the section titled "Extending the Domain for SOA Components using the Configuration Wizard."
Note:
For asynch request/response interactions over direct binding, the SOA composites must provide their jndi provider URL for the invoked service to look up the beans for callback.
If soa-infra config properties are not specified, but the WebLogic Server Cluster address is specified, the cluster address from the JNDI provider URL is used. This cluster address can be a single DNS name which maps to the clustered servers' IP addresses or a comma separated list of server ip:port. Alternatively, the soa-infra config property JndiProviderURL/SecureJndiProviderURL can be used for the same purpose if explicitly set by users.
This note also applies to Section 11.3, "Running the Configuration Wizard on SOAHOST1 to Extend a SOA Domain to Include Oracle Service Bus" in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.
In the "Recommended Locations for the Different Directories" section of the "Preparing the File System for an Enterprise Deployment" chapter, the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle WebCenter Content implies that you can use a shared domain directory for Oracle WebCenter Content Server. A shared domain directory for a managed server with Content Server does not work because certain files within the domain, such as intradoc.cfg, are specific to each node.
To prevent issues with node-specific files, use a local (per node) domain directory for each Oracle WebCenter Content and Oracle WebCenter Content: Inbound Refinery managed server.
This note also applies to the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle WebCenter Portal.