8 Configuring High Availability for Oracle Application Development Framework

This chapter describes procedures specific to Oracle Application Development Framework (ADF).

Note:

Unlike the other chapters in this guide, this chapter is written for ADF developers rather than administrators.

This chapter includes the following topics:

See Also:

For more information on ADF, see the following:

  • Oracle ADF Key Concepts in Understanding the Oracle Application Development Framework

  • Oracle Fusion Middleware Administering Oracle ADF Applications

8.1 Oracle ADF High Availability Considerations

Fusion web applications built on the Oracle ADF technology stack are Java EE applications (and J2EE applications). This section includes the following topics:

8.1.1 Oracle ADF Scope and Session State

At runtime, ADF objects such as the binding container and managed beans are instantiated. Each of these objects has a defined life span set by its scope attribute.

For more information, see "About Object Scope Lifecycles" in the guide Oracle Fusion Middleware Developing Fusion Web Applications with Oracle Application Development Framework for more information.

8.1.2 Oracle ADF Failover and Expected Behavior

An Oracle WebLogic cluster provides application high availability. If one member of the cluster is unavailable, any other available member of the cluster is able to handle the request. This topic describes failover requirements and the expected behavior of an application in the event of a failover.

Session Failover Requirements

For seamless failover of a Fusion web application, the application must meet the following conditions:

  • The application is in a cluster and at least one member of the application cluster is available to serve the request.

  • For stateful applications, state replication is configured correctly as Section 8.2, "Configuring Oracle ADF for High Availability" describes.

  • If you use Oracle HTTP Server, the server is configured with the WebLogicCluster directive to balance among all available application instances.

  • If you use a hardware load balancer, the load balancer is:

    • Routing traffic to all available instances

    • Configured correctly with a health monitor to mark unavailable instances

    • Configured to support persistence of session state

Expected Behavior for Application Failover

If the environment is configured correctly, application users do not notice when an application instance in a cluster becomes unavailable. The sequence of events in an application failover is, for example, as follows:

  1. A user makes a request and is routed by a hardware load balancer to Instance A of the application.

  2. Instance A of the application becomes unavailable because of node failure, process failure, or network failure.

  3. The hardware load balancer marks Instance A as unavailable.

  4. The user makes a subsequent request. The request is routed to Instance B.

  5. Instance B is configured as a replication partner of Instance A and has the user's session state.

  6. The application resumes using the session state on Instance B and the user continues working without interruption.

Oracle JRF Asynchronous Web Services (Pinned Service Behavior)

When you use Oracle JRF Asynchronous Web Services, the asynchronous web service is pinned to a service and does not fail over. When you use a reliability protocol such as WS-RM, the higher-level protocol reconnects to a new server after a failure.

For more information on Oracle JRF Asynchronous Web Services, see the Domain Template Reference.

8.1.3 Configuring the ADF Application Module for Oracle RAC

When you configure the ADF application module to access a highly available database system, such as redundant databases or Oracle Real Application Clusters (Oracle RAC) as the backend, the data source must be container-defined. In this scenario, you must use a GridLink data source or a multi data source. However, from the standpoint of the application module configuration, the naming convention for the multi data source or GridLink data source is the same as it is for a non-multi data source or GridLink data source. This naming convention ensures that the correct data source is used at runtime.

To configure GridLink data sources for high availability applications, see Section 4.4, "Configuring Active GridLink Data Sources with Oracle RAC." For multi data sources, see Section 4.5.1, "Configuring Multi Data Sources with Oracle RAC."

Modifying the mdsDS Data Source URL

When you use a GridLink data source or a multi data source with a customer- provided ADF application, you must change the mdsDS data source URL in the Weblogic Server Administration Console after you start the Administration Server.

To modify the mdsDS data source URL:

  1. Log into the WebLogic Server Administration Console. Select Services then Data Sources.

  2. Select mdsDS then Connection Pool.

  3. Change the JDBC URL to specify it in the format:

    jdbc:oracle:thin:@hostname:port/service-name
    

    For example, if the JDBC URL is:

    jdbc:oracle:thin:@(DESCRIPTION=(ENABLE=BROKEN)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=slc00erterw-r.us.example.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=srv2_koala.us.example.com)))
    

    Replace the URL with the following:

    jdbc:oracle:thin:@slc00erterw-r.us.example.com:1521/srv2_koala.us.example.com
    
  4. Save the changes then restart all servers, including the Administration Server.

8.2 Configuring Oracle ADF for High Availability

To support automatic replication and failover for web applications in a clustered environment, Oracle WebLogic Server supports mechanisms for replicating HTTP session state across clusters. You can configure Oracle ADF to ensure the Fusion web application's state can be restored from any server in the cluster.

This section includes the following topics:

8.2.1 Configuring Application Modules

An application module is the transactional component that UI clients use to work with application data. It defines an updateable data model and top-level procedures and functions, called service methods, related to a logical unit of work related to an end-user task. An application module supports passivating, or storing, its transaction state as a snapshot in the database. It also supports the reverse operation of activating the transaction state from one of these saved snapshots.

For more information on application module state management, see "Introduction to Fusion Web Application State Management" in Developing Fusion Web Applications with Oracle Application Development Framework.

To enable support for ADF Business Components failover, set the jbo.dofailover parameter to true so that the application module state is saved on release. This enables Oracle ADF to restore the application module state from a snapshot saved from a previous check in. By contrast, when the failover feature is disabled, which it is by default, then application module state is saved only when the application is reused by a subsequent user session and only when the application module pool cannot find an unused application module.

You can set this parameter in your application module configuration on the Pooling and Scalability tab of the Edit Business Components Configuration dialog.

To configure application modules for high availability:

  1. Launch JDeveloper and open the application.

  2. In the Application Navigator, expand the project that contains the data model and locate the application module.

  3. Right-click the application module and select Configurations.

  4. Click Edit.

  5. Click the Pooling and Scalability tab.

  6. Select the Failover Transaction State Upon Managed Release checkbox.

  7. Click OK to close the Edit Business Components Configuration dialog.

  8. Click OK to close the Manage Configurations dialog.

8.2.2 Configuring weblogic.xml

To enable support for replicating HTTP session state, you must assign a value to the persistent-store-type element in the Oracle WebLogic Server weblogic.xml file. The value replicated_if_clustered ensures that the in-effect persistent store type will be replicated so that sessions on the clustered environment are stored in accordance with the value set for the cluster of servers to which this server belongs.

Note:

Oracle ADF applications such as the Oracle WebCenter Portal Suite are preconfigured and do not need additional configuration.

To configure the weblogic.xml file for high availability:

  1. Launch JDeveloper and open the application.

  2. In the Application Navigator, expand the project that contains the web application and expand the WEB-INF folder.

  3. Double-click the weblogic.xml file, and click the Source tab to edit the file.

  4. In the file, add the persistent-store-type definition to the session-descriptor element:

    <weblogic-web-app>
        <session-descriptor>
        <persistent-store-type>
        replicated_if_clustered
        </persistent-store-type>
        </session-descriptor>
    </weblogic-web-app>
    

8.2.3 Configuring adf-config.xml

When you design an application to run in a clustered environment, you must ensure that Oracle ADF is aware of changes to managed beans stored in ADF scopes (view scope and page flow scope).

When a value within a managed bean in either view scope or page flow scope is modified, the application must notify Oracle ADF so that it can ensure the bean's new value is replicated.

To enable ADF Controller to track changes to ADF memory scopes and replicate the page flow scope and view scope within the server cluster, you must set the ADF Controller parameter <adf-scope-ha-support> in the application's adf-config.xml file to true. For example, when set to true for an application and that application adds or removes a bean from a page flow scope during a request, the change will automatically replicated within a cluster.

The adf-config.xml file is the central configuration file for all ADF components. The file contains sections to configure the runtime behavior for ADF components, including, ADF Controller.

Note:

If your application uses MDS and will use an Oracle database that supports failover, Oracle recommends enabling MDS retry on failover. To do this, add the following retry-connection entry to the MDS configuration section of adf-config.xml.

<persistence-config>
         <metadata-namespaces>...
         <metadata-store-usages>...
         <external-change-detection enabled="false" />
         <read-only-mode enabled="true"/>
         <retry-connection enabled="true"/>
</persistence-config>

To configure the adf-config.xml file for high availability:

  1. Launch JDeveloper and open the application.

  2. In the Application Navigator, expand the Application Resources.

  3. Select Descriptors, and then select the ADF META-INF node.

  4. Double-click the adf-config.xml file, and click the Source tab to edit the file.

  5. Add the following to the file:

    <adf-controller-config xmlns="http://xmlns.example.com/adf/controller/config">
     <adf-scope-ha-support>true</adf-scope-ha-support>
    </adf-controller-config>
    

For more information about using the adf-config.xml file to configure ADF, see the chapter on creating complex task flows in the guide Developing Fusion Web Applications with Oracle Application Development Framework.

8.2.4 Configuring org.apache.myfaces.trinidad.CHECK_FILE_MODIFICATION

The org.apache.myfaces.trinidad.CHECK_FILE_MODIFICATION parameter must not be set to true when it runs in a high availability environment. Setting this context parameter to true can lead to errors after failover occurs. Note that this limitation should not affect your work because you typically use this parameter in a development environment only, not a high availability environment.

8.3 Troubleshooting Oracle ADF High Availability

This section describes procedures for troubleshooting possible issues with Oracle ADF. Topics in this section include the following:

8.3.1 Troubleshooting Oracle ADF Development Issues

When you develop the Fusion web application in Oracle JDeveloper, the integrated development environment provides support for detecting potential High Availability issues. The warnings that JDeveloper provides are generated by the audit framework and are triggered to appear in the JDeveloper source editors. The warnings the editors show are based on the audit rules for High Availability applications.

The High Availability audit rules that JDeveloper enables by default are:

  • ADF Controller Configuration - High Availability for ADF Scopes is not Enabled warns the developer that the adf-scope-ha-support flag in the adf-config.xml file is set is not set to true. This audit rule fires only when the <adf-controller-config> element is in the ADF application-level configuration file (adf-config.xml).

  • ADF Page Flows - Bean in Scope Map is Modified warns the developer when the some code calls a setter method on a bean to indicate that the code did not subsequently call the ControllerContext.markScopeDirty() method. This audit rule fire only when the adf-scope-ha-support flag in the adf-config.xml file is set to true.

  • ADF Page Flows - EL Bean is Modified warns the developer when some code evaluates an EL expression that mutates a bean to indicate that the code did not subsequently call the ControllerContext.markScopeDirty() method. This audit rule fire only when the adf-scope-ha-support flag in the adf-config.xml file is set to true.

  • ADF Page Flows - Managed Bean Class Not Serializable warns the developer that a managed bean has a non-serializable class defined in viewScope, pageFlowScope, or sessionScope. This audit rule fire only when the adf-scope-ha-support flag in the adf-config.xml file is set to true.

You can modify the High Availability audit rule settings using the Preference dialog in JDeveloper. From the JDeveloper toolbar, choose Tools - Preferences, under Audit - Profiles expand ADF Controller Configuration or ADF Pages Flows and make the desired audit rule selections.

You can also trigger the audit by choosing Build - Audit project.jpr from the JDeveloper toolbar.

8.3.2 Troubleshooting Oracle ADF Deployment Issues

There are two actions to take if you encounter ADF deployment issues: verify the JRF Runtime and verify the status of application deployments.

8.3.2.1 Verifying the JRF Runtime Installation

The first step to troubleshooting an ADF deployment is to verify that the JRF Runtime is installed on the WebLogic Server domain. ADF applications require the JRF and ADF runtime. You cannot run ADF applications with a standalone WebLogic Server domain or just the WebLogic Server portion of the Application Development product; you must extend it with the JRF extension template.

For more information on the JRF Template, see "Oracle JRF and ADF Templates" in the Oracle Fusion Middleware Domain Template Reference.

8.3.2.2 Verifying the Success of All Application Deployments

Fusion web applications deploy when the Managed Server first starts. Use the Administration Console to check that all application deployments were successful:

  1. Click Deployments in the left hand pane. The right hand pane shows the application deployments and their status. The state of all applications, assuming all the servers are running, should be ACTIVE.

  2. If an application deployment has failed, the server logs may provide some indication of why the application was not deployed successfully. The server logs are located in the DOMAIN_HOME/servers/server_name/logs directory. Common issues include:

  • Unavailability of external resources, such as database resources. Examine the error, fix it, and attempt to redeploy the application.

  • The appropriate applications or libraries are not targeted correctly to the right Managed Server or Cluster.

8.3.3 Troubleshooting Oracle ADF Replication and Failover Issues

State Replication is most prominent in failover scenarios. A user working on one server may discover that, upon failover:

  • Windows may close or the state might reset.

  • Screens may require a reset.

  • The application may redirect to the logon screen.

To diagnose and troubleshoot state replication issues.

  1. Confirm that this is not a known replication issue.

    See Section 8.1.2, "Oracle ADF Failover and Expected Behavior" for possible expected behaviors. Before proceeding to further diagnose the issue, first confirm that the failover behavior is not an expected behavior.

  2. Check load balancer settings.

    For replication and failover to function correctly, the load balancer must be configured with the appropriate persistence settings. For more details on configuring Hardware Load Balancers for Oracle WebLogic Server, see Oracle Fusion Middleware Using Clusters for Oracle WebLogic Server.

  3. Check the cluster status.

    Replication occurs within the context of a cluster. For failover to be successful, there must be at least one other healthy member of the cluster available. You can check cluster status in one of two ways:

    • Check the cluster status using the Oracle WebLogic Server Administration Console - In the Left-hand pane, click on Servers. Verify the state of all servers in the cluster.

    • Check the cluster status using the weblogic.Admin utility. You can use the weblogic.Admin utility to query the state of all servers in a specific cluster. For example:

      $ java weblogic.Admin -url Adminhost:7001 -username <username> -password
       <password>  CLUSTERSTATE -clustername Spaces_Cluster
      

      This example returns:

      There are 2 server(s) in cluster: Spaces_Cluster
      The alive servers and their respective states are listed below:
      Application Server---RUNNING
      Managed Server---RUNNING
      
  4. Check cluster communications.

    Even if all cluster members are running, there may be communication issues which prevent them from communicating replication information to each other. There are two types of cluster communication configurations. Troubleshooting depends on the cluster type:

    • Checking Unicast cluster communications - For Unicast clusters, Managed Servers must be able to access each other's hosts and each other's default listening port.

      Ensure that all individual Managed Servers have their Listen Address set correctly. You can find this setting by selecting Configuration, General for each Managed Server.

    • Checking Multicast cluster communications - For multicast clusters, servers must be able to intercept the same multicast traffic. Ensure that multicast is configured correctly by running the WebLogic utility utils.MulticastTest on each machine. For example:

      $ java utils.MulticastTest -H
      
  5. Confirm Oracle WebLogic Server application configuration.

    Oracle WebLogic Server is not configured by default for failover. In-memory replication takes place only with the proper setting in the weblogic.xml file:

    <session-descriptor>
    <persistent-store-type>replicated_if_clustered</persistent-store-type>
    </session-descriptor>
    

    A persistent-store-type of replicated is also acceptable. This setting can be made in JDeveloper, as described in Section 8.2.2, "Configuring weblogic.xml."

  6. Confirm Oracle ADF Business Components configuration.

    Oracle ADF is not configured by default for failover. Failover is supported only with the proper setting in the ADF Business Components configuration file (bc4j.xcfg):

    <AppModuleConfig ...
     <AM-Pooling jbo.dofailover="true"/>
    </AppModuleConfig>
    

    This setting is made in JDeveloper through the Edit Business Components Configuration dialog, as described in Section 8.2.1, "Configuring Application Modules."

  7. Confirm Oracle WebLogic Server connection pool parameter.

    Set an appropriate value for the weblogic-application.xml deployment descriptor parameter inactive-connection-timeout-seconds on the element <connection-check-params> pool-params.

    When enabling application module state passivation, a failure can occur when Oracle WebLogic Server is configured to forcibly release connections back into the pool. The failure creates an exception "Connection has already been closed" that gets saved to the server log. The user interface does not show this exception.

    Set inactive-connection-timeout-seconds to several minutes. In most cases, this setting avoids forcing the inactive connection timeout and passivation failure. Adjust the setting as needed for your environment.

  8. Confirm ADF Controller configuration.

    Oracle ADF is not configured by default to replicate changes to ADF objects in ADF memory scopes. ADF object replication is supported only with the proper setting in the ADF application-level configuration file (adf-config.xml):

    <adfc:adf-controller-config>
     <adfc:adf-scope-ha-support>true</adfc:adf-scope-ha-support>
    </adfc:adf-controller-config>
    

    This setting is made in JDeveloper through the source editor. See Section 8.2.1, "Configuring Application Modules."

  9. Check default logger messages.

    By default the ADF log shows high-level messages (INFO level). The default logging often reports problems with serialization and replication without the need to enable more detailed log messages.

  10. Enable log messages for ADF high availability applications.

    Configure the ADF logger to output runtime messages for high availability. By default the ADF log shows high-level messages (INFO level). You enable high availability diagnostics for ADF Controller by setting the logging level in Fusion Middleware Control to FINE.

    When enabled, the logger outputs a warning if the adfc:adf-scope-ha-support setting in the adf-config.xml file is not set.

  11. Enable debug.

    Check the server logs for any unusual messages on Managed Server startup. In particular, if the Managed Server is unable to locate other members of the cluster. The server logs are located in the DOMAIN_HOME/servers/SERVER_NAME/logs directory.

    For further debugging, enable the flags DebugCluster, DebugClusterAnnouncements, DebugFailOver, DebugReplication, and DebugReplicationDetails. You can enable each flag with the weblogic.Admin utility:

    $ java weblogic.Admin -url Adminhost:7001 -username <username> -password
     <password> SET -type ServerDebug -property DebugCluster true
    
  12. Enable component state serialization checking.

    Enable server checking to ensure no unserializable state content on session attributes is detected. This check is disabled by default to reduce runtime overhead. Serialization checking is supported by the Java server system property org.apache.myfaces.trinidad.CHECK_STATE_SERIALIZATION.

    Table 8-1 shows the options you can use with the property. Use commas to delimit the options.

    Table 8-1 CHECK_STATE_SERIALIZATION Options

    Option Description

    tree

    Checks whether the entire component tree is serializable. This is the fastest component check. Most testing should be performed with this flag enabled.

    component

    Checks each component individually for serializability. This option is much slower than "tree." It is typically turned on only after testing with "tree" reports an error. This option narrows down the problematic component.

    property

    Checks each component attribute individually for serializability. This is slower than "component" and narrows down the specific problematic component attribute after a failure has been detected in the "tree" or "component" modes.

    session

    Checks that all attributes in the JSF Session Map that are marked as Serializable are serializable.

    application

    Checks that all attributes in the JSF Application Map that are marked as Serializable are serializable.

    beans

    Checks that any serializable object in the appropriate map has been marked as dirty if the serializable content of the object changes during the request.

    all

    Checks everything.


    For high availability testing, start off by validating that the Session and JSF state is serializable by launching the application server with the system property:

    -Dorg.apache.myfaces.trinidad.CHECK_STATE_SERIALIZATION=session,tree
    

    Add the beans option to check that any serializable object in the appropriate map has been marked as dirty if the serialized content of the object has changed during the request:

    -Dorg.apache.myfaces.trinidad.CHECK_STATE_SERIALIZATION=session,tree,beans
    

    If a JSF state serialization failure is detected, relaunch the application server with the system property to enable component and property flags and rerun the test:

    -Dorg.apache.myfaces.trinidad.CHECK_STATE_SERIALIZATION=all
    

    These are Java system properties and you must specify them when you start the application server.