Skip Headers
Oracle® Fusion Applications Administrator's Troubleshooting Guide
11g Release 1 (11.1.3)

Part Number E25450-01
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
View PDF

9 Troubleshooting Oracle SOA Suite

This chapter describes common problems that you might encounter when using Oracle SOA Suite and explains how to solve them.

This chapter contains the following topics:

Some procedures in this chapter reference content in the Oracle Fusion Middleware guides. These guides describe using Fusion Middleware Control. These procedures also apply to Fusion Applications Control.

To gain insight into the log details generated by Oracle Fusion applications, see the "Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests" chapter in the Oracle Fusion Applications Administrator's Guide. Also, review the Oracle Fusion Middleware Error Messages Reference for information about the error messages you may encounter.

Note:

Some sections of this chapter describe how to set properties in the System MBean Browser of Fusion Applications Control. Some MBean properties are applicable to Oracle WebLogic Server and others are applicable to the SOA Infrastructure. Oracle WebLogic Server Mbeans are properties that impact the Java Virtual Machine (JVM) server level process that runs on the operating system. Example properties include the port on which the server is listening, and so on. These properties are agnostic to the type of application running on that server. SOA Infrastructure level Mbeans are properties that impact the SOA Infrastructure application running on the same Oracle WebLogic Server. Example properties include audit levels, transaction retries, and so on.

9.1 Introduction to Troubleshooting Oracle SOA Suite

This section provides guidelines and a process for using the information in this chapter. Using the following guidelines and process will focus and minimize the time you spend resolving problems.

Guidelines

When using the information in this chapter, Oracle recommends:

Process

Follow the process outlined in Table 9-1 when using the information in this chapter. If the information in a particular section does not resolve your problem, proceed to the next step in this process.

Table 9-1 Process for Using the Information in this Chapter

Step Section to Use Purpose

1

Section 9.2

Get started troubleshooting Oracle SOA Suite. The procedure in this section quickly addresses a wide variety of problems.

2

Section 9.3 through Section 9.9

Perform problem-specific troubleshooting procedures. These sections describe:

  • Possible causes of the problems

  • Solution procedures corresponding to each of the possible causes

3

Section 13.1

Use My Oracle Support to get additional troubleshooting information about Oracle Fusion Applications or Oracle SOA Suite. My Oracle Support provides access to several useful troubleshooting resources, including Knowledge Base articles and Community Forums and Discussions.

4

Section 13.1

Log a service request if the information in this chapter and My Oracle Support does not resolve your problem. You can log a service request using My Oracle Support at https://support.oracle.com.


9.2 Getting Started with Troubleshooting and Logging Basics for Oracle SOA Suite

SOA troubleshooting must be performed in the following scenarios:

  1. The Oracle Fusion application transaction flow has an error and it has been identified as an error in SOA (or you want to check if it is a SOA-related error).

  2. The Oracle Fusion application transaction flow has not completed and is taking longer than it should and it has been identified as stuck in SOA (or you want to check if it is stuck in SOA).

  3. The Oracle Fusion application transaction flow is not working as expected, and the wrong data is passed.

The recommended approach to troubleshooting SOA issues is as follows:

  1. Check the server and composite status log in Oracle WebLogic Server Administration Console and check if the SOA server is up and running successfully (that is, not in a failed state).

    1. Log in to Oracle WebLogic Server Administration Console.

    2. In the Domain Structure, click Deployments.

    3. In the Name column of the Deployments section, find soa-infra.

    4. Check that the State column is set to Active and the Health column is set to OK.

    5. Click soa-infra > Monitoring tab.

    6. Expand to ensure that all components have Health set to OK.

    7. Note the server name on which SOA is deployed.

    8. Go to the Domain Structure and click Environment > Servers to verify that the State column is set to RUNNING and the Health column is set to OK.

    9. Note the Listen Port column value, as this is the SOA runtime port.

  2. Check if any Oracle Fusion application incident was created for the problem encountered.

    For information about Oracle Fusion application incident processing, see the "Process for Investigating, Reporting, and Solving a Problem" section in the Oracle Fusion Applications Administrator's Guide.

  3. Check whether the event was delivered to the SOA cluster by checking the log messages in Fusion Applications Control.

    This requires the oracle.integration.platform.blocks.event logger to be set to TRACE:32 (FINEST). There are potentially many types of exceptions that can appear in the log messages on the client side that raised the event. For example:

    • If a Java class is not in the class path, then a ClassNotFoundException may appear.

    • If the Java Naming and Directory Interface (JNDI) for the context or connection factory is not configured properly, then a NamingException or FabricException may appear.

    • If the event payload XML that you create is invalid or contains invalid characters, you may receive a FabricException indicating that enqueueing of the event fails at the PL/SQL API with invalid characters, and so on.

    For information about enabling this log and troubleshooting business events, see Section 9.3.2.

  4. Check the composite instance flow. If the event was fired, then find your composite instance based on the application data.

    See Section 9.3.1 for primary key mapping to the composite instance ID.

  5. Review the composite instance flow and fault details. If the instance is in error and it is recoverable, attempt recovery using the Recovery tab of the BPEL process service engine in Fusion Applications Control.

  6. View the logs.

    If the problem still cannot be solved, increase the log level of the system to debug the transactions. See the list of loggers and log levels in the "Troubleshooting Oracle SOA Suite" appendix of Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite. To simplify troubleshooting, it is recommended that you enable the following parent loggers at the TRACE32 (FINEST) level in Fusion Applications Control:

    • oracle.soa

    • oracle.fabric

    • oracle.integration

    The oracle.wsm logger can remain set to the ERROR level where it logs the required error messages. The oracle.apps logger should be set to ALL. To change logger levels, perform the following steps:

    1. Go to Fusion Applications Control.

    2. In the navigation pane, select WebLogic Domain.

    3. Right-click a Managed Server from within the domain (each server's log levels can be independently set).

    4. Choose Logs > Log Configuration.

    5. In the Logger Name column, expand the oracle runtime loggers to display loggers such as oracle.soa.

    6. Change the logging level to TRACE:32. The change should take effect within a few minutes.

      Description of loglevel.gif follows
      Description of the illustration loglevel.gif

      Note that in a production system, setting the trace at a fine-grained level can result in a large amount of output that must be diagnosed. You can alternately use selective tracing that provides a way to get a detailed, on-disk trace selectively (for example, by user name, thereby eliminating trace output for other users).

    7. To activate selective tracing, right-click the domain under WebLogic Domain and choose Logs > Selective Tracing.

      Note that Selecting Tracing does not display as an option when you right-click an Administration Server or Managed Server and choose Logs.

    8. From the Option Name list, choose the type of selective trace (for example, based on user name), and start the trace.

    9. When the problem has been reproduced, disable the trace and view the output to narrow down the issue.

      For more information on selective tracing, see the "Configuring and Using Selective Tracing" section of Oracle Fusion Middleware Administrator's Guide.

    10. Review the error logs (from Fusion Applications Control) for more information on the error.

      Cross layer, server, and family functionality can be correlated through the execution context ID (ECID) (for example, you can look up the composite instance for a given expense report by correlating all the log entries with the ECID associated with that expense report transaction). For more information, see the "Correlating Messages Across Log Files and Components" section of Oracle Fusion Middleware Administrator's Guide.

  7. Check for incorrect configurations and any networking issues, especially around potentially incorrect settings of the following:

    • External load balancers

    • Oracle HTTP Server

    • Virtual host/IP address

    • Oracle WebLogic Server front-end URL

    • SSL host name verification settings

    For more details, see the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

  8. Verify that custom applications have been deployed on the correct servers using the correct configuration plan during deployment.

    For more details, see the "Customizing SOA Composite Applications for the Target Environment" section of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

  9. Contact Oracle Support Services.

    If the error still cannot be resolved, file a ticket with Oracle Support Services and provide the logs and information shown in Table 9-2.

    Table 9-2 SOA Log Information for Oracle Support Services

    Log Description

    Application diagnostic log

    View the following log:

    $DOMAIN_HOME/servers/domain_name/logs/apps/server_name-diagnostic.log
    

    Server diagnostic log

    View the following log:

    $DOMAIN_HOME/servers/server_name/logs/server_name-diagnostic.log
    

    For example, soa_server1-diagnostic.log, if server_name is soa_server1.

    This is where the log output is available. By default, only the last 100 MB of the diagnostic logs are retained.

    Server log

    View the following log:

    server_name.log
    

    For example, soa_server1.log, if server_name is soa_server1.

    Server console output

    <stdout> is also helpful, especially for deployment or patching issues.

    Server thread dump

    Enter the following at the operating system command prompt:

    kill -3 managed_server_process_ID
    

    You can also use Oracle WebLogic Server Administration Console.

    1. In the navigation tree of Oracle WebLogic Server Administration Console, select Environment > Servers.

    2. In the table, select the server.

    3. Select the Monitoring tab.

    4. Select the Threads tab.

    5. Click Dump Thread Stacks.

    The output is in the console logs.

    OPatch thread dump

    kill -3 opatch_client_process_ID
    

    OWSM message log

    The following log captures all SOAP messages on the wire.

    $DOMAIN_HOME/servers/server_name/logs/owsm/msglogging/diagnostic.log
    

    This log is not enabled by default. To enable this log:

    1. Go to Fusion Applications Control > Weblogic Domain > Web Services > Policies.

    2. Choose the security level for which to enable logging.

    3. Edit the policy to enable the log assertion.


    For more information about setting logging levels for SOA components, see the "Setting Logging Levels for Troubleshooting" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

  10. Send Oracle Support Services information about the error packaged with Automatic Diagnostic Repository-based incident processing.

    For information about how to investigate, report, and, in some cases, resolve a problem, see the "Investigating, Reporting, and Solving a Problem" section of Oracle Fusion Middleware Administrator's Guide.

9.3 Runtime Diagnostics

This section contains the following topics that describe common problems and solutions for Oracle SOA Suite runtime:

9.3.1 Correlating Application Issues to the SOA Composite IDs

Problem

How does an administrator translate an issue faced in the application to a composite instance in Fusion Applications Control for debugging purposes? For example, a purchase order transaction is stuck in processing for days and the administrator wants to see what the composite instance is doing. How does the administrator map the purchase order ID to the composite instance ID?

Solution

Oracle Fusion Applications modules write context-specific and business user-identifiable keys to the log files. This includes primary keys for any of the logical entities processed by the composite. After the primary key and other context information is included in the log messages, the administrator can search for that context (for example, purchase order ID) in the log file and derive the composite instance ID and ECID. From there, they can diagnose the issue in Fusion Applications Control. Usually, the primary key information is written to the logs in case of any error/incident/message level.

Cross layer, server, and family functionality are all designed to be correlated through the ECID. For example, you can look up the composite instance for a given expense report by correlating all the log entries with the ECID associated with that expense report transaction.

During a specific time interval, you may create several hundred instances., This is a fairly common use case. If there is no error, or if the application has not implemented a BPEL process with sensor logging (should not be common), then instead of searching for the primary key, perform one of the following tasks:

  • Try using the approximate timestamp of the transaction.

  • Inspect the input payload of the audit trail flow (until you find the right one) to identify the right composite.

Any extensions/customizations are also expected to log the same context-specific keys. You can set AF_LOG and AF_LOG_MODULE properly (such as setting log channel levels) to specify finer-grained logging from a particular family/logical business area (LBA)/composite.

9.3.2 Business Event Subscriptions Are Not Firing

Problem

When a business event is published, the business event subscription defined in the composite does not fire and a composite instance is not created.

This can occur, for example, with an ADF application invoking a SOA composite.

Solution

To resolve this problem.

  1. Enable EDN logging in Fusion Applications Control. The following server loggers specific to EDN are available for selection:

    • oracle.integration.platform.blocks.event

    • oracle.integration.platform.blocks.event.saq

    • oracle.integration.platform.blocks.event.jms

    Note that oracle.integration.platform.blocks.event.jms only appears if EDN is running in EDN-JMS mode instead of the default EDN-DB mode.

    You can set the server loggers to one of the following levels:

    • TRACE:1 (FINE)

    • TRACE:16 (FINER)

    • TRACE:32 (FINEST)

    You can alternately use selective tracing to get detailed, on-disk trace selectively (for example, by user name, thereby eliminating trace output for other users). See Section 9.2 for details.

    Detailed logging goes into SOA server's diagnostic.log file configured in Fusion Applications Control. To set the log level for the loggers:

    1. Go to the navigation pane.

    2. Right-click soa-infra.

    3. Choose Logs > Log Configuration.

    4. Expand oracle.integration > oracle.integration.platform > oracle.integration.platform.blocks > oracle.integration.platform.blocks.event.

    5. Set the loggers described in step 1 to an appropriate logging level.

    The following sample shows a portion of a server log file:

    [2011-03-22T11:52:37.038-07:00] [soa_server1] [TRACE] [SOA-31010]
    [oracle.integration.platform.blocks.event.saq] [tid:
    weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
    <anonymous>] [ecid:
    5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
    [SRC_CLASS:
    oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
    [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
    PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B] [SRC_
    METHOD: fineDequeuedEvent] [composite_instance_id: 90087] Dequeued event,
    Subject: null [source type "J"]:[[
        <business-event
    xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
    xmlns="http://oracle.com/fabric/businessEvent">
          <name>ns:MyEvent</name>
          <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
    
    <source>default/MediatorPubSub!2.0*soa_
    58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
    58454bff-3b36-4e29-a883-949085d85719/PublishEvent</source>
        <tracking>
          <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
          <compositeInstanceId>90087</compositeInstanceId>
    <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B</parentCom
    ponentInstanceId>
        </tracking>
          <content>
             <inp1:singleString
    xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
          </content>
    </business-event>
         
       ]]
       [2011-03-22T11:52:37.042-07:00] [soa_server1] [TRACE] [SOA-31011]
    [oracle.integration.platform.blocks.event.saq] [tid:
    weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
    <anonymous>] [ecid:
    5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
    [SRC_CLASS:
    oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
    [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
    PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B] [SRC_
    METHOD: fineFilterResults] [composite_instance_id: 90087] Filter [XPath Filter:
    /be:business-event/be:content/ns1:singleString = 'PO123'] for subscriber
    "default/MediatorPubSub!2.0*soa_
    58454bff-3b36-4e29-a883-949085d85719/SubsEvent" returned true
    . . .
    . . .
    

    For more information about setting logging levels for Oracle SOA Suite, see the "Setting Logging Levels for Troubleshooting" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

  2. Check whether the event publication is being delivered to the SOA cluster.

    One way to do this is to use the log messages in Fusion Applications Control. The log messages include information about the incoming events and the event subject and payload. Before you can view the event information in the log messages, you must first set the oracle.integration.platform.blocks.event logger to TRACE:32 (FINEST), as described in step 1, and then raise the event again. To search for the event-related messages:

    1. Right-click soa-infra, and choose Logs > View Log Messages.

    2. Specify the date range.

    3. In the Message Types section, select the Notification and Trace checkboxes.

    4. To search for all events delivered to the SOA cluster, specify Message contains Dequeued event as the search criteria, and click Search.

      The subject associated with the event is displayed in the Message field and the business event namespace and local name, payload, ECID, and message details are displayed in the Supplemental Detail field.

    5. To search for a particular event by event name, click Add Fields.

    6. Select Supplemental Detail, and click Add.

    7. Specify that Supplemental Detail contains either the event namespace or event local name as the search criteria.

    8. Click Search.

    9. Use additional information such as the time interval or data in the payload to identify the specific event message of interest.

      The log message is generally a good way to identify whether the issue is with the publisher or subscriber. For example, if the subject is missing or the fmw-context is missing, it typically means there is an issue with the publisher. If the information reported in the log message is correct, then it typically means there is a problem with the subscriber.

      If you see the Dequeued event log message for your business event, then go to step 9.

      If you do not see the Dequeued event log message, there may be issues with the following:

      For This Issue... See...

      EDN is in paused mode and temporarily stopped delivering events to its subscribers.

      Step 3

      Events are stuck in the advanced queue.

      Step 4

      More than one SOA cluster is pointing to the same advanced queue.

      Step 5

      There are internal SOA issues in the database layer.

      Step 6

      The data source is configured to point to the wrong SOAINFRA schema.

      Step 7

      There are issues with the Java, SOA, or PL/SQL code raising the event.

      Step 8


  3. When patches are applied to the SOA cluster, EDN is automatically placed in paused mode to prevent delivery of events during patching. After patching is complete, EDN exits paused mode and resumes event processing. When a patch fails, EDN may remain in paused mode to prevent event subscriptions from firing after the failure. Follow these steps to identify if EDN is in paused mode, and restart it:

    1. From the navigation pane, expand SOA and right-click soa-infra.

    2. Choose Administration > System MBean Browser.

    3. Go to Application Defined MBeans > oracle.as.soainfra.config > Server: SOA_cluster_name > EDNConfig > edn.

    4. In the Attributes column, locate the Paused property.

    5. If Paused is set to true, the listener threads are decreased to zero, which temporarily stops event delivery.

    6. To restart event delivery, select false.

    7. Click Apply.

  4. At times, the message may be stuck in the advanced queue in different stages of delivery. Check the count of (potentially stuck) events currently in the following queue tables:

    • EDN_EVENT_QUEUE_TABLE: Every event published is temporarily enqueued into this table (for edn_event_queue).

    • EDN_OAOO_DELIVERY_TABLE: Only events with a one-and-only-one (OAOO) delivery target(s) are temporarily enqueued into this table (for edn_oaoo_queue).

    You can also check the count in the edn-db-log. The total number of messages in the EDN_EVENT_QUEUE_TABLE and EDN_OAOO_DELIVERY_TABLE are displayed at the top of the page.

    1. Navigate to the following URL to see the total number of messages at the top of the page:

      http://host_name:port/soa-infra/events/edn-db-log
      

      You do not need to enable the edn-db-log to view these messages.

  5. There may be more than one SOA cluster pointing to the same SOAINFRA schema. This is highly unlikely in a provisioned environment, but for environments that are created by developers, it may be an issue. In this case, the expected SOA runtime environment may not be receiving the events properly. It is expected that only one SOA runtime environment listens for business events on a SOAINFRA schema. Use the following query to identify the list of SOA runtime environments subscribing to business events from a SOAINFRA schema. Have only the required SOA runtime environment up, which points to the SOAINFRA schema, and shut down the others.

    1. Navigate to the edn-db-log to see log messages that display each SOA cluster that is connecting to the EDN advanced queue:

      http://host_name:port/soa-infra/events/edn-db-log
      

      You do not need to enable the edn-db-log to view these messages. The messages continue to be displayed in the edn-db-log until the next time you click the Clear link. Here is a sample message that you see in the edn-db-log:

      Starting EDN bus. Timestamp=Wed Feb 23 10:11:26 PST 2011.
          Parameters: platform="weblogic", cluster="", server="soa_server1",
          domain="fusion_domain", admin server="AdminServer", host="adc2180440".
      
    2. Alternatively, you can run the following select statement against the SOAINFRA schema. For example, use FIN_FUSION_SOAINFRA for the SOA cluster in the Financials domain.

      select * from V$SESSION where username like 'family_FUSION_SOAINFRA';
      
  6. Verify that the event is properly being enqueued and dequeued by the underlying EDN PL/SQL procedures. You can do this using the edn-db-log that displays all events, including their namespace, names, payload, and subject, which are enqueued and dequeued by the PL/SQL EDN procedures.

    1. Navigate to the following URL:

      http://host_name:port/soa-infra/events/edn-db-log
      
    2. Click the Enable link.

    3. Raise the business event again.

    4. Click the Refresh link.

    Note:

    • You must have the administrator privilege to enable/disable the edn-db-log.

    • You must always disable the edn-db-log after debugging to disable logging. This prevents excessive database growth in the EDN database log table. If the edn-db-log remains enabled, then debugging messages related to events that are published/enqueued into the database and subscribed to/dequeued from the database continue to be persisted into certain EDN database log tables. This causes the table to grow indefinitely.

  7. The EDNSource and EDNLocalTxSource data sources may be pointing to a different database than the EDNDataSource and EDNLocalTxDataSource data source connections of the SOA server. Ensure that these four data source connections on ADF and the SOA server are pointing to the same database schema.

    Check or set the data sources in the Oracle WebLogic Server Administration Console:

    1. In the Domain Structure, select Services > Data Sources.

    2. In the Name column of the table, select data_source_name.

      The connection pool and driver name are only available for generic data sources, and not for multidata sources.

    3. Select the Connection Pool tab.

    4. In the Driver Class Name field, set the data source.

    For more information, see the "Configuring the SOA Infrastructure" chapter in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

  8. Check the log files in Fusion Applications Control for the code that raises the event to see if there were failures in raising the event or failures in the code that prevented the event from being raised.

    For ADF, check the log files. Look for the ECID corresponding to the business key (for example, purchase order).

    To enable event-related log messages in the server running the ADF application that raises the event, set the log level for the oracle.integration.platform.blocks.event logger to TRACE (32) on that server.

  9. By the time you reach this step, you have confirmed that the business event is properly delivered to the SOA cluster. To identify whether any composite instances were created, perform the following steps:

    1. Because the ECID is different for a subscription with a guaranteed consistency level and a subscription with an OAOO level defined for the same event, you must first identify the consistency level for the event subscription. To do this, right click soa-infra, choose Business Events, and click the Subscriptions tab. Find the composite that subscribes to the business event in the Components Subscriptions table and get the value from the Consistency Level column.

    2. If the value is set to Guaranteed, then get the ECID from the Dequeued event log message.

    3. If the value is set to One And Only One, then search for the log message with Message contains Dequeued OAOO event as criteria and use the ECID from that message.

    4. Locate the composite instance by right-clicking soa-infra, selecting the Instances tab, entering the ECID in the search criteria, and clicking Search.

    If you see composite instances, then continue to step 12.

    If you do not see any composite instances, this may be due to the following issues:

    • There are issues with the subscription logic in the composite, possibly a mismatch in the event namespace or event name or error in the filter logic. See step 10.

    • There may be a SOA issue. See step 11.

  10. If a filter is being used, there may be an issue with the filter logic. Use Fusion Applications Control to review the event subscriptions and filters defined for the composite and the payload content.

    1. Log in to Fusion Applications Control in the domain where the composite with the subscription is deployed.

    2. Navigate to domain_name > SOA.

    3. Right-click soa-infra (SOA_cluster_name), and choose Business Events.

    4. Click the Subscriptions tab.

    5. Review the information in the Event Name, Namespace, and XPath Filter columns of the Component Subscriptions table.

    Alternatively, you can also check the SOA server diagnostic.log file for filter-related logging. You must set the log level to at least TRACE:1 (FINE) for the oracle.integration.platform.blocks.event.saq logger. The following is an example log message that has the following statement pattern: Filter [XPath Filter: ….] for subscriber "…" returned true/false.

    [2011-03-22T11:52:36.976-07:00] [soa_server1] [TRACE] [SOA-31011]
    [oracle.integration.platform.blocks.event.saq] [tid:
    [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
    (self-tuning)'] [userId: <anonymous>] [ecid:
    5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1] [SRC_
    CLASS:
    oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
    [WEBSERVICE_PORT.name: execute_pt] [APP: soa-infra] [composite_name:
    MediatorPubSub] [component_name: PublishEvent] [component_instance_id:
    8E0411C054B511E0AF455DABE1395E7B] [J2EE_MODULE.name: fabric] [SRC_METHOD:
    fineFilterResults] [WEBSERVICE.name: PublishEvent_ep] [J2EE_APP.name:
    soa-infra] [composite_instance_id: 90087] Filter [XPath Filter:
    /be:business-event/be:content/ns1:singleString = 'PO123'] for subscriber
    "default/MediatorPubSub!2.0*soa_
    58454bff-3b36-4e29-a883-949085d85719/SubsEvent" returned true
    
  11. Check the SOA diagnostic log for stack traces related to the issue. For example, the SOA instance may have been rolled back due to an error.

  12. By the time you reach this step, you have confirmed that the business event has been delivered to the SOA cluster and a composite instance has been created. However, you may be encountering issues with the composite components:

    • If you see a composite instance, but not a BPEL instance, check if the BPEL process can be executed independently by a client, as opposed to the event subscription.

    • If you see a composite instance, but not a BPEL instance, this symptom may indicate a stuck thread on a service call. In this scenario, the BPEL process service engine is hung on a service call and has not created the audit trail. To verify the behavior, take three thread dumps approximately 30 seconds apart on all servers in the cluster. See Table 9-2 for more information on taking thread dumps. If the same thread shows it is stuck after the three thread dumps (that is, after 1.5 minutes), the thread is stuck. In this case, wait for the call to time out, which then enables you to recover the instance.

    • If the audit trail shows the following message:

      Invoked 1-way operation "initiate" on target service service_name
      

      There may be an issue with an incorrect driver type for the data sources. Note that the driver XA configuration should already be created by the SOA/provisioning template. If the driver type is changed or is not set properly, routing from Oracle Mediator does not occur. Instead, it tries to route and fails several times. This can be confirmed by instances in Fusion Applications Control. As per Oracle Fusion Applications standards, the SOADataSource and EDNDataSource data sources should use the XA driver type; that is:

      • EDNDataSource: oracle.jdbc.xa.client.OracleXADataSource

      • SOADataSource: oracle.jdbc.xa.client.OracleXADataSource

9.3.3 Long Delays After an Event is Published and Before a Subscriber is Triggered

Problem

After a business event is published, there is a long delay (minutes, hours, or even days) before the event subscriber is triggered and its composite instance is created.

This can occur, for example, with an ADF application invoking a SOA composite.

Solution

To resolve this problem, perform the following steps.

  1. Enable EDN logging in Fusion Applications Control. The following server loggers specific to EDN are available for selection:

    • oracle.integration.platform.blocks.event

    • oracle.integration.platform.blocks.event.saq

    • oracle.integration.platform.blocks.event.jms

    Note that oracle.integration.platform.blocks.event.jms only appears if EDN is running in EDN-JMS mode, instead of the default EDN-DB mode.

  2. Set the server loggers to one of the following levels:

    • TRACE:1 (FINE)

    • TRACE:16 (FINER)

    • TRACE:32 (FINEST)

    You can alternately use selective tracing to selectively get a detailed, on-disk trace (for example, by user name, thereby eliminating trace output for other users). See Section 9.2 for details.

    Detailed logging goes into the server's diagnostic.log file configured in Fusion Applications Control.

  3. Set the log level for the loggers.

    1. Go to the navigation pane.

    2. Right-click soa-infra.

    3. Select Logs > Log Configuration.

    4. Expand oracle.integration > oracle.integration.platform > oracle.integration.platform.blocks > oracle.integration.platform.blocks.event.

    5. Set the loggers described in step 1 to an appropriate logging level.

      The following sample shows a portion of a server log file:

      [2011-03-22T11:52:37.038-07:00] [soa_server1] [TRACE] [SOA-31010]
      [oracle.integration.platform.blocks.event.saq] [tid:
      weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
      <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
      [SRC_CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
      PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B]
       [SRC_
      METHOD: fineDequeuedEvent] [composite_instance_id: 90087] Dequeued event,
      Subject: null [source type "J"]:[[
          <business-event
      xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
      xmlns="http://oracle.com/fabric/businessEvent">
            <name>ns:MyEvent</name>
            <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
      <source>default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/PublishEvent</source>
          <tracking>
      <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
            <compositeInstanceId>90087</compositeInstanceId>
      <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B</paren
      tCom
      ponentInstanceId>
          </tracking>
            <content>
               <inp1:singleString
      xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
            </content>
      </business-event>
         ]]
         [2011-03-22T11:52:37.042-07:00] [soa_server1] [TRACE] [SOA-31011]
      [oracle.integration.platform.blocks.event.saq] [tid:
      weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
      <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
      [SRC_CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
      PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B]
       [SRC_
      METHOD: fineFilterResults] [composite_instance_id: 90087] Filter [XPath
       Filter:
      /be:business-event/be:content/ns1:singleString = 'PO123'] for subscriber
      "default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/SubsEvent" returned true
      . . .
      . . .
      

      For more information about setting logging levels for Oracle SOA Suite, see the "Setting Logging Levels for Troubleshooting" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

  4. Find out when the event is published, dequeued, and sent to the subscriber by EDN.

    Use the Java logging in step 1 to identify the timestamps of event publishing, dequeuing, and sending to the subscriber. From these timestamps, you can identify if the delay is on the publishing side or the EDN delivery side.

    To search for the event-related messages in the log messages.

    1. Right-click soa-infra.

    2. Select Logs > View Log Messages.

    3. Specify the date range.

    4. To search for all events delivered to the SOA cluster, specify Message contains Dequeued event as the search criteria, and click Search.

      The subject associated with the event is displayed in the Message field and the business event namespace and local name, payload, ECID, and message details are displayed in the Supplemental Detail field.

    5. To search for a particular event by event name, click Add Fields.

    6. Select Supplemental Detail, and click Add.

    7. Specify that Supplemental Detail contains either the event namespace or the event local name as the search criteria.

    8. Click Search.

    9. Use additional information such as the time interval or data in the payload to identify the specific event message of interest.

      The following sample shows a portion of a server log file containing information about event publishing with a timestamp of 2011-03-22T11:52:36.968-07:00.

      [2011-03-22T11:52:36.968-07:00] [soa_server1] [TRACE] [SOA-31000]
      [oracle.integration.platform.blocks.event.saq] [tid:
      [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
      (self-tuning)'] [userId: <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1] [SRC_
      CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [WEBSERVICE_PORT.name: execute_pt] [APP: soa-infra] [composite_name:
      MediatorPubSub] [component_name: PublishEvent] [component_instance_id:
      8E0411C054B511E0AF455DABE1395E7B] [J2EE_MODULE.name: fabric] [SRC_METHOD:
      fineEventPublished] [WEBSERVICE.name: PublishEvent_ep] [J2EE_APP.name:
      soa-infra] [composite_instance_id: 90087] Received event: Subject: null 
      Sender: default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/PublishEvent
      Event:[[
      <business-event xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
      xmlns="http://oracle.com/fabric/businessEvent">
        <name>ns:MyEvent</name>
        <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
        <tracking>
       <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
           <compositeInstanceId>90087</compositeInstanceId>
      
      <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B</paren
      tComponentInstanceId>
        </tracking>
        <content>
           <inp1:singleString
      xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
        </content>
      </business-event>
      
      ]]
      

      The following sample shows a portion of a server log file containing information about an event dequeued by EDN with a timestamp of 2011-03-22T11:52:37.038-07:00.

      [2011-03-22T11:52:37.038-07:00] [soa_server1] [TRACE] [SOA-31010]
      [oracle.integration.platform.blocks.event.saq] [tid:
      weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
      <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
      [SRC_CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
      PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B]
      [SRC_METHOD: fineDequeuedEvent] [composite_instance_id: 90087] Dequeued
      event, Subject: null [source type "J"]:[[
      <business-event xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
      xmlns="http://oracle.com/fabric/businessEvent">
        <name>ns:MyEvent</name>
        <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
      
      <source>default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/PublishEvent</source>
       <tracking>
        <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
            <compositeInstanceId>90087</compositeInstanceId>
        <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B
        </parentComponentInstanceId>
       </tracking>
         <content>
           <inp1:singleString
      xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
         </content>
      </business-event>
      
      ]]
      

      The following sample shows a portion of a server log file containing information of an event sent to a subscriber by EDN with a timestamp of 2011-03-22T11:52:37.150-07:00.

      [2011-03-22T11:52:37.150-07:00] [soa_server1] [TRACE] [SOA-31033]
      [oracle.integration.platform.blocks.event.saq] [tid:
      weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@b402db] [userId:
      <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000141]
      [SRC_CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [APP: soa-infra] [SRC_METHOD: fineSentOAOOEvent] Sent OAOO event [QName:
      "{http://schemas.oracle.com/events/edl/MyEventDefn}MyEvent" to target:
      "default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/SubsEvent"]:[[
      <business-event xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
      xmlns="http://oracle.com/fabric/businessEvent">
        <name>ns:MyEvent</name>
        <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
      <source>default/MediatorPubSub!2.0*soa_
       58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
       58454bff-3b36-4e29-a883-949085d85719/PublishEvent</source>
      <tracking>
      <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
           <compositeInstanceId>90087</compositeInstanceId>
      
      <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B</paren
      tComponentInstanceId>
      </tracking>
        <content>
           <inp1:singleString
      xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
        </content>
      </business-event>
      ]]
      
  5. If the delay comes from the event publishing side, then check the following:

    1. If you use the remote EDN API to publish events from, for example, an ADF component to the SOA server, and you use the following event connection factory lookup code:

      BusinessEventConnectionFactory BusinessEventConnectionFactorySupport.
      findRelevantBusinessEventConnectionFactory(boolean forceRetry)
      

      You may need to set the forceRetry parameter to true to cache the BusinessEventConnectionFactory object after it is found, and avoid forced retries of JNDI lookups that can cause a significant delay.

    2. If you publish multiple events with a loop statement in a batch fashion, you may need to move the above event connection factory lookup outside the loop to avoid a lookup per event.

  6. If the delay comes from event dequeuing after the event is published, then check the following:

    Ensure That... Description

    The SOA server is up running, and EDN has been configured with the correct amount of dequeuer threads through an EDNConfig MBean.

    To examine the configuration of the EDNConfig MBean in Fusion Applications Control:

    1. Go to the navigation pane.

    2. Right-click soa-infra.

    3. Select SOA Infrastructure > Administration > System MBean Browser.

    4. Expand Application Defined MBeans > oracle.as.soainfra.config > Server: soa_server-x > EDNConfig > edn.

    5. Ensure that the NumberOfThreads attribute (for EDN dequeuer threads) is set to an appropriate number (the default is 3).

    Set the number of EDN dequeuer threads (the default is 3) to a value that matches the number of events published. If there are a large number of events published, then you may need to increase this number to avoid a bottleneck of events dequeuing. Note that the EDN dequeuer threads come from a thread pool that is provided by the SOA server and shared with other SOA processes. Therefore, when you increase the number of EDN dequeuer threads, you may potentially impact server performance for other SOA functionality.

    The backing store of EDN is functioning properly.

    For example, if the backing store of EDN is Oracle AQ, and there are a significant amount of dead events stuck in the AQ that are not cleared, then event dequeuing by EDN may slow down significantly.


  7. If the delay comes from event consumption by its subscriber after the event is dequeued, then check the following:

    1. Ensure that all the event subscribing components such as Oracle Mediator or a BPEL process can consume and process the event within the expected time frame. If not, investigate why there is a delay.

    2. If the event delivery is configured with an OAOO delivery guarantee, then EDN delivers the event to its subscriber in a global transaction context with built-in retry logics, should the transaction fail to commit. Retries may delay overall event processing. If the transaction fails to commit, ensure that you research the root cause of transaction failure.

9.3.4 Events Are Consumed by Multiple Revisions of the Same Composites

Problem

Events are consumed by multiple revisions of the same composites.

Solution

By design, different composites can subscribe to the same event. However, it is an error if multiple revisions of the same composite subscribe to the same event. This occurs when you have multiple active versions of the composite that may be a result of a patch failure.

Perform the following steps to identify whether there are multiple, active revisions of the same composite and to retire the composite revision that should not be active:

  1. Go to Fusion Applications Control.

  2. Click SOA > soa-infra.

    In the Deployed Composites section of the Dashboard tab, you see a list of deployed composite names, revisions, and modes (for example, active).

  3. Identify composites with the same name and with an active mode, but with different revisions.

  4. Click the composite revision that should not be active.

  5. Click the Retire button.

9.3.5 Rolled Back OAOO Event Delivery Messages are Displayed in the Log Files

Problem

If the Java debug level is set to TRACE:16 (FINER) or a lower value, you may see log messages such as the following:

Began XA for OAOO
Rolled back XA for OAOO

Solution

These are normal messages of OAOO event delivery when there are no events waiting to be delivered. These are not error conditions. You can turn off these messages by setting the Java logging level to TRACE:1 (FINE) or a higher value.

9.3.6 Application Transaction Does Not Complete and the Underlying Composite Is Stuck in a Running State

Problem

An application transaction is not completing. For example, a purchase order status may remain processing. Checking the composite instance shows that the composite is stuck in the running state. In this case, the component is probably not running. Instead, it has likely faulted and may need recovery.

Faults may occur for various reasons:

  • A BPEL activity faulted with an error (for example, a business error, security authorization error, or some other error).

  • A BPEL activity invoked an external web service that was unavailable.

  • A BPEL activity has already been terminated by the administrator using Fusion Applications Control.

  • A BPEL activity invoked an asynchronous ADF service and the message is stuck in the AQ/JMS queue.

  • A BPEL activity invoked an asynchronous ADF service, but because SOA was unavailable, the callback message did not arrive.

  • A BPEL activity invoked a synchronous ADF service, which is taking a long time (or is hanging).

  • A network error occurred.

Solution

  1. Log in to Fusion Applications Control.

  2. In the navigator pane, go to domain_name > SOA.

  3. Click soa-infra (SOA_cluster_name).

  4. Click the Instances tab.

  5. Search for the composite instance, and click the instance ID.

    The Flow Trace page appears.

    If the instance is not visible (and the Audit Level is not set to Off in the SOA Infrastructure Common Properties page), this implies that the message is stuck outside of SOA. Check the ADF log (if invoked through an ADF service) to find out if it is stuck in the AQ/JMS queue.

    If the message has reached Oracle Mediator, but not instantiated the BPEL flow, the BPEL instance may have been rolled back from the start due to an error.

    If the BPEL flow exists, the Faults section of the Flow Trace page typically shows the faulted service that can trace the root cause.

  6. In the Trace section, click the BPEL process.

  7. Expand the BPEL audit trail to see the exact point at which the service faulted.

    This information is also available in the Faults tab of the BPEL flow trace. It also indicates whether the fault can be recovered.

  8. Click the View Raw XML link.

    The same information is also available through this link, where you can see the error. For example:

    ...
    <message>Faulted while invoking operation "modifyUserRoles" on provider
     "UserService". </message><details>
    ...
    <tns:message>JBO-27023: Failed to validate all rows in a
     transaction.</tns:message>
    <tns:severity>SEVERITY_ERROR</tns:severity>
    ...
    <message>The transaction was rolled back. The work performed for bpel instance
     "451042" was rolled back to the previous dehydration point, but the audit
     trail has been saved.
    You can recover the instance from the recovery console by resubmitting the
     callback message or activity for execution.</message
    

    Since the instance was rolled back to its previous dehydration point, the status remains as Running.

  9. In the Audit Trail and Faults tabs, make a note of the following:

    • Composite name (for example, HcmUsersSpmlComposite)

    • Component (for example, UpdateGuid BPEL process)

    • BPEL instance ID (for example, bpel:451042)

    This is all used in the recovery of the instance, if it is recoverable. Note that the audit trail window may mark the error as a nonrecoverable business fault, but the recoverability of the message can be found in the Recovery tab of the BPEL process service engine.

  10. To attempt to recover the instance, right-click soa-infra (SOA_cluster_name), and choose Service Engines > BPEL.

  11. Click the Recovery tab.

  12. From the Type list, select Activity.

  13. Specify the composite and component names captured in step 9, and click Search.

  14. Find the specific BPEL instance ID. You can recover faults marked as Recoverable.

  15. Check the other recovery options in the Type list (for example, Invoke and Callback), if they exist.

    Note:

    You can also search for recoverable messages from the Faults and Rejected Messages tab of the SOA Infrastructure by clicking the message and selecting the appropriate action from the Recovery Actions list.

  16. If the instance is not marked as recoverable, then reinvoking the service is not allowed (most probably because it is not idempotent). In some cases, you may need to provide diagnostic information to Oracle Support Services to resolve issues with nonrecoverable, nonidempotent transactions.

  17. If the BPEL activity has invoked an asynchronous ADF service and the message is stuck in the AQ JMS queue, you can view the server.log and server-diagnostic.log files to see the logging of the message metadata logged by the JRF web services infrastructure.

    In addition, the ADF diagnostic logs are also available to debug, if needed.

    1. Use the ECID field to correlate and track ADF service logging corresponding with the SOA composite that invoked it.

    2. When viewing the log in Fusion Applications Control, click the Broaden Target Scope dropdown list and select the farm_name/domain_name (Oracle WebLogic Domain) to view messages across the domain.

    3. In the Log Messages page for the Oracle WebLogic Server domain, in the Selected Targets section, ensure that the search includes the ECID field with the value noted in step a and the Component Name field is set to adf-bc.

    4. Search and view log records for the Oracle Application Development Framework (Oracle ADF) Business Component (BC) and the ECID and note any issues.

      For a specific ECID, you find several root instances (top level clients). You must drill down to the appropriate instance to find a specific fault.

    5. Observe if the Oracle ADF Business Components completed successfully or completed with an error. See the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.

      For more details on diagnosing Oracle ADF-BC asynchronous web service calls, see Section 9.3.7.

See the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite for the various ways to recover transactions and messages:

9.3.7 BPEL Process Received No Response from an ADF Asynchronous Service

Problem

A BPEL activity has invoked an asynchronous ADF service and not received a response. The message may be stuck in the request or response queues.

Solution

  1. Check if the reply address is valid, the server is running, and the name of the server and port match with the server on which the BPEL process is running.

  2. Verify that the policy on the callback receive activity matches with the policy advertised in the WSDL for the response port type/binding.

  3. Look at the server-name_diagnostic.log file and check for all status messages of the request.

    1. Go to Fusion Applications Control.

    2. Right-click soa-infra.

    3. Choose Logs > View Log Messages.

    4. Click Target Log Files.

    5. Select the server diagnostic file to view, and click View Log File.

      For more information about the different types of loggers to set, see the "Setting Logging Levels for Troubleshooting" section of Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

      Note that every log contains the message ID and other details, such as ECID, that can help isolate it from other messages. After you find the message ID (that is, MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33), search for the same ID in the log file. You should see messages such as the following for each successful asynchronous operation.

      If the following log is not available, this means the asynchronous operation was never called on this server:

      [2010-12-17T12:27:13.537-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: <anonymous>]
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0:1] 
      [WEBSERVICE_PORT.name: AsyncEjbPort] [APP: AsyncEjb] [J2EE_MODULE.name:
       AsyncEjb-ejb] [WEBSERVICE.name: AsyncEjbService] [J2EE_APP.name: AsyncEjb]
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33] Sending message
       to JMS queue* 
      "oracle.j2ee.ws.server.async.NonDefaultRequestQueue" for async processing
       of service "1d7551aa-73d9-4624-ad52-c15e35b5b25dRequest"
      

      If the following log is not available, this means the asynchronous operation was never called on this server:

      [2010-12-17T12:27:13.783-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: <anonymous>] 
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0:1] 
      [WEBSERVICE_PORT.name: AsyncEjbPort] [APP: AsyncEjb] [J2EE_MODULE.name:
       AsyncEjb-ejb] [WEBSERVICE.name: AsyncEjbService] [J2EE_APP.name: AsyncEjb]
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33] An asynchronous
       request message is received*
      and successfully recorded for service "AsyncEjbService" with a replyTo
      address as
      "http://adc2180314:7001/AsyncEjbCallback/AsyncEjbResponseImplService"
       
      [2010-12-17T12:27:13.986-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: weblogic] 
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0] [APP: AsyncEjb]  
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33]
       Startedasynchronous request processing for the service* 
      "AsyncEjbService" with the message selector
       "1d7551aa-73d9-4624-ad52-c15e35b5b25dRequest". Transaction enabled: "true"
       
      [2010-12-17T12:27:14.296-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: weblogic] 
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0] [APP: AsyncEjb]  
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33] Completed
       asynchronous request processing. A response will be sent to the client.*
       
      [2010-12-17T12:27:14.312-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: weblogic] 
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0] [APP: AsyncEjb]  
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33]
       Startedasynchronous response processing for the service* 
      "AsyncEjbService" with the message selector
      "1d7551aa-73d9-4624-ad52-c15e35b5b25dResponse".
       
      [2010-12-17T12:27:15.018-08:00] [AdminServer] [NOTIFICATION] []
      [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
      (self-tuning)'] [userId: weblogic] 
      [ecid: 77e271e3-afd5-48e8-a83d-9b326860f3f2-0000000000000018,0] [APP:
      AsyncEjb] *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33]
      Completed asynchronous response processing successfully. Client must have
      received the response by now.*
      

    To monitor the traffic to the asynchronous ADF web services, monitor the Oracle AQ queues that are used for each family. The queue names for each family are shown in the following table:

    Family AQ Name

    Oracle Fusion Customer Relationship Management Product

    CRM_AsyncWS_Request and CRM_AsyncWS_Response

    Oracle Fusion Human Capital Management Product

    HCM_AsyncWS_Request and HCM_AsyncWS_Response

    Oracle Fusion Financials Product

    FIN_AsyncWS_Request and Fin_AsyncWS_Response

    Oracle Fusion Procurement Product

    PRC_AsyncWS_Request and PRC_AsyncWS_Response

    Oracle Fusion Project Product

    PRJ_AsyncWS_Request and PRJ_AsyncWS_Response

    Oracle Fusion Supply Chain Management Product

    SCM_AsyncWS_Request and SCM_AsyncWS_Response

    Oracle Fusion Setup Product

    COMMON_AsyncWS_Request and COMMON_AsyncWS_Response

    IC

    IC_AsyncWS_Request and IC_AsyncWS_Response


9.3.8 Business Event Is Picked Up Twice (Or More) By SOA Server

Problem

Business events are raised from Fusion J2EE applications and are picked up more than once. Expected behavior is that they should be picked up only once.

Solution

If a patch fails, it may be possible for two versions of a given composite to be active (that is, the older version has not been retired). When multiple versions of a composite are active, they all become subscribers and the event is picked up more than once. This should not happen under normal scenarios. Reapply the patch and ensure that the deployed composite only has one active version.

9.3.9 Long Running, Synchronous Calls To Remote Web Services Error Out or Asynchronous Transactions Return with an Error after a Long Time

Problem

Long running synchronous calls to remote web services end with JTA transaction rolled-back errors. When executing a transaction making an asynchronous call (for example, to the SOA server), the application returns with an error. Server logs show JTA transaction timeouts, which can cause this behavior.

Solution

Check the JTA transaction timeout in Oracle WebLogic Administration Console.

  1. Log in to Oracle WebLogic Server Administration Console.

  2. In the Domain Structure, select Services > JTA to check the timeout value.

    If the transaction is always timing out beyond 30 seconds and is a custom composite synchronous client invocation, then you may need to revisit the design approach. It may be best for the external web service to be invoked as an asynchronous transaction.

    Increasing the JTA for supporting long running synchronous transactions is simply an interim mechanism.

    For information on changing the JTA transaction timeout setting, see the "Resolving Connection Timeouts" section of Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

  3. If synchronous client invocations to Oracle Fusion Applications code take a long time, check for any performance issues with the system and try to resolve them.

  4. Check for appropriate values for the syncMaxWaitTime property and BPEL's EJB transaction timeout settings in relation to the JTA timeout settings and only then increase the value of the JTA timeout, if needed.

    For information on viewing and changing the syncMaxWaitTime property, see the "Configuring BPEL Process Service Engine Properties" section of Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

  5. To view and change the BPEL EJB transaction timeout settings, perform the following steps:

    1. Log in to Oracle WebLogic Server Administration Console.

    2. In the Domain Structure, click Deployments.

    3. Expand soa-infra > EJBs.

    4. Update the following EJBs:

      - BPELActivityManagerBean

      - BPELDeliveryBean

      - BPELDispatcherBean

      - BPELEngineBean

      - BPELFinderBean

      - BPELInstanceManagerBean

      - BPELProcessManagerBean

      - BPELSensorValuesBean

      - BPELServerManagerBean

    5. Click Save.

    6. Restart Oracle WebLogic Server.

  6. For asynchronous transactions, check the values for both the BPEL EJB transaction timeout and the JTA transaction timeout and adjust as needed.

9.3.10 Some Messages Are Lost Between EDN and Composites or Composites Across Clusters

Problem

Messages primarily get lost for the following reasons:

  1. The EDN message was not delivered.

  2. The target asynchronous service did not respond.

  3. The message was roll backed (though it was not lost; you still see it in Fusion Applications Control).

Solution

This issue typically requires you to identify where the messages are supposed to be and to diagnose the path.

  • If the calling component is an ADF web service invoking a SOA composite, then check for any client errors in the ADF logs.

  • Because ADF invokes SOA composites using EDN, check the JMS/AQ and XA data source configurations to ensure that events are being fired. For more information about EDN issues, see Section 9.3.2.

  • Check if the Oracle WSM Policy Manager security configurations allow the client to invoke the SOA composite.

  • Verify that the transaction is not transient (nonpersistent).

    Oracle BPEL Process Manager uses the dehydration store database to maintain long-running, asynchronous processes and their current state information in a database while they wait for asynchronous callbacks. Storing the process in a database preserves the process and prevents any loss of state or reliability if a system shuts down or a network problem occurs. There are two types of processes in Oracle BPEL Process Manager. These processes impact the dehydration store database in different ways.

    • Transient processes: This process type does not incur any intermediate dehydration points during process execution. If there are unhandled faults or there is system downtime during process execution, the instances of a transient process do not leave a trace in the system. Instances of transient processes cannot be saved in-flight (whether they complete normally or abnormally). Transient processes are typically short-lived, request-response style processes. The synchronous process you design in JDeveloper is an example of a transient process.

    • Durable processes: This process type incurs one or more dehydration points in the database during execution because of the following activities:

      • Receive activity

      • OnMessage branch of a pick activity

      • OnAlarm branch of a pick activity

      • Wait activity

      Instances of durable processes can be saved in-flight (whether they complete normally or abnormally). These processes are typically long-living and initiated through a one-way invocation. Because of out-of-memory and system downtime issues, durable processes cannot be memory-optimized.

  • If a composite instance is not visible in Fusion Applications Control and the SOA Infrastructure is running, check that the Audit Level is not set to Off in the SOA Infrastructure Common Properties page. This can be checked in Fusion Applications Control:

    1. Right-click soa-infra (SOA_cluster_name).

    2. Choose SOA Administration > Common Properties.

  • Check the ADF logs to see if there was a bad/malformed message or incorrect SOAP headers (if invoking from an external web service). The transaction may have rolled back before being dehydrated due to an internal error. Again, the SOA logs can identify the cause of this issue.

  • If the composite instance is available, check the bpel.config.oneWayDeliveryPolicy BPEL property value. You can check the value in Fusion Applications Control:

    1. In the navigation pane, expand soa-infra (SOA_cluster_name).

    2. Expand the partition, and select the composite.

      The Dashboard page for the composite is displayed.

    3. In the upper right corner, click the Show XML Definition icon.

      The contents of composite.xml for that composite are displayed.

    If this is set to async.cache, you may lose messages. Set it to async.persist for reliable messages. This is typically specified in the BPEL process service component section of the composite.xml file, so this can be set for custom composites. If the value is not set in composite.xml, the value for oneWayDeliveryPolicy in the System MBean Browser in Fusion Applications Control is used. The following values are possible:

    • async.persist: Messages are persisted in the database hash map.

    • async.cache: Messages are stored in memory.

    • sync: Direct invocation occurs on the same thread.

    For more information about these settings, see the "Deployment Descriptor Properties" appendix and the "Transaction and Fault Propagation Semantics in BPEL Processes" chapter in the Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

    For information about setting the oneWayDeliveryPolicy property in the System MBean Browser, see the "Configuring BPEL Process Service Engine Properties" section of Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

    It is also possible that the transaction has invoked a target asynchronous service that has not responded back. In this case, the composite instance flow shows the call to the target asynchronous service.

  • Check if the instance has rolled back and the message is in recovery.

    1. Log in to Fusion Applications Control.

    2. Right-click soa-infra (SOA_cluster_name), and choose Service Engines > BPEL.

    3. Click the Recovery tab.

    This may occur if any external references receiving the message are not reachable (for example, an external web service, enterprise applications such as Siebel, and so on).

9.3.11 Some Composites Are Being Retried Multiple Times on Failure

Problem

When a BPEL process flow errors out, it is retried with all its invocations. This is undesirable in some cases.

Solution

The property GlobalTxMaxRetry (default value is 3) specifies how many retries are performed if an error is identified as a retriable one. For example, after several web service invocations, if dehydration fails due to a data source error, then this is identified as a retriable error and all activities from the prior dehydration state are retried. If the activities being retried are not idempotent (that is, their state can change with each retry and is not guaranteed to give the same behavior), then multiple retries can be problematic.

To rectify this, customize the composite by specifically marking the nonidempotent activities with idempotentset set to false in the partner link settings section of the composite.xml file to prevent retries.

<property name="bpel.partnerLink.partner_link_name.idempotent">false</property>

You cannot set the idempotentset property in Fusion Applications Control.

For more information about Oracle SOA Suite customizations in JDeveloper, see the "Customizing and Extending SOA Components" chapter in the Oracle Fusion Applications Extensibility Guide.

You can also set GlobalTxMaxRetry to 0 in the Systems MBean Browser.

  1. Right-click soa-infra (SOA_cluster_name).

  2. Choose SOA Administration > Common Properties.

  3. Click More SOA Infra Advanced Configuration Properties.

  4. Click GlobalTxMaxRetry.

  5. In the Value field, enter an appropriate value.

  6. Click Apply.

For information about the idempotent property, see the "BPEL Properties Set Inside a Composite" section of Oracle Fusion Middleware Performance and Tuning Guide.

9.3.12 Some Fusion Applications Control Features are Missing the No Recover Button or Export Composite Capability

Problem

Fusion Applications Control is missing features needed for diagnostics and recovery. For example, the Recovery tab for the BPEL process service engine shows a BPEL instance as recoverable along with instructions to click the Recover button. However, there is no Recover button available. In addition, the ability to export a composite is not available.

Solution

Check that you are logged in as a user with administrator privileges and are not logged in as a read-only user. Read-only users do not see administrative features such as the Recover button or the ability to export composites.

See the "Oracle Enterprise Manager Roles" appendix of Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.3.13 Automatic Recovery of BPEL Instances is Not Recovering A Specific Instance

Problem

BPEL processes have an automatic recovery feature that attempts to automatically recover activities that are recoverable such as unresolved invoke/callback messages, activities not completed over a provided threshold time, and so on. However, the automatic recovery feature only tries to recover a few instances and only retries a fixed number of times. If some instances are not being automatically recovered, they are likely not being picked up because of the configuration of the automatic recovery parameters.

Solution

  • Set the maximum number of messages to automatically recover.

    By default, the automatic recovery feature of Oracle BPEL Process Manager processes fifty messages to submit for each recovery attempt. This is controlled by the maxMessageRaiseSize property.

    1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

    2. Choose SOA Administration > BPEL Properties > More BPEL Configuration Properties > RecoveryConfig.

    3. Expand both RecurringScheduleConfig > maxMessageRaiseSize and StartupScheduleConfig > maxMessageRaiseSize.

      The default value is 50 for each. A negative value causes all messages selected from the database to be submitted for recovery. A value of 0 causes no messages to be selected from the database (effectively disabling recovery). To recover more than fifty messages, set the property value to that value. Use this property to limit the impact of recovery on the server.

  • Set the maximum number of automatic recovery attempts on a given message.

    You can also configure the number of automatic recovery attempts to submit in the same recoverable instance. The value you provide specifies the maximum number of times that invoke and callback messages are recovered. If the value is 0 (the default value), it recovers all messages. After the number of recovery attempts on a message exceeds the specified value, a message is marked as nonrecoverable.

    Follow these steps to configure automatic recovery attempts for invoke and callback messages in Fusion Applications Control.

    1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

    2. Choose SOA Administration > BPEL Properties > More BPEL Configuration Properties.

    3. Select MaxRecoverAttempt, and enter a value in the Value field.

    4. Click Apply.

      It may not be desirable in all cases to use automatic recovery. If services are not idempotent, then corruption can occur. Moreover, the automatic recovery restores the composite to the last save point that can be immediately after an asynchronous invoke, wait, and so on. Therefore, it is important to understand the process behavior and what it does next before performing mass recoveries. Attempt mass automatic recovery only after the root cause of the composite failures has been fixed (for example, a service that was unavailable is now available, a database running out of space was fixed, and so on). Automatic recovery can also trigger an unexpected load during failure scenarios. This causes more threads to block on a remote server that can induce hangs in the SOA server in a cascading fashion.

      For more information, see Section 9.3.11 and the "Configuring Automatic Recovery for Oracle BPEL Process Manager" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.3.14 SOA Runtime Fails with a "Cannot read WSDL" Error

Problem

The following error is displayed if either the endpoint is not available for a reference or a composite is deployed with an incorrect deployment plan file.

oracle.fabric.common.FabricException: Cannot read WSDL

Solution

  1. Ensure that the endpoint for the reference is up and running.

    The WSDL or endpoint is stated in the error. The service can then be looked up from Fusion Applications Control to check if it is active. After the service is active and the endpoint is reachable, search for the SOA instance and retry through the Recovery tab for the BPEL process service engine in Fusion Applications Control.

  2. For extensions/customizations, ensure that the correct URL is updated in the deployment configuration plan.

    For information about deployment plans, including examples of using sca_extractPlan to extract plans, see the "Customizing Your Application for the Target Environment Prior to Deployment" section of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite and the "Customizing SOA Composites with JDeveloper" section of Oracle Fusion Applications Extensibility Guide.

9.3.15 Uploading a Composite for Oracle Support Services

Problem

Oracle Support Services has requested you to upload your composite to review the flow and associated configuration files.

Solution

  1. Log in to Fusion Applications Control as a user with administrator privileges, and not as a read-only user.

  2. In the navigation pane, expand soa-infra (SOA_cluster_name).

  3. Expand the partition in which the composite is located (for example, default).

  4. Right-click the composite and choose Export.

  5. Accept the default selections, and save the resulting JAR file.

  6. Provide the JAR file to Oracle Support Services so that it can be imported for review in JDeveloper by selecting File > Import.

9.3.16 Confirming SOA Component Configuration Properties for Oracle Support Services

Problem

Oracle Support Services has requested you to confirm configuration properties of various SOA components.

Solution

  1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

  2. Choose Administration > MDS Configuration > Export.

  3. Upload the resulting file to Oracle Support Services for review.

9.4 Security and Oracle WSM Policy Manager Configuration

For information about troubleshooting Oracle WSM Policy Manager, see the "Diagnosing Problems" chapter in the Oracle Fusion Middleware Security and Administrator's Guide for Web Services.

9.5 Human Workflow

This section contains the following topics that describe common problems and solutions for human workflow:

9.5.1 Recovering Human Task Instances Stuck in the Alerted State

Problem

If a human task instance fails due to a business error, the task status stays in the alerted state.

Solution

This can happen if, for example, the following occurs:

  • The assigned user does not exist.

  • The manager is not defined.

  • A rule does not return assignees.

In these cases, the task is set to alerted and is assigned to the process owner or the error assignee (if defined in the task metadata).

The administrator or process owner must go to Oracle BPM Worklist and reassign these tasks to the correct users for the workflow to continue forward.

9.5.2 Task Detail Page is Not Available

Problem

When you click the human task in Oracle BPM Worklist, the following message is displayed:

Task detail is not available

Solution

This problem typically occurs during the deployment of the ADF J2EE application containing the task details page definition. As a result, the URI for the details page is not registered with the SOA cluster.

To confirm whether the task details page was registered in Fusion Applications Control:

  1. Log in to the domain where the composite with the subscription is deployed.

  2. In the navigation pane, right-click SOA > soa-infra (SOA_cluster_name).

  3. Choose Service Engines > Human Workflow.

  4. In the Components section, click the appropriate task.

  5. Click the Administration tab to see a URI for the task details page.

    If there is no URI displayed, then check the log files for any errors during deployment.

    A possible problem may be incorrect entries in the wf_client_config.xml file. The wf_client_config.xml file resides in the exploded EAR file's APP-INF/classes directory. For example:

    APPLICATIONS_BASE/fusionapps/applications/fin/deploy/EarFinPayables.ear/
    APP-INF/classes/wf_client_config.xml
    

    The file contains the cluster URL information for the local SOA runtime of that particular Oracle WebLogic Server domain. If the cluster name is wrong or that cluster does not exist in the domain, then a problem exists with the deployment and configuration of the environment.

    As a workaround, you can directly enter the task details page URI in Fusion Applications Control. See the "Managing the URI of the Human Task Service Component Task Details Application" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite. For the values, enter the following:

    Field Value

    Application Name

    worklist

    Host Name

    Your server host name for SOA

    HTTP Port

    Your HTTP server port used for SOA if SSL is disabled.

    HTTPS Port

    Your HTTPS server port used for SOA if SSL is enabled.

    URI

    For example, for financials:

    /payables/faces/adf.task-flow?_id=FinApInvoiceApprovalTaskFlow&_
    document=WEB-INF/oracle/apps/financials/payables/invoices/transact
    ions/ui/invoiceApprovalTask/flow/FinApInvoiceApprovalTaskFlow.xml
    

9.5.3 Task Chooses an Incorrect User If a Number of Rules Are Defined or Errors with "Ruleset returned lists with different list builder"

Problem

At runtime, when a human workflow task tries to fetch the list of users, it errors with the following message:

Ruleset returned lists with different list builder

This can be seen in the Task Detail Comments field. Alternately, it may select a user or approver, which may not seem to be the correct or expected one. This is primarily caused by having overlapping rules with different list builders. When participants of a task are specified using business rules, it is expected that business rules return list builders of the same type.

Moreover, only one rule from a ruleset should be applicable for a transaction. In case a number of rules are true, the actions associated with the applicable rule with the highest priority get executed. If multiple applicable rules have the same priority, then the first rule in the list is picked and its actions are executed.

Solution

To resolve this issue, avoid writing overlapping rules. Constraints from different list builders are different and cannot be mixed. If more than one rule gets triggered with a different list builder, this error occurs. Moreover, only one set of constraints is honored.

Check that all rules in the ruleset have priorities defined so that multiple rules with the same priority are not applicable for the same transaction. For more details, see the "Using Approval Management" chapter in the Oracle Fusion Middleware Modeling and Implementation Guide for Oracle Business Process Management.

9.5.4 Document Is Stuck in Pending Approval and Logs Show Error "Unable to walk hierarchy for user user_name"

Problem

The document is stuck pending approval and the logs show the following error:

"Unable to walk hierarchy for user username'

This occurs when the hierarchy for the given user has not been set up correctly. For example, assume you have approval rules set up so that the rule getting fired uses the supervisory hierarchy and you expect it to route to USER01 > USER02 > USER03 > . . . > USER10 (ten level hierarchy). If the supervisory/management hierarchy of USER01 has not been set up, then the approval task throws an error stating the inability to go through the hierarchy for USER01.

Solution

Since the user level hierarchy does not exist for the user_name, verify the Human Capital Management (HCM) setup by manually running the HCM service to which workflow-identity-config.xml is pointing. If it does not return any result, then configure the correct hierarchy for the user in HCM.

Run the HCM service, select the fetchManager(s) API (for supervisory), and provide the following payload, where <ns2:Id>DD7A1614BBFAA0F0A4511ACD96D2C88D</ns2:Id> is the GUID of the user whose manager you are trying to find. The supervisory hierarchy and position hierarchy services can be found under HcmCore. The name is HierarchyProviderService. The corresponding WSDLs are as follows:

  • Supervisory hierarchy:

    http://host:port/hcmEmploymentCoreApprovalHierarchy/HierarchyProviderService?wsdl
    
  • Position hierarchy:

    http://host:port/hcmTreesModel/HierarchyProviderService?wsdl
    
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>
<wsse:Security soap:mustUnderstand="1"
xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-sec
ext-1.0.xsd">
<wsse:UsernameToken>
<wsse:Username>FUSION_APPS_AMX_APPID</wsse:Username>
<wsse:Password
Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profi
le-1.0#PasswordText">Welcome1</wsse:Password>
</wsse:UsernameToken>
</wsse:Security>
</soap:Header>
<soap:Body
xmlns:ns1="http://xmlns.oracle.com/apps/hcm/employment/core/approvalHierarchyServi
ce/types/">
<ns1:fetchManagers>
<ns1:principal
xmlns:ns2="http://xmlns.oracle.com/apps/hcm/employment/core/approvalHierarchyServi
ce/">
<ns2:Id>DD7A1614BBFAA0F0A4511ACD96D2C88D</ns2:Id>
</ns1:principal>
<ns1:level>2</ns1:level>
</ns1:fetchManagers>
</soap:Body>
</soap:Envelope> 

The following example shows the workflow-identity-config.xml file used in a development environment:

<?xml version='1.0' encoding='UTF-8'?>
<ISConfiguration xmlns="http://www.oracle.com/pcbpel/identityservice/isconfig">
<configurations>
<configuration realmName="myrealm">
<provider name="JpsProvider" providerType="JPS" service="Identity">
<property name="jpsContextName" value="default"/>
<property name="IdentityServiceExtension" value="HCMIdentityServiceExtension"/>
</provider>
</configuration>
</configurations>
<property name="caseSensitive" value="false"/>
<property name="caseSensitiveGroups" value="true"/>
<serviceExtensions>
<serviceExtension name="HCMIdentityServiceExtension">
<serviceProvider
classname="oracle.bpel.services.identity.hierarchy.providers.hcm.HCMHierarchyProvi
der" type="supervisoryHierarchyProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmEmploymentCoreApprovalHierarchy/Hie
rarchyProviderService?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
<serviceProvider
classname="oracle.bpel.services.identity.hierarchy.providers.hcm.HCMHierarchyProvi
der" type="jobLevelHierarchyProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmEmploymentCoreApprovalHierarchy/Hie
rarchyProviderService?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
<serviceProvider
classname="oracle.bpel.services.identity.hierarchy.providers.hcm.HCMPositionHierar
chyProvider" type="positionHierarchyProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmTreesModel/HierarchyProviderService
?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
<serviceProvider
classname="oracle.bpel.services.identity.position.provider.hcm.PositionLookupServi
ceProvider" type="positionLookupProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmEmploymentCore/positionLookupServic
e?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
<serviceProvider
classname="oracle.bpel.services.identity.position.provider.hcm.PositionDisplayName
Provider" type="positionDisplayNameProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmTreesModel/HierarchyProviderService
?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
</serviceExtension>
</serviceExtensions>
</ISConfiguration>

The workflow-identity-config.xml file is in the MDS repository. For information about how to export the workflow-identity-config.xml file from MDS, see Section 9.5.6.

9.5.5 Task History: Cannot Remove or Move a Participant After Yourself

Problem

You cannot remove or move a participant that was added through future participant editing. This is because the participant was anchored to you and you are the current assignee.

Solution

This feature is not supported.

9.5.6 Hierarchy List Builders Display an Error Due to Security Configuration Issues and Log Report Errors in Authentication or Authorization with HCM Services

Problem

Hierarchy list builders (for example, supervisory, job-level, and position) used in rules are not working as expected and are throwing security configuration-related errors. For example:

SOAPFaultException: FailedAuthorization

Hierarchy services are protected services using Oracle Web Services Manager policies (for example, supervisory, job-level, and position services). Elevated privileges are used for authentication. These privileges must be configured in Fusion Applications Control and in configuration files.

Solution

To resolve this problem:

  1. Check the MDS store to see if workflow-identity-config.xml and workflow-config.xml are set up with the correct information (pointing to HCM services and the JAZN name).

    1. Run the following script to connect to the WLST shell on Linux operating systems. There is no equivalent script on Windows operating systems.

      $MW_HOME/oracle_common/common/bin/wlst.sh
      
    2. Run connect(). This takes you to the prompt as shown in the following example:

      wls:/offline> connect()
      Please enter your username [weblogic] :weblogic
      Please enter your password [weblogic] :
      Please enter your server URL [t3://localhost:7001]
      :t3://example.com:9401
      
    3. Run the following command after connecting:

      exportMetadata(application='soa-infra',server='soa_server1',toLocation='any
      Location on
      server',docs='/soa/configuration/default/workflow-identity-config.xml')
      

      The file is stored in the soa/configuration/default directory in the location given in the toLocation attribute. After verifying, you can upload the file with the following command.

      importMetadata(application='soa-infra',server='soa_
      server1',fromLocation='any Location on
      server',docs='/soa/configuration/default/workflow-identity-config.xml')
      
  2. Verify that the csf-keys are specified as initialization parameters.

    1. Ensure that the csf-key-name value is FUSION_APPS_AMX_APPID-KEY in workflow-identity-config.xml, as described in Section 9.5.4.

  3. Check in Fusion Applications Control that the keys defined for the services have the correct values for authentication.

    1. Run the following script to connect to the WLST shell:

      $MW_HOME/oracle_common/common/bin/wlst.sh
      
      wls:/offline> connect()
      Please enter your username [weblogic] :weblogic
      Please enter your password [weblogic] :
      Please enter your server URL [t3://localhost:7001]
      :t3://example.com:9401
      
    2. Run the following command after connecting:

      listCred(map="oracle.wsm.security", key="FUSION_APPS_AMX_APPID-KEY")
      

      Output similar to the following is displayed:

      [Name : FUSION_APPS_AMX_APPID, 
      Description : HCM Hierarchy Service, expiry 
      Date : null]PASSWORD:password
      

9.5.7 HCM Services are Timing Out or Getting Stuck

Problem

Hierarchy services sometimes get overloaded and do not respond (for example, supervisory, job-level, and position services).

Solution

Check that the workflow-identity-config.xml file for the problematic service has correct timeout settings defined as initialization parameters. The timeout parameter is http-read-timeout. For an example of the configuration file, see Section 9.5.4. Note that increasing the timeout may cause errors, such as other timeouts and only puts more load on an already overloaded server.

9.5.8 Error/Warning Reported in Retrieving the Position for a User

Problem

Not all LDAP users are defined as people within Oracle Fusion HCM. For example, users may be defined in LDAP for Oracle Fusion Supply Chain Management that are not defined as people with positions in Oracle Fusion HCM. Therefore, the position lookup can generate a message that the user does not have positions.

Solution

This is a tolerated condition and the warning can be ignored if the user name specified in the warning is not an HCM user. If this is an HCM user, the problem is on the HCM side. Add a position to these users in the HCM application.

9.5.9 Users See No Approvals in the Inbox when Using Position Hierarchy

Problem

This issue may be noticed by organizations using position hierarchy such as governments and universities. After users log in, they do not see any approvals in their inbox. This is caused by a failure to look up the position. This can be caused by network issues when accessing the HCM services. Sometimes due to load balancing or network setup, HCM services may not be accessible and Oracle BPM Worklist has errors such as the following:

WSDL port is not accessible

or

Could not find the host

Solution

  1. Check the network setup and ensure that the HCM services are accessible and WSDL ports can be accessed in the browser. Tools like ping and traceroute can also diagnose the problem.

  2. If the services are not available, bring up the services and retry.

  3. Restart the HCM server and SOA server. For information, see the following documentation:

9.5.10 Task Escalation Does Not Follow Job Level, Supervisory, or Position Hierarchies

Problem

This feature is not supported. Escalation does not follow the job level, supervisory, or position hierarchies.

Solution

Build your own custom plug-in. For more details, see Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

9.5.11 User Is Not Allowed to Perform an Action on a Human Workflow Task

For troubleshooting information about this issue, see the "Task Action Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.12 Email Notification Is Not Being Sent Out

For troubleshooting information about this issue, see the "Notification Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

For additional details about configuring email driver properties, see the "Configuring Email Notifications for Oracle Fusion Applications" section in the Oracle Fusion Applications Post-Installation Guide.

9.5.13 Notifications Sent Are Not Actionable

For troubleshooting information about this issue, see the "Notification Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite. This section describes how to make the task actionable in the Human Task Editor during design time.

In addition, if notifications are sent, but are not actionable, the administrator may have configured the notification to be nonactionable through Oracle BPM Worklist. To make notifications actionable, select the Make notification actionable checkbox under the Event Driven subtab of the Task Configuration tab.

For additional details, see the "Configuring Email Notifications for Oracle Fusion Applications" section in the Oracle Fusion Applications Post-Installation Guide.

9.5.14 Actionable Notifications Are Sent But No Action is Taken After Responding

For troubleshooting information about this issue, see the "Notification Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.15 After LDAP Configuration with Oracle WebLogic Server, Users Appear in the Console, But a "No Role found matching the criteria" Error Appears

For troubleshooting information about this issue, see the "Identity Service Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.16 Only a Subset of Users in LDAP Can Log In to Oracle BPM Worklist

For troubleshooting information about this issue, see the "Identity Service Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.17 "Unknown Macro" Exception Appears when Myrealm Is Passed as a Parameter to the Identity Context To Identity Service APIs

For troubleshooting information about this issue, see the "Identity Service Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.18 Add Participant Button in Oracle BPM Worklist Is Disabled

For troubleshooting information about this issue, see the "Task History Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.19 Task Modifications Made in a Runtime Tool Are Not Appearing for the Task

For troubleshooting information about this issue, see the "Design Time at Runtime Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.20 How Can I Set Commonly Used Human Workflow Configuration Parameters?

For troubleshooting information about this issue, see the "Human Workflow Service/System MBean Browser Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.21 How Do I Set Human Workflow Configuration Parameters Not Available in the Fusion Applications Control Properties Pages?

For troubleshooting information about this issue, see the "Human Workflow Service/System MBean Browser Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.22 System MBean Browser Does Not Reflect My Changes After Editing the Human Workflow Config MBeans

For troubleshooting information about this issue, see the "Human Workflow Service/System MBean Browser Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.23 Human Workflow Services Are Not Locating Resource Bundles or Classes Located at the Workflow Customizations Class Path URL

For troubleshooting information about this issue, see the "Human Workflow Service/System MBean Browser Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.24 How Do I Manually Set the URL for Displaying Task Details for a Particular Human Workflow Task Component?

For troubleshooting information about this issue, see the "Human Workflow Service/System MBean Browser Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.25 How to Test the Health of the Installed Server

For troubleshooting information about this issue, see the "Test-to-Production Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.26 User Authentication Error (in this example, the user is FMW_USERID and identityContext is jazn.com)

For troubleshooting information about this issue, see the "Test-to-Production Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.27 Error During Import of Task Payload Mapped Attribute Mappings

For troubleshooting information about this issue, see the "Test-to-Production Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.28 Error During Rule Migration

For troubleshooting information about this issue, see the "Test-to-Production Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.5.29 When Defining a Rule Based on the Approval Group List Builder During Runtime, a Message Says the Group Does Not Exist

For troubleshooting information about this issue, see the "AMX Extension Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.6 Patching and Deployment

For troubleshooting information about patching and deployment issues, see the "Troubleshooting Patching Sessions for SOA Composites" section in the Oracle Fusion Applications Patching Guide.

9.7 Performance

This section contains the following topics that describe common problems and solutions for Oracle SOA Suite performance:

9.7.1 SOA Transactions are Failing and Logs Indicate the SOA Database is Running Out of Space

Problem

SOA transactions are failing and logs indicate that the database is running out of space. The problem may occur with the SOA dehydration store or MDS store running out of space due to a high volume of transactions. In the latter example, you see errors such as the following:

java.sql.SQLException: ORA-01653: unable to extend table SH_MDS.CUBE_INSTANCE by
16 in tablespace FUSION_TS_TOOLS 

This indicates that the tablespace is full and the database cannot extend it.

Solution

  1. Purge the SOA dehydration store tables periodically, taking into account the appropriate record retention policies and ensuring that the applications have no dependencies on runtime data.

    The purge should be followed by commands to coalesce the space. For the purging strategy to work, it is important to understand how long to retain the data in the database. Factors that drive the retention policy include the following:

    • Legal requirements

    • Line of business requirements

    • Overall company policy on retention of data

    The longer the retention policy, the greater the volume of data that must be stored and, correspondingly, the higher the disk capacity requirements.

    For details on creating a purging strategy, see the "Managing Database Growth" chapter in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

  2. Ensure that the database hardware has sufficient resources to handle the demands of Oracle Database partitioning prior to configuring your tables for partitioning.

    For the SOA dehydration store, database partitioning using range partitioning and hash partitioning is an optimal solution. Partitioning by definition means storing data in multiple tables to reduce bigger data sets into smaller, more manageable data sets. Partitioning strategies play a large role in easing maintenance overheads (dropping and pruning the partition) and improving performance. Partitioning should at least be done for tables having high activity. This plays a large role in balancing disk I/O and preventing hot disks. One important requirement that you must meet prior to configuring your tables for partitioning is to ensure that the database hardware has sufficient resources to handle the demands of Oracle Database partitioning. If preproduction testing has indicated that the installation is large, Oracle expects that you have sized your environment (CPU, memory, and disk space) correctly to take advantage of the partitioning features.

  3. Tune database parameters for memory, tablespace, and partitions to get maximum performance. For more information, see the "Tuning Database Parameters" section in the Oracle Fusion Middleware Performance and Tuning Guide.

  4. For other tablespaces running out of space, use the following query to check for free tablespace:

    SELECT Total.tablespace_name "Tablespace Name", Free_space, (total_space-Free_
    space) Used_space, total_space, round((Free_space*100/total_space),2) "Free %"
    FROM (select tablespace_name, sum(bytes/1024/1024) Free_Space from sys.dba_
    free_space group by tablespace_name) Free,
    (select tablespace_name, sum(bytes/1024/1024) TOTAL_SPACE from sys.dba_data_
    files group by tablespace_name) Total
    WHERE Free.Tablespace_name = Total.tablespace_name AND Total.tablespace_name =
     '<tablespacename>'
    ORDER BY 5;
    
  5. To increase tablespace settings, use the administrator account. For example:

    alter tablespace tablespace_name add datafile 'datafile_name' size 500m
    autoextend on;
    

For more details, see the "Resolving Message Failure Caused by Too Many Open Files" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.7.2 Slow Application Performance Such as Longer Time to Serve Pages or Finish Transactions

Problem

You observe slow application performance and/or memory trashing. For example, it may take longer to load and serve pages or to complete composite transactions. Response time may seem slower compared to normal behavior.

There are various reasons for slow performance. It may be due to a large number of servers running on the same host, or there may be a large number of records/sessions/locking at the database. Thread contention can also be a reason for poor performance.

Note:

All configuration changes in Fusion Applications Control may be unavailable due to this problem.

Solution

  1. Check the CPU utilization to see if it is saturated due to a heavy load or an excessive number of processes in relation to CPU capacity.

    If CPU utilization is at 100% during normal load hours (the target should be 70-80%), you have no capacity to handle a peak load and the hardware resources are insufficient. Add scale-out servers to handle the additional load.

  2. Check applications using Fusion Applications Control to report on performance. Check the performance of ADF services and the invoke/response times in the BPEL process audit trail to identify if the issue is in Oracle Fusion Applications.

  3. To maximize performance, it is recommended that you not set the logging level higher than the default INFO level.

    For debugging purposes, you must set the logging level to the FINEST level. However, after issues are resolved, reset the logging level to the default level for best performance. It is also recommended that you set Audit Level to Production in the SOA Infrastructure Common Properties page. This can be set in Fusion Applications Control as follows:

    1. Right-click soa-infra (SOA_cluster_name).

    2. Choose SOA Administration > Common Properties.

    3. Set Audit Level to Production.

  4. Purge periodically based on retention requirements to maintain the service level agreements (SLAs).

    To identify the tables where data growth may lead to performance issues, see the "Tables Impacted By Instance Data Growth" section in the Oracle Fusion Middleware Performance and Tuning Guide.

    For information about the use of the purge scripts, see the "Managing Database Growth" chapter in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

  5. Check the memory/IO/paging/swapping/CPU usage load statistics using Top or Glance or another monitoring tool.

  6. Optimize the JVM to avoid full garbage collection or out-of-memory errors.

    Frequent garbage collection can be either due to higher memory usage or memory leaks.

    1. Ensure that the sum of the maximum heap size of all the JVMs running on your system does not exceed the amount of available physical RAM to avoid operating system level paging activity.

    2. Use the JRockit mission control memory profiling tools to get thread dumps and memory snapshots, which helps Oracle Support Services debug any code issues.

  7. Optimize threads to avoid contention. Get a thread dump to investigate and submit it to Oracle Support Services.

  8. Check the JVM and thread dumps for methods invoking the database to identify if database performance is a bottleneck.

  9. Run database Automatic Workload Repository (AWR) snapshots to identify causes of database performance issues.

  10. Ensure that database statistics are updated at regular intervals and other tunable parameters for memory, tablespace, and partitions are used effectively to obtain maximum performance.

    Here are some common tuning recommendations. For more information, see the "Tune Database Parameters" section in the Oracle Fusion Middleware Performance and Tuning Guide.

    • Put indexes and tables in as physically separate disk areas if possible.

    • Never put rollback segments with data or index segments.

    • Separate highly active tables and indexes into their own tablespaces.

    • Partition high activity tables and indexes to help balance disk I/O and prevent hot disks.

    • Have processes in place to generate database table statistics at regular intervals.

  11. Tune database tables to control the high watermark (HWM) contention of large objects. Tune database advanced queues (AQ) to control high watermark (HWM) contention and ensure consistent performance of producing and consuming messages from AQ.

    Specifically, the AQs to be aware of are as follows:

    Family AQ Queue Name

    EDN

    EDN_EVENT_QUEUE_TABLE and EDN_OAOO_DELIVERY_TABLE

    CRM

    CRM_AsyncWS_Request and CRM_AsyncWS_Response

    HCM

    HCM_AsyncWS_Request and HCM_AsyncWS_Response

    FIN

    FIN_AsyncWS_Request and Fin_AsyncWS_Response

    PRC

    PRC_AsyncWS_Request and PRC_AsyncWS_Response

    PRJ

    PRJ_AsyncWS_Request and PRJ_AsyncWS_Response

    SCM

    SCM_AsyncWS_Request and SCM_AsyncWS_Response

    COMMON

    COMMON_AsyncWS_Request and COMMON_AsyncWS_Response

    IC

    IC_AsyncWS_Request and IC_AsyncWS_Response

    Cross Family Business Event Subscriptions

    ACR_XFAMILY_EVENT_Q and ACR_XFAMILY_EVENT_QT


  12. Tune the BPEL and EDN thread counts to ensure optimal settings (for example, the Dispatcher Invoke Threads and Dispatcher Engine Threads properties on the BPEL Service Engine Properties page in Fusion Applications Control). If the thread configuration is too high, the servers run out of memory. If they are too low, the messages start backing up.

  13. Tune the BPEL properties to reduce overhead (for example, disable the ValidateXML and StatsLastN (statistics gathering batch size) properties on the BPEL Service Engine Properties page in Fusion Applications Control) if they are not needed.

  14. In case of integration with packaged applications (for example, Siebel), check if the issue lies with the legacy applications.

For more information about performance tuning the various components, see the "Top Performance Areas" chapter and the "SOA Suite Components" part in the Oracle Fusion Middleware Performance and Tuning Guide.

For information about setting properties on the BPEL Service Engine Properties page, see the "Configuring BPEL Process Service Engine Properties" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.7.3 Slow Fusion Applications Control Performance

Problem

Fusion Applications Control pages are loading very slowly.

Solution

To improve Fusion Applications Control page load times on pages with instance and fault metrics, set the SOA Infrastructure Display Data Counts properties.

  1. Right-click soa-infra (SOA_cluster_name).

  2. Choose SOA Administration > Common Properties.

  3. In the Display Data Counts section, deselect the Disable fetching of instance and fault count metrics checkbox.

  4. Reduce the Duration Details value.

  5. Click Apply.

In addition to setting this property, it is also useful to scope queries by time where available (for example, only show items occurring in the last hour).

For more information, see the "Optimizing the Loading of Pages with Instance and Fault Metrics" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.8 Maintenance

This section contains the following topics that describe common problems and solutions for Oracle SOA Suite maintenance:

9.8.1 JVM is Suspended If Any In-use Log Directories are Deleted

Problem

JRockit writes log information (called flight recorder files) under /tmp when using JRockit Mission Control. When it rotates to a new flight recorder file, the JVM is suspended until it completes. If the old directories in use under /tmp are deleted (for example, a cleanup or custom maintenance script to run to clear /tmp when it is close to full to avoid server downtime), the domain may go into a suspended state because the JVM may wait for a directory that no longer exists.

Solution

  1. Change any cleanup scripts/procedures to first shut down the domain cleanly.

  2. Perform a cleanup of /tmp.

  3. Restart the domain.

To recover existing, suspended JVMs without restarting them, you must recreate the directories that the JVM was trying to use.

9.8.2 Server Start Script Does Not Work for Scale-Out (Postprovisioning) Servers

Problem

Scale-out servers, created after provisioning, cannot be started using back-end commands. The script start_fusion_Allservers.sh only starts servers that are created during provisioning time.

Solution

To start scale-out servers, use Fusion Applications Control. See the "Starting and Stopping an Oracle WebLogic Server Cluster Using Fusion Applications Control" section in the Oracle Fusion Applications Administrator's Guide.

9.8.3 SOA Server Does Not Start

Problem

The SOA server (soa_infra) does not start.

Solution

There can be various reasons for server startup issues. A discovery-based approach to find the root cause is required. Check the server and diagnostic logs as a first attempt to diagnose the issue. In addition, check the following:

  1. Check if the database is not available or there are not enough connections available. Some failures with SOA runtime can result from database outage/connectivity issues. Perform the following steps:

    1. Log in to Oracle WebLogic Server Administration Console.

    2. In the Domain Structure, view the status by selecting Services > Data Sources > SOADatasource > Monitoring > Testing to test the data source. You can also check this from a SQL prompt.

  2. Check the list of ports used for port conflicts.

  3. Check if the MDS repository is unable to load shared documents (incorrect MDS configuration or the database that holds the MDS schema is not reachable).

  4. Check the Coherence configuration if using a SOA cluster (use of unicast versus multicast).

    This issue only applies after provisioning is complete. Clustering of SOA servers may fail if there are port conflicts (used by Coherence). For example, one SOA server may be picking up the deployment, but the other server is not. In some cases, this may only present the following error:

    [soa_server1] [ERROR] [] [Coherence] [tid: Logger@352821903 3.6.0.4] 
    [ecid: 46f620208907e045:63f295ec:12dd091ec2e:-8000-0000000000000003,1:27187] 
    [APP: soa-infra] 2011-01-28 23:06:19.463/414.816 Oracle Coherence GE 3.6.0.4 
    <Error> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
     (self-tuning)', member=n/a): 
    Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout
     during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster[[...
    

    In this example, Coherence timeouts are prominently available. However, sometimes it presents itself with an unrelated error, such as the following:

    Error creating bean with name 'SensorManager' defined in ServletContext
    resource
    

    The root cause of this is still primarily related to Coherence configuration.

  5. Check if both the Administration Server and Managed Server ports are open and accessible.

  6. Check if Managed Server startup failed because the Administration Server is not reachable.

  7. Check network issues (for example, IP routing filtering/rules that may be causing issues).

  8. Check Oracle WebLogic Server LDAP security corruption.

    The Managed Server may report that policies for the application System MBeans Browser already exist or do not exist.

For more information, see the "Setting the Frontend URL for the Administration Console and Setting Redirection Preferences" section and the "Setting the Frontend HTTP Host and Port" section in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

9.8.4 SOA Server Is Not Responding To Administrative Commands

Problem

The SOA server (soa_infra) is not responding to administrative commands. This problem can be due to your environment using a load balancer, and the default listener address or channel is local-only. Therefore, the connection cannot be made from the external client or server.

Solution

To resolve this problem, perform the following steps from the Oracle WebLogic Server Administration Console:

  1. Ensure that the listener address is correctly defined and accessible.

    1. From the Domain Structure, expand Environment and select Servers.

    2. From the Servers table, click AdminServer(admin).

    3. From the Settings for AdminServer page, click the Protocols tab.

    4. Click the HTTP subtab.

    5. Set the Frontend Host property to the load balancer address.

  2. Similarly, set the Frontend HTTP Host and Frontend HTTP Port properties for the SOA cluster:

    1. From the Domain Structure, expand Environment and select Clusters.

    2. From the Clusters table, click domain_SOACluster

    3. From the Settings for the domain_SOACluster page, click the HTTP subtab.

    4. Set the Frontend Host property to the load balancer address.

    5. Change the Frontend HTTP Host and Frontend HTTP Port properties.

For more information, see the "Setting the Frontend URL for the Administration Console and Setting Redirection Preferences" section and the "Setting the Frontend HTTP Host and Port" section in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

9.8.5 Undeploying a Composite Left in an Inconsistent State Due to a Failed Patch

For troubleshooting information about this issue, see the "Troubleshooting SOA Composite Deployment Failures" section in the Oracle Fusion Applications Patching Guide.

9.8.6 Increased Load Observed on RAC Instances when Using Multiple Oracle RAC Servers

Problem

An increased load is seen on Oracle Real Application Clusters (Oracle RAC) instances when using multiple Oracle RAC servers.

Solution

Use a multidata source against Oracle RAC instances using load balancing rather than failover. This ensures that the load is evenly distributed rather than all load staying on one node until it fails over, at which point all the connections are failed over to the other node.

Ensure that you set the Connection Test Frequency Seconds property at the multidata source level to a nonzero value. A value that is too low means more load on the listeners/RAC instances when multiple servers attempt to reconnect at the same time.

For more information about configuring JDBC multidata sources, see Oracle Fusion Middleware Configuring and Managing JDBC Data Sources for Oracle WebLogic Server.

Note:

The Oracle Fusion Applications default configuration for the these settings has been tuned for best performance. Only change these settings when the configuration has not been set out-of-the-box by Oracle Fusion Applications.

9.8.7 List of SOA Composite State Values

Problem

Get a list of all composite state values from the *_SOAINFRA.COMPOSITE_INSTANCE state table (for diagnosis).

Solution

Table 9-3 shows the composite state values.

Table 9-3 Composite State Values

State Description

0

Running

1

Completed

2

Running with faults

3

Completed with faults

4

Running with recovery required

5

Completed with recovery required

6

Running with faults and recovery required

7

Completed with faults and recovery required

8

Running with suspended

9

Completed with suspended

10

Running with faults and suspended

11

Completed with faults and suspended

12

Running with recovery required and suspended

13

Completed with recovery required and suspended

14

Running with faults, recovery required, and suspended

15

Completed with faults, recovery required, and suspended

16

Running with terminated

17

Completed with terminated

18

Running with faults and terminated

19

Completed with faults and terminated

20

Running with recovery required and terminated

21

Completed with recovery required and terminated

22

Running with faults, recovery required, and terminated

23

Completed with faults, recovery required, and terminated

24

Running with suspended and terminated

25

Completed with suspended and terminated

26

Running with faulted, suspended, and terminated

27

Completed with faulted, suspended, and terminated

28

Running with recovery required, suspended, and terminated

29

Completed with recovery required, suspended, and terminated

30

Running with faulted, recovery required, suspended, and terminated

31

Completed with faulted, recovery required, suspended, and terminated

32

Unknown

64

State


9.8.8 Log Files Have Been Deleted or Are Missing

Problem

The SOA log files are getting deleted. This is typically due to the retention policy set in Fusion Applications Control.

Solution

Change the retention policy for the odl-handler to the appropriate value in Fusion Applications Control:

  1. Right-click soa-infra.

  2. Choose Logs > Log Configuration.

9.9 Custom Development (Extensibility)

This section contains the following topics that describe common problems and solutions for Oracle SOA Suite runtime:

9.9.1 Unable to Use Layered Customization on Specific Artifacts

Problem

Only certain artifacts allow layered customization in Oracle JDeveloper because it requires editors that use XMLEF to make delta changes to the underlying Document Object Model (DOM). XMLEF is the XML Editing Framework. XMLEF is an Oracle JDeveloper framework that supports flexible editing of XML documents and is used extensively in layered customizations.

Solution

The following SOA artifact types are customizable:

  • Composite (composite.xml)

  • BPEL process (.bpel)

  • Oracle Mediator (.mplan)

The following SOA artifact types are not customizable:

  • XSL map (.xsl)

  • Human task (.task)

  • Decision service (.decs)

  • Rules (.rules)

  • Event definition (.edl)

  • Domain value map (.dvm)

The following common artifact types are not customizable because Oracle XDK/WSDL APIs do not support customization (although their editors do use XMLEF to make delta changes to the underlying DOM):

  • XSD schema (.xsd)

  • WSDL document (.wsdl)

Noncustomizable artifacts are not sensitive to a customization context. Any changes to those artifacts from any customization context are visible to other customization contexts.

For more information about customizations in Oracle JDeveloper and runtime tools (such as Oracle SOA Composer and Oracle BPM Worklist), see the "Customizing and Extending SOA Components" chapter in the Oracle Fusion Applications Extensibility Guide.

9.9.2 New or Deleted Artifacts Do Not Appear as a Layered Customization

Problem

Because MDS customization does not capture artifact creations or deletions as deltas in the customization role, artifact creations and deletions in one customization context are visible to other customization contexts.

Solution

Artifact creations and deletions are not customization context sensitive. Therefore, only existing artifact customizations are supported in layers. Because new artifacts cannot be added as part of a layer, the artifacts are added to the project like any other base artifact. Any component artifacts that are added to the project are not incorporated into the runtime composite model unless the composite.xml file has explicitly referred to it. Therefore, if a customization did not alter composite.xml to include the newly added component, the new artifacts have no impact. Of course, the new artifacts are saved along with the base artifacts.

Additions of any new artifact introduced in the composite definition as a customization (for example, XSLT artifacts) must be named appropriately to prevent collisions with base XSLT artifacts. References to the new XSLT files must be made within the Oracle BPEL Process Manager/Oracle Mediator artifacts (through a layer).

9.9.3 Warnings Regarding xml:id when Building or Deploying a Customized Composite

Problem

When building or deploying the customized composite, you may get warnings similar to this because the xml:id element is not supported:

[scac] [WARNING] Line [13] Column [29] Schema validation failed for
 C:\customizationdemo\project1\composite.xml
<Line 13, Column 29>: XML-24535: (Error) Attribute
 'http://www.w3.org/XML/1998/namespace:id' not expected.

Solution

This warning can be ignored and should not prevent you from deploying the composite successfully.

9.9.4 Distributed Queue Topic Messages Are Retrieved Multiple Times by Subscribers

Problem

In a clustered environment, a queue topic is being retrieved multiple times by the subscribers. SOA clusters are typically homogenous (that is, each node has the same services running). Therefore, if a queue topic has multiple subscribers (for example, an error queue is subscribed to by a fallout notification service and a trouble ticketing service), then in a clustered environment, each service is running on every node. If it is a four-node cluster, then there are four instances of the fallout notification service and four instances of the trouble ticketing service, each of which retrieve the topic message. This is not desirable.

Solution

Set the consuming service singleton property to ensure that in a multinode environment, only one of the subscriber instances can consume the topic message. In the example in the "Problem" section, the fallout notification service and the trouble ticketing service each should set their singleton property so that each of them consumes the message only once by any one of the four instances.

<service name="ConsumeFaultMessage" ui:wsdlLocation="ConsumeFaultMessage.wsdl">
<interface.wsdl
interface="http://xmlns.oracle.com/pcbpel/adapter/jms/EH/AIAReadJMSNotificationPro
cess/ConsumeFaultMessage/#wsdl.interface(Consume_Message_ptt)"/>
<binding.jca config="ConsumeFaultMessage_jms.jca">
  <property name="singleton">true</property>
</binding.jca>
</service>

9.9.5 Internal Key is Displayed Instead of a Translated String

Problem

An internal key is displayed instead of a translated string in the human task title, human task outcome, email subject and body, and human task mapped attributes displayed in Oracle BPM Worklist.

Solution

To resolve this problem:

  1. Check the SOA server-diagnostic.log for errors for each server in the cluster. If a resource bundle or the key is not found, you see a similar error stack that contains the composite name and resource bundle name.

    2010-09-29T23:45:46.702-07:00 soa_server1 ERROR \[\]
     oracle.soa.services.workflow.soa tid: orabpel.invoke.pool-4.thread-7 userId:
     weblogic
    ecid:59ab739a26595dc1:-38d33932:12b614ddb44,0:1:100000026
    APP: soa-infra composite_name: RC3HumanTaskComposite component_name: Humantask1
     component_instance_id: composite_instance_id: 310003
    <WorkflowServiceEngine.request> Operation 'initiateTask' failed with exception
       'EJB Exception: : java.util.MissingResourceException:
    Can't find bundleforbase name
       oracle.apps.hcm.people.soa.resource.HcmPeopleTopSoaBundleDummy, locale
     en\[\[
    at
    java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1427
    )
    at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1250)
    at java.util.ResourceBundle.getBundle(ResourceBundle.java:952)
    
  2. Confirm that the resource bundle has been deployed to the correct location in SOA MDS by first exporting the contents of SOA MDS and confirming that the resource bundle class is defined under /apps/resource. Follow these steps to export the metadata contents from Fusion Applications Control:

    1. From the navigation pane, expand SOA and select soa-infra.

    2. From the SOA Infrastructure menu, choose Administration > MDS Configuration.

    3. Select Export metadata documents to an archive on the machine where the web browser is running.

    4. Click Export.

  3. If the resource bundle is not defined in SOA MDS or is in the wrong location in SOA MDS, then you may have encountered an error during patching or there is a problem with the resource bundle definition.

    If the resource bundle is properly defined in SOA MDS and the string that is not translated is a mapped attribute that appears in the column heading of the worklist, then perform the followings steps to:

    • Set the System MBean Browser WorkflowCustomClasspathURL property to a value of "".

    • Apply the changes.

    • Set WorkflowCustomClasspathURL to "oramds:///apps/resource/".

    • Apply the changes in Fusion Applications Control.

    1. Log in to Fusion Applications Control in the domain where the JAR file was deployed.

    2. From the navigation pane, expand SOA and select soa-infra.

    3. From the SOA Infrastructure menu, choose SOA Administration > System MBean Browser.

    4. Go to Application Defined MBeans > oracle.as.soainfra.config > Server: SOA_cluster_name > WorkflowConfig > human-workflow.

    5. Remove the value in the Value column for the WorkflowCustomClasspathURL attribute.

    6. Click Apply.

    7. Enter oramds:///apps/resource/ in the Value column of the WorkflowCustomClasspathURL attribute.

    8. Click Apply.

    If the resource bundle is properly defined in SOA MDS and the string that is not translated is not a mapped attribute that appears in the column heading of the worklist, then stop and restart the composite that references the resource bundle by following these steps:

    1. Log in to Fusion Applications Control in the domain where the JAR file was deployed.

    2. From the navigation pane, expand SOA and select soa-infra.

    3. Click Shut Down.

    4. Click Yes in the confirmation window.

    5. Click Start Up.

    6. Click Yes in the confirmation window.

  4. Retest your transaction flow.

9.9.6 BPEL Activity Errors and the Log Shows Error "com.oracle.bpel.entity.dataprovider.EntityVarMgrException:zero Item"

Problem

If entity variables are used in Oracle BPEL Process Manager, this error may be thrown at runtime:

com.oracle.bpel.entity.dataprovider.EntityVarMgrException:zero item

Solution

When the entity variable tries to bind the key provided to it at runtime in Oracle BPEL Process Manager and if it cannot find the key in the database, it throws this error. This may be due to human error. Either the developer has passed an incorrect primary key value to bind to the entity variable or the composite has been deployed to the wrong business component (BC) URL in the deployment plan. To resolve, ensure that the primary key value that gets bound to the entity variable at runtime exists in the database used by the BC service.

9.9.7 BPEL Activity Errors and Log Shows Error "XPath variable or expression <expression> is empty"

Problem

If the payload node used in an activity such as an assign activity in BPEL is empty/null, then it throws this error.

The XPath variable or expression
/ns4:findDtsPurchasingDeliverable1FindByDelStatusResponse/ns4:resultns2:Deliverabl
eId=$DeliverableIdVar/ns2:BatchId is empty at line 344. selectionFailure:
 com.oracle.bpel.client.BPELFault: faultName:
 {{http://schemas.xmlsoap.org/ws/2003/03/business-process/}selectionFailure}

Solution

This error is thrown when a selection operation performed either in a function or in an assignment encounters an error. If there is a possibility of getting a null/empty value for a payload element at runtime, then you can get around this error by enclosing this element in a switch case with a condition check of (string-length(payload_element) > 0) or a bpelx:assert element to test for null conditions in their code. If these errors are encountered at runtime, there is information in the log and the BPEL audit trail flow.

9.9.8 Unavailable Composite Errors Occurring Even Though the Target Service Is Up and Running

Problem

When a BPEL process has partner links targeted to external servers, during start up of the SOA server or activation of the BPEL composite, the BPEL service engine recognizes that the target server is down and marks the BPEL composite as unavailable. Even after the target server becomes available, there is no way to make the BPEL process available without restarting the complete SOA server or redeploying the BPEL process. Until then, the BPEL composite is not listed in the navigation pane of Fusion Applications Control and is only visible through the Deployed Composites tab. Even though the composite is displayed here (sometimes as active), if you click the process, you receive this error:

The composite <Process Name>[ <Version> ] is not available. This could happen
because either the composite has been undeployed or EM has not yet discovered
this composite.

Solution

The issue is caused by using a concrete WSDL file in the BPEL definition. Using an abstract WSDL prevents the BPEL composite/process from becoming inactive or its state from becoming retired when the target server or service is down.

Oracle Fusion Applications develop composites with oramds:// references to WSDL files at design time for both the binding and the WSDL location. At deployment, the binding's concrete location (which is only used at runtime) is fixed.

Follow these steps to use an abstract WSDL by making a local copy of the target WSDL file in your BPEL project.

  1. When you create a web service partner link in a composite in the Create Web Service dialog, select the copy WSDL and its dependent artifacts into the project checkbox.

  2. After you are done with web service creation and the partner link is wired (connected) with the BPEL process service component, you see two WSDL files in the Application Navigator of Oracle JDeveloper:

    • Webservice.WSDL (the local copy of your target WSDL)

    • WebserviceRef.WSDL

  3. Redeploy the SOA composite.

  4. Retest the scenario.

9.9.9 High-Volume Cross Reference (XREF_DATA) Table Impacts Performance and Maintenance

Problem

When the volume of data in the XREF Oracle SOA schemas grows very large, maintaining the database can become difficult. It may have an impact on performance and may be identified in AWR snapshots.

Solution

To address this maintenance challenge, Oracle SOA 11g has been instrumented with partition keys that enable database administrators (DBAs) to take advantage of the Oracle RDBMS partitioning features and capabilities. Even thought multiple types of partitioning are possible, the list partitioning strategy is the preferred one for partitioning the cross reference data table.

List partitioning allows greater flexibility in the mapping of rows to partitions than range or hash partitioning. Because the cross reference data (XREF_DATA) is distributed based on discrete column values, unordered and unrelated sets of data can be grouped together with no relationship between the partitions. For this table, the list partitioning is based on the column XREF_TABLE_NAME. Because you know the list of entities that have cross reference data, this strategy is ideal. The only caveat is that you must repeat this activity if a new entity gets added to the system.

Better understanding of data distribution can lead to further refinement and space management by employment of composite partitioning strategies such as list - list and list - range schemes. For information about partitioning, see the Oracle Database Performance Tuning Guide.

In addition to this partitioning strategy, you can also use Oracle Mediator's feature by which custom database tables can be created to store cross reference data for certain high volume entities. For more information, see Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

9.9.10 Access Denied Error While Invoking an Oracle ADF BC Service from BPEL

Problem

A common case in Oracle Fusion Applications is Oracle ADF BC service invocations from various SOA components. If the user context attached with the SOA composite instance or identity switching used in a SOA reference does not have proper permission to access the Oracle ADF BC service, then an access denied error is displayed.

Solution

Common causes for this error are as follows. These must be resolved.

  • The Oracle ADF BC service does not have required permissions in jazn-data.xml.

  • The permissions provided in jazn-data.xml are not properly reflected in system-jazn-data.xml.

  • The ejb-jar.xml file does not have the <interceptors> tag added or it has an incorrect application name. An example of ejb-jar.xml with the correct tags is as follows:

    <ejb-jar xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
     [http://java.sun.com/xml/ns/j2ee/ejb-jar_3_0.xsd]"version="3.0">
    <enterprise-beans>
    <session>
    <ejb-name>oracle.apps.financials.payables.invoices.transactions.invoiceApproval
    Service.InvoiceApprovalBCServiceBean</ejb-name>
    <resource-ref><res-ref-name>jdbc/ApplicationDBDS</res-ref-name><res-type>javax.
    sql.DataSource</res-type><res-auth>Container</res-auth></resource-ref>
    <resource-ref><res-ref-name>jdbc/ApplicationServiceDBDS</res-ref-name><res-type
    >javax.sql.DataSource</res-type><res-auth>Container</res-auth></resource-ref>
    </session>
    </enterprise-beans>
    &nbsp;
    <interceptors> <interceptor>
    <interceptor-class>oracle.security.jps.ee.ejb.JpsInterceptor</interceptor-class
    >
    <env-entry> <env-entry-name>application.name</env-entry-name>
     <env-entry-type>java.lang.String</env-entry-type>
     <env-entry-value>fscm</env-entry-value>
    <injection-target>
    <injection-target-class>oracle.security.jps.ee.ejb.JpsInterceptor</injection-ta
    rget-class>
    <injection-target-name>application_name</injection-target-name>
    </injection-target>
    </env-entry>
    </interceptor> </interceptors>
    &nbsp;
    <assembly-descriptor>
    <interceptor-binding>
    <ejb-name>*</ejb-name><interceptor-class>oracle.security.jps.ee.ejb.JpsIntercep
    tor</interceptor-class> </interceptor-binding>
    </assembly-descriptor>
    &nbsp;
    </ejb-jar>
    

9.9.11 Clicking Manage Approvals Page Gives a "SOA server may be down" or "No data to display" Error

Problem

At runtime, when you click the Manage Approvals page (for both a requisition and purchase order), you may get a SOA server may be down error among other errors. The page may also show a No data to display error.

Solution

Check to see if you have wf_client_config.xml pointing to the correct SOA server. Oracle Fusion Applications environments are all multidomain and applications that must query their local SOA runtime must do so through the wf_client_config.xml file configuration.

For more information about the wf_client_config.xml file, see Section 9.5.2.

9.9.12 Synchronous Service Invocation Errors Due to WS-Addressing

Problem

When integrating with external web services, you may have issues related to the WS-Addressing headers in the request and response XML documents used in synchronous request-response scenarios. This may even occur if there is a dynamic endpoint that resolves to an external web service that does not support WS-Addressing.

  • A SOA composite acting as a client fails to invoke a service provider that does not support WS-Addressing headers in the request message for synchronous interactions.

  • An external service consumer that does not support WS-Addressing headers in the response message fails to invoke a synchronous operation of a SOA composite.

  • A service provider throws a wsa:InvalidAddressingHeader error when SOAPAction is defined in the WSDL <binding> element of the service provider interface.

Solution

The WS binding component of Oracle Fusion Middleware 11g generates WS-Addressing headers by default, which is as per the specification, but may be incompatible with some web services. This behavior can be overwritten using properties that allow you to control the generation of WS-Addressing header information for interoperability purposes.

  • A SOA composite acting as a client fails to invoke a service provider that does not support WS-Addressing headers in the request message for synchronous interactions.

    When your Oracle SOA Suite components act as service clients (consumers) and invoke synchronous operations of external web services, the invocation fails. The invocation failure is due to the WS-Addressing headers included in the request message when invoking the service provider.

    The stack trace displays the following information:

    NSGetCustomer (faulted)
    Nov 10, 2009 4:26:40 PM Faulted while invoking operation "get" on provider
    "NetSuite2009_PL".
    <payload>
    Nov 10, 2009 4:26:40 PM
    "{urn:platform_2009_1.webservices.netsuite.com}UnexpectedErrorFault" has been
    thrown.
    <payload>
    Nov 10, 2009 4:26:40 PM There is a system exception while performing the BPEL
    instance, the reason is "Invalid SOAP Header: '<ns1:To
    xmlns:ns1="http://www.w3.org/2005/08/addressing">https://webservices.netsuite.c
    om/services/NetSuitePort_2009_1</ns1:To>'. Value is
    'https://webservices.netsuite.com/services/NetSuitePort_2009_1'."
    <payload>
    

    In Oracle SOA Suite 11g, the WS-Binding component includes the following WS-Addressing headers:

    • wsa:to

    • wsa:replyTo

    • wsa:action

    These headers are included by default regardless of interacting with synchronous or asynchronous web service operations. Although the SOAP message generated by 11g is fully specification compliant, there are service provider implementations that do not understand these WS-Addressing headers when included in the request message for synchronous operations. This is why the invocation fails.

    Oracle Fusion Middleware 11g uses new standards, WS-Addressing (wsa) and WS-I Basic profile, which demand that WS-Addressing headers be added by default to the request message.

    To increase interoperability, the property oracle.soa.ws.outbound.omitWSA has been introduced for the <binding.ws> element in the composite.xml file. This property enables you to optionally suppress generation of WS-Addressing headers in the request message when set to true. The default value is false.

    1. Manually add the following property to the <binding.ws> element in the composite.xml file.

      [snippet]
      <binding.ws
      ......
          <property name="oracle.soa.ws.outbound.omitWSA" type="xs:boolean"
      many="false" override="may">true</property>
      </binding.ws>
      [/snippet]
      
    2. Redeploy the composite.

  • An external service consumer that does not support WS-Addressing headers in the response message fails to invoke a synchronous operation of a SOA composite.

    When external services invoke the endpoints of Oracle SOA Suite composites, which expose synchronous operation(s) acting as a server (service provider), the external web service fails with invocation errors. This is commonly due to the WS-Addressing headers included in the response message, which are not supported by some web service implementations.

    In SOA Suite 11g, the WS-Binding component includes the WS-Addressing headers by default in the response messages. Although the SOAP message generated by 11g is fully specification compliant, there are service consumer implementations that do not understand the WS-Addressing headers when included in the response message in synchronous request-response scenarios. This is why the invocation fails.

    To increase interoperability with clients (consumers) that do not support WS-Addressing headers, a property named oracle.soa.addressing.response.enabled has been introduced. This property can be used within the <binding.ws> element in the composite.xml file. This property enables you to optionally suppress generation of WS-Addressing headers in the response message when set to true. The default value is false.This means that in a synchronous request-response scenario, in which the Oracle SOA Suite composite acts as a service provider, WS-Addressing headers are included in the response message by default. If the property is set to false, no WS-Addressing-related headers are returned in the response message.

    1. Set the property in the composite.xml file as follows:

      [snippet]
      <binding.ws>
      ......
      <property name="oracle.soa.addressing.response.enabled" type="xs:boolean"
      many="false" override="may">false</property>
      </binding.ws>
      [/snippet]
      
  • A service provider throws a wsa:InvalidAddressingHeader error when SOAPAction is defined in the WSDL <binding> element of the service provider interface.

    When invoking a web service provider with a synchronous interface from a composite deployed on 11.1.1.1, a wsa:InvalidAddressingHeader message is returned. This happens when both of the following conditions are met:

    • The service interface (WSDL) of the service provider does not include the wsa:Action element in the WSDL input message for the port operations

      and

    • The soapAction attribute in the WSDL binding section of the service provider interface is specified.

    The difference with the first scenario is that the service provider interface may accept WS-Addressing headers for synchronous endpoint operations by using the WS-Addressing extensibility attribute wsa:UsingAddressing in the WSDL binding section. A WSDL binding section sample is shown as follows:

    <wsdl:binding name="mySoapBinding"
    type="impl:myService">
      <wsaw:UsingAddressing wsdl:required="false"
     xmlns:wsaw="http://www.w3.org/2006/05/addressing/wsdl"/>
      <wsdlsoap:binding style="document"
     transport="http://schemas.xmlsoap.org/soap/http"/>
      <wsdl:operation name="myOperation">
        <wsdlsoap:operation soapAction="myAction"/>
        <wsdl:input name="myRequestMsg">
          <wsdlsoap:body use="literal"/>
        </wsdl:input>
        <wsdl:output name="myResponseMsg">
          <wsdlsoap:body use="literal"/>
        </wsdl:output>
      </wsdl:operation>
    </wsdl:binding>
    

    As the following code sample shows, neither the input message nor the output message (as defined in the service provider WSDL) contain a <wsa:Action> element:

    ....
    <wsdl:message name="myRequestMsg">
      <wsdl:part name="parameters" 
    element="impl:myRequestElement"/>
    </wsdl:message>
    .....
    <wsdl:message name="myresponseMsg">
      <wsdl:part name="parameters" 
    element="impl:myResponseElement"/>
    </wsdl:message>
    

    The problem has been observed when integrating with synchronous web services deployed on the IBM WebSphere 6.1 Application Server.

    The stack trace looks similar to the following code example:

    ......
    oracle.fabric.common.FabricInvocationException:
    javax.xml.ws.soap.SOAPFaultException: A header representing a Message
    Addressing Property is not valid and the message cannot be processed
    at
    oracle.integration.platform.blocks.soap.WebServiceExternalBindingComponent.thro
    wFabricInvocationException(WebServiceExternalBindingComponent.java:415)
    at
    oracle.integration.platform.blocks.soap.WebServiceExternalBindingComponent.thro
    wFabricInvocationExceptionForSoapFault(WebServiceExternalBindingComponent.java:
    411)
    at
    oracle.integration.platform.blocks.soap.WebServiceExternalBindingComponent.proc
    essSOAPFault(WebServiceExternalBindingComponent.java:394)
    at
    oracle.integration.platform.blocks.soap.WebServiceExternalBindingComponent.proc
    essOutboundMessage(WebServiceExternalBindingComponent.java:253)
    at
    oracle.integration.platform.blocks.soap.WebServiceExternalBindingComponent.send
    SOAPMessage(WebServiceExternalBindingComponent.java:644)
    at
    oracle.integration.platform.blocks.soap.WebServiceExternalBindingComponent.requ
    est(WebServiceExternalBindingComponent.java:526)
    at
    oracle.integration.platform.blocks.mesh.SynchronousMessageHandler.doRequest(Syn
    chronousMessageHandler.java:139)
    at
    oracle.integration.platform.blocks.mesh.MessageRouter.request(MessageRouter.jav
    a:179)
    at
    oracle.integration.platform.blocks.mesh.MeshImpl.request(MeshImpl.java:144)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    ......
    

    If you use HTTP Analyzer or Apache TCP Monitor to trace the message exchange, you observe that the issue is related to the <wsa:action> element in the request SOAP header, which is not the same as SOAPAction in the HTTP header. The response message looks like the following code sample:

    <soapenv:Envelope>
    .......
      <soapenv:Header>
         <wsa:FaultDetail xmlns:wsa="http://www.w3.org/2005/08/addressing">
            <wsa:ProblemHeaderQName>wsa:Action</wsa:ProblemHeaderQName>
         </wsa:FaultDetail>
      </soapenv:Header>
      <soapenv:Body>
        <soapenv:Fault>
          <faultcode>wsa:InvalidAddressingHeader</faultcode>
          <faultstring>
            <![CDATA[A header representing a Message Addressing Property is not
            valid and the message cannot be processed]]></faultstring>
        </soapenv:Fault>
       </soapenv:Body>
    </soapenv:Envelope>
    

    As mentioned in the previous sections, the WS Binding component of Oracle Fusion Middleware 11g generates WS-Addressing headers by default. According to the W3C specification (Web Services Addressing 1.0 - WSDL Binding W3C Candidate Recommendation 29 May 2006), Section 4.4.1 Explicit Association:

    "In the absence of a wsa:Action attribute on a WSDL input element where a SOAPAction value is specified, the value of the [action] property for the input message is the value of the SOAPAction specified."

    The generated value for the <wsa:Action> element of the input message matches the value of the SOAPAction element, if that is specified in the WSDL binding section. Otherwise, it generates a default value for the <wsa:Action> element.

    A sample of the generated request is as follows (provided WSDL <binding> and <messages> section, as shown previously).

    POST /currencyratesapi/services/myAction HTTP/1.1
    Host: <host>:<port>
    Connection: TE
    TE: trailers, deflate, gzip, compress
    User-Agent: Oracle HTTPClient Version 10h
    SOAPAction: "myAction"
    Accept-Encoding: gzip, x-gzip, compress, x-compress
    ECID-Context: 1.0000IUhXOGw3R9BpNWK6ye1BPi6A0002Wk;kYhgp8TLo0hgv0
    Content-type: text/xml; charset=UTF-8
    Content-Length: 1473
     
    <env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:wsa="http://www.w3.org/2005/08/addressing">
    <env:Header>
     
    <wsa:To>http://<host>:<port>/currencyratesapi/services/myAction</wsa:To>
    <wsa:Action>myAction</wsa:Action>
    <wsa:MessageID>...</wsa:MessageID>
    <wsa:RelatesTo>...</wsa:RelatesTo>
    <wsa:ReplyTo>
    <wsa:Address>http://www.w3.org/2005/08/addressing/anonymous</wsa:Address>
    <wsa:ReferenceParameters>
    ......
    </wsa:ReferenceParameters>
    </wsa:ReplyTo>
    </env:Header>
    <env:Body>
    ......
    </env:Body>
    </env:Envelope> 
    

9.9.13 Deploying Human Workflow Application Throws "Unable to resolve 'TaskQueryService'" Error

Problem

Almost all Oracle Fusion Applications have the hwtaskflow.xml file with the IntegrateTaskFlowWithTask entry in the web.xml file to register the notification projects. When deploying the J2EE EAR file, it throws the following error if you did not configure the foreign JNDI connection or wf_client_config.xml with a valid SOA environment.

javax.naming.NameNotFoundException:Unable to resolve 'TaskQueryService'. Resolved
''; remaining name 'TaskQueryService'

Solution

If the SOA server is not up and running, then start the SOA server using Fusion Applications Control. If it is up and the problem exists, then check that the wf_client_config.xml file has a valid SOA environment. If possible, avoid foreign JNDI references because the JNDI resources get bound into all servers in the domain that causes BindException errors for servers in which the actual service exists when source and target JNDI names are the same. Oracle Fusion Applications do not use foreign JNDI references, instead bundling a correctly configured wf_client_config.xml file in the EAR file or using APIs at runtime to generate the Java Architecture for XML Binding (JAXB) for the human workflow APIs.

Use wf_client_config.xml (bundled at the application level) for all ADF worklist client applications communicating with Oracle SOA Suite. Ensure that you specify the correct URL in the file configuration.

<?xml version="1.0" encoding="UTF-8" ?>
 
<workflowServicesClientConfiguration
 xmlns="http://xmlns.oracle.com/bpel/services/client">
 <server name="default" default="true" clientType="REMOTE">
 <localClient>
  <participateInClientTransaction>false</participateInClientTransaction>
  </localClient>
  <remoteClient>
  <serverURL>t3:cluster://CRMSoaCluster</serverURL>
<initialContextFactory>weblogic.jndi.WLInitialContextFactory</initialContextFa
ctory>
  <participateInClientTransaction>false</participateInClientTransaction>
  </remoteClient>
 <soapClient>
<rootEndPointURL>http://adc60048fems.us.oracle.com:6361/soa-infra</rootEndPointURL
>
  </soapClient>
  </server>
</workflowServicesClientConfiguration>

9.9.14 "Invalid Subject" Error Thrown with Human Workflow API or Notification

Problem

When Oracle SOA Suite is installed on Domain1 and ADF on Domain2 and if any remote calls are performed between them for any human workflow API use or for notifications, errors similar to the following are thrown if the domain trust is not set.

javax.servlet.ServletException:
javax.servlet.ServletException:java.lang.SecurityException: Security:090398Invalid
Subject:principals=CVBUYER01, AR_MANAGER_VISION_OPERATIONS_DATA, AR_ACCOUNTS_
RECEIVABLE_MANAGER_JOB, FBI_TRANSACTIONAL_BI_WORKER

Solution

Set the domain trust password on both domains (global trust).

  1. Log in to Oracle WebLogic Server Administration Console.

  2. In the Domain Structure, click the domain name.

  3. Click the Security tab, and then click Advanced.

  4. Set the password in the Credential and Confirm Credential fields.

Note:

Oracle Fusion Applications do not support RMI across domains. That is, all remote calls between ADF and SOA are always within the same domain. When cross domain communication is required, SOAP is used.You may encounter the scenario described in this section in your development environment. For example, you may be running the ADF task flow for human tasks in the JDeveloper-integrated Oracle WebLogic Server and running the composite in an independent standalone SOA server. Both are in different domains.

9.9.15 Task is Assigned to the Group/Role When It Is Expected to Go to Every User in the Group/Role Individually

For troubleshooting information about this issue, see the "Task Assignment/Routing/Escalation Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.9.16 Task Completes Without Any Assignment Occurring

For troubleshooting information about this issue, see the "Task Assignment/Routing/Escalation Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.9.17 Parallel Assignees Must Approve or Reject the Task Even Though the Parallel Completion Criteria is Met

For troubleshooting information about this issue, see the "Task Assignment/Routing/Escalation Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.9.18 All Added Adhoc Participants Disappear After a Page Refresh

For troubleshooting information about this issue, see the "Task History Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.9.19 Future Approvers Are Not Visible in the History Table

For troubleshooting information about this issue, see the "Task History Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.9.20 Message Appears in the History Table About a Correlation ID Not Being Passed or Any Exception Related to the Correlation ID

For troubleshooting information about this issue, see the "Task History Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.9.21 Edit Toolbar Is Disabled or Not Shown

For troubleshooting information about this issue, see the "Task History Issues" section in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

9.9.22 oracle.jrf.UnknownPlatformException Error When Customizing an Oracle Mediator

Problem

You can receive the following error when making customization modifications to an Oracle Mediator service component in a SOA application.

For example, assume you perform the following steps:

  1. Log in to JDeveloper with the Fusion Application Developer role.

  2. Create a SOA application and project with the Customizable option selected.

  3. Add appropriate JAR files under the Libraries and Classpath section of the Project Properties dialog for the project.

  4. Under Application Resources > Descriptors > ADF META-INF > adf-config.xml, add appropriate customization classes.

  5. Add an Oracle Mediator service component to the application.

  6. Save the application and restart JDeveloper.

  7. Add appropriate routing rule and XSL transformations to the Oracle Mediator service component.

    The following error is displayed in the command prompt window:

    oracle.jrf.UnknownPlatformException: JRF is unable to determine the current
    application server platform.
    

Solution

This error can be ignored. After this error occurs, everything still works correctly.

9.9.23 java.lang.NullPointerException When Customizing an Oracle Mediator

Problem

You can receive the following error when making a customization modification to an Oracle Mediator service component in a SOA application.

For example, assume you perform the following steps:

  1. Log in to JDeveloper with the Fusion Application Developer role.

  2. Create a SOA application and project with the Customizable option selected.

  3. Add appropriate JAR files under the Libraries and Classpath section of the Project Properties dialog for the project.

  4. Under Application Resources > Descriptors > ADF META-INF > adf-config.xml, add customization classes.

  5. Add an Oracle Mediator service component to the application.

  6. Add a routing rule to the Oracle Mediator service component.

    The following error is displayed in the command prompt window:

    java.lang.NullPointerException
    

Solution

If you do not see any alerts indicating that a WSDL file cannot be found or parsed, then this exception in the log/console can be ignored.

9.9.24 JDeveloper Compilation Error in a SOA Project with SOA MDS Service Location

Problem

  1. Start JDeveloper with the Fusion Application Developer role.

  2. Create a SOA application.

  3. Create and deploy a SOA bundle to the server.

  4. Create another SOA application.

  5. Create an Oracle Mediator service component.

  6. Create a reference web service and access the SOA MDS connection and browser for the service of the SOA application that you created in Step 2.

  7. Wire the components.

  8. Update the empty port and location values in the composite.xml file. For the location value, enter the oramds path. For example:

    <reference name="Service1"
                
    ui:wsdlLocation="oramds:/apps/CalleeProject/CalleeBPELProcess.wsdl">
        <interface.wsdl
    interface="http://xmlns.oracle.com/Comp2Comp_
    EDGApp/CalleeProject/CalleeBPELProcess#wsdl.interface(CalleeBPELProcess)"/>
        <binding.ws
    port="http://xmlns.oracle.com/Comp2Comp_
    EDGApp/CalleeProject/CalleeBPELProcess#wsdl.endpoint(calleebpelprocess_client_
    ep/CalleeBPELProcess_pt)"
        location="oramds:/apps/CalleeProject/CalleeBPELProcess.WSDL"/>
      </reference>
    
  9. Compile the application.

    You receive the following error message.

    Warning(26,76): Failed to Find Binding
    

Solution

This is only a warning message thrown by the compiler. Runtime works fine. Having a reference from SOA MDS in a composite makes the port and location values empty in the bindings. To resolve this warning message:

  • Explicitly enter the corresponding details. For example, replace the binding.ws location value with the concrete WSDL file.

  • Deploy the composite using the configuration plan, which replaces the port and oramds location with the concrete WSDL file.