13 Troubleshoot Oracle SOA Suite

Problem

When a business event is published, the business event subscription defined in the composite does not fire and a composite instance is not created. This can occur, for example, with an ADF application invoking a SOA composite.

Solution

To resolve this problem.

1. Enable EDN logging in Fusion Applications Control. The following server loggers specific to EDN are available for selection:

   • oracle.integration.platform.blocks.event

   • oracle.integration.platform.blocks.event.saq

   • oracle.integration.platform.blocks.event.jms

     which only appears if EDN is running in EDN-JMS mode instead of the default EDN-DB mode.

   The server loggers can be set to one of the following levels:

   • TRACE:1 (FINE)

   • TRACE:16 (FINER)

   • TRACE:32 (FINEST)

   Alternately selective tracing can be used to get detailed, on-disk trace selectively (for example, by user name, thereby eliminating trace output for other users). See Get Started with Troubleshooting and Logging Basics for Oracle SOA Suite for details.

   Detailed logging goes into SOA server's diagnostic.log file configured in Fusion Applications Control. To set the log level for the loggers:

   1. Go to the navigation pane.

   2. Right-click soa-infra.

   3. Choose Logs, and then Log Configuration.

   4. Expand oracle.integration, oracle.integration.platform, oracle.integration.platform.blocks, and then oracle.integration.platform.blocks.event.

   5. Set the loggers described in step 1 to an appropriate logging level.

   The following sample shows a portion of a server log file:

   [2011-03-22T11:52:37.038-07:00] [soa_server1] [TRACE] [SOA-31010]
   [oracle.integration.platform.blocks.event.saq] [tid:
   weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
   <anonymous>] [ecid:
   5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
   [SRC_CLASS:
   oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
   [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
   PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B] [SRC_
   METHOD: fineDequeuedEvent] [composite_instance_id: 90087] Dequeued event,
   Subject: null [source type "J"]:[[
       <business-event
   xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
   xmlns="http://oracle.com/fabric/businessEvent">
         <name>ns:MyEvent</name>
         <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
   
   <source>default/MediatorPubSub!2.0*soa_
   58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
   58454bff-3b36-4e29-a883-949085d85719/PublishEvent</source>
       <tracking>
         <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
         <compositeInstanceId>90087</compositeInstanceId>
   <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B</parentCom
   ponentInstanceId>
       </tracking>
         <content>
            <inp1:singleString
   xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
         </content>
   </business-event>
        
      ]]
      [2011-03-22T11:52:37.042-07:00] [soa_server1] [TRACE] [SOA-31011]
   [oracle.integration.platform.blocks.event.saq] [tid:
   weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
   <anonymous>] [ecid:
   5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
   [SRC_CLASS:
   oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
   [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
   PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B] [SRC_
   METHOD: fineFilterResults] [composite_instance_id: 90087] Filter [XPath Filter:
   /be:business-event/be:content/ns1:singleString = 'PO123'] for subscriber
   "default/MediatorPubSub!2.0*soa_
   58454bff-3b36-4e29-a883-949085d85719/SubsEvent" returned true
   . . .
   . . .
   

2. Check whether bad composites exist in the SOA domain that slow down the overall EDN event delivery.

   EDN internally uses two advanced queuing (AQ) queues in a serial processing fashion for all business events. One or more bad SOA composite applications that fail to consume certain types of events can slow down EDN delivery of all other event types and cause an event backlog in the AQ queues.

   1. Check if there are repeated errors from bad SOA composite applications in the SOA server log with EDN logging enabled. Even if the errors are in completely separate SOA composite applications that are unrelated to the event types of interest, it can delay the overall delivery of events of interest to the target SOA composite applications.

   2. Check the event backlog count as detailed in Step 2.d (events are stuck in advanced queue). Check backlog event counts and subscriber details by navigating to the following URL: http://host_name:port/soa-infra/events/edn-db-backlog.

      If bad composites are found in Step 2.a and large numbers of backlogged events occur in Step 2.b, then proceed to 2.c.

   3. Stop bad composites identified in Step 2.a. This can be achieved from Oracle Enterprise Manager Fusion Middleware Control or the WLST command line utility.

   4. Temporarily increase the number of EDN threads to a larger number, (for example, from 3 to 5 or 7) to clear up the backlogged events faster.

      To change the number of EDN threads through the configuration of the EDNConfig MBean in Oracle Enterprise Manager Fusion Middleware Control:

      1. Go to the navigation pane.

      2. Right-click soa-infra.

      3. Select SOA Infrastructure then Administration and then, System MBean Browser.

      4. Expand Application Defined MBeans, choose oracle.as.soainfra.config, and then Server: soa_server-x , EDNConfig, and then edn.

      5. Change the value for the NumberOfThreads attribute, and click Apply.

         If backlog event processing is not important, the backed up events can be purged directly in AQ. The purge script is as follows:

         DECLARE
           purge_options dbms_aqadm.aq$_purge_options_t;
         BEGIN
           purge_options.block := FALSE;
           DBMS_AQADM.PURGE_QUEUE_TABLE(
             queue_table     => '&edn_user..edn_event_queue_table',
             purge_condition => NULL,
             purge_options   => purge_options);
           DBMS_AQADM.PURGE_QUEUE_TABLE(
             queue_table     => '&edn_user..edn_oaoo_delivery_table',
             purge_condition => NULL,
             purge_options   => purge_options);
         END;
         /
         commit;
         

   5. Periodically monitor the EDN backlog count (see Step 2.b) to confirm that it is being reduced. If the clear-up rate is very slow, the EDN thread number can be further increased (see Step 2.d).

   6. Continue to monitor the EDN backlog count over a period of time. Also monitor the recent instances and faults in Oracle Enterprise Manager Fusion Middleware Control to see if composite instances are being created or new faults are showing up both across the board (that is, at SOA Infrastructure level) and for the composite of interest.

   7. Once the backlog is cleared up, revert the number of EDN threads to the previous setting or an appropriate number.

   8. Publish a business event of interest and verify that the target composite is triggered with expected instance creations.

3. Check whether the event publication is being delivered to the SOA cluster.

   One way to do this is to use the log messages in Fusion Applications Control. The log messages include information about the incoming events and the event subject and payload. Before the event information can be viewed in the log messages, first the oracle.integration.platform.blocks.event logger must be set to TRACE:32 (FINEST), as described in step 1, and then raise the event again. To search for the event-related messages:

   1. Right-click soa-infra, choose Logs, and then View Log Messages.

   2. Specify the date range.

   3. In the Message Types section, select the Notification and Trace checkboxes.

   4. To search for all events delivered to the SOA cluster, specify Message contains Dequeued event as the search criteria, and click Search.

      The subject associated with the event is displayed in the Message field and the business event namespace and local name, payload, ECID, and message details are displayed in the Supplemental Detail field.

   5. To search for a particular event by event name, click Add Fields.

   6. Select Supplemental Detail, and click Add.

   7. Specify that Supplemental Detail contains either the event name space or event local name as the search criteria.

   8. Click Search.

   9. Use additional information such as the time interval or data in the payload to identify the specific event message of interest.

      The log message is generally a good way to identify whether the issue is with the publisher or subscriber. For example, if the subject is missing or the fmw-context is missing, it typically means there is an issue with the publisher. If the information reported in the log message is correct, then it typically means there is a problem with the subscriber.

      If the Dequeued event log message is seen for the business event, then go to step 10.

      If the Dequeued event log message is not seen, there may be issues with the following:

      For This Issue...	See...
      EDN is in paused mode and temporarily stopped delivering events to its subscribers.	Step 4
      Events are stuck in the advanced queue.	Step 5
      More than one SOA cluster is pointing to the same advanced queue.	Step 6
      There are internal SOA issues in the database layer.	Step 7
      The data source is configured to point to the wrong SOAINFRA schema.	Step 8
      There are issues with the Java, SOA, or PL/SQL code raising the event.	Step 9

4. When patches are applied to the SOA cluster, EDN is automatically placed in paused mode to prevent delivery of events during patching. After patching is complete, EDN exits paused mode and resumes event processing. When a patch fails, EDN may remain in paused mode to prevent event subscriptions from firing after the failure. Follow these steps to identify if EDN is in paused mode, and restart it:

   1. From the navigation pane, expand SOA and right-click soa-infra.

   2. Choose Administration then, System MBean Browser.

   3. Go to Application Defined MBeans, choose oracle.as.soainfra.config, and then Server: SOA_cluster_name, EDNConfig, and then edn.

   4. In the Attributes column, locate the Paused property.

   5. If Paused is set to true, the listener threads are decreased to zero, which temporarily stops event delivery.

   6. To restart event delivery, select false.

   7. Click Apply.

5. At times, the message may be stuck in the advanced queue in different stages of delivery. Check the count of (potentially stuck) events currently in the following queue tables:

   • EDN_EVENT_QUEUE_TABLE: Every event published is temporarily enqueued into this table (for edn_event_queue).

   • EDN_OAOO_DELIVERY_TABLE: Only events with a one-and-only-one (OAOO) delivery target(s) are temporarily enqueued into this table (for edn_oaoo_queue).

   The count can also be checked in the edn-db-log. The total number of messages in the EDN_EVENT_QUEUE_TABLE and EDN_OAOO_DELIVERY_TABLE are displayed at the top of the page.

   1. Navigate to the following URL to see the total number of messages at the top of the page:

      http://host_name:port/soa-infra/events/edn-db-log
      

      There is no need to enable the edn-db-log to view these messages.

6. There may be more than one SOA cluster pointing to the same SOAINFRA schema. This is highly unlikely in a provisioned environment, but for environments that are created by developers, it may be an issue. In this case, the expected SOA runtime environment may not be receiving the events properly. It is expected that only one SOA runtime environment listens for business events on a SOAINFRA schema. Use the following query to identify the list of SOA runtime environments subscribing to business events from a SOAINFRA schema. Have only the required SOA runtime environment up, which points to the SOAINFRA schema, and shut down the others.

   1. Navigate to the edn-db-log to see log messages that display each SOA cluster that is connecting to the EDN advanced queue:

      http://host_name:port/soa-infra/events/edn-db-log
      

      The edn-db-log do not need to be enabled to view these messages. The messages continue to be displayed in the edn-db-log until the next time the Clear link is clicked on it. Here is a sample message seen in the edn-db-log:

      Starting EDN bus. Timestamp=Wed Feb 23 10:11:26 PST 2011.
          Parameters: platform="weblogic", cluster="", server="soa_server1",
          domain="fusion_domain", admin server="AdminServer", host="adc2180440".
      

   2. Alternatively, the following select statement can be run against the SOAINFRA schema. For example, use FIN_FUSION_SOAINFRA for the SOA cluster in the Financials domain.

      select * from V$SESSION where username like 'family_FUSION_SOAINFRA';
      

7. Verify that the event is properly being enqueued and dequeued by the underlying EDN PL/SQL procedures. This can be done by using the edn-db-log that displays all events, including their name space, names, payload, and subject, which are enqueued and dequeued by the PL/SQL EDN procedures.

   Before proceeding, consider the following:

   • The administrator privilege must be active in order to enable/disable the edn-db-log.

   • The edn-db-log must always be disabled after debugging to disable logging. This prevents excessive database growth in the EDN database log table. If the edn-db-log remains enabled, then debugging messages related to events that are published/enqueued into the database and subscribed to/dequeued from the database continue to be persisted into certain EDN database log tables. This causes the table to grow indefinitely.

   1. Navigate to the following URL:

      http://host_name:port/soa-infra/events/edn-db-log
      

   2. Click the Enable link.

   3. Raise the business event again.

   4. Click the Refresh link.

8. The EDNSource and EDNLocalTxSource data sources may be pointing to a different database than the EDNDataSource and EDNLocalTxDataSource data source connections of the SOA server. Ensure that these four data source connections on ADF and the SOA server are pointing to the same database schema.

   Check or set the data sources in the Oracle WebLogic Server Administration Console:

   1. In the Domain Structure, select Services, and then Data Sources.

   2. In the Name column of the table, select data_source_name.

      The connection pool and driver name are only available for generic data sources, and not for multidata sources.

   3. Select the Connection Pool tab.

   4. In the Driver Class Name field, set the data source.

9. Check the log files in Fusion Applications Control for the code that raises the event to see if there were failures in raising the event or failures in the code that prevented the event from being raised.

   For ADF, check the log files. Look for the ECID corresponding to the business key (for example, purchase order).

   To enable event-related log messages in the server running the ADF application that raises the event, set the log level for the oracle.integration.platform.blocks.event logger to TRACE:32 on that server.

10. By the time this step is reached, it has been confirmed that the business event is properly delivered to the SOA cluster. To identify whether any composite instances were created, perform the following steps:

    1. Because the ECID is different for a subscription with a guaranteed consistency level and a subscription with an OAOO level defined for the same event, the consistency level for the event subscription must be first identified. To do this, right click soa-infra, choose Business Events, and click the Subscriptions tab. Find the composite that subscribes to the business event in the Components Subscriptions table and get the value from the Consistency Level column.

    2. If the value is set to Guaranteed, then get the ECID from the Dequeued event log message.

    3. If the value is set to One And Only One, then search for the log message with Message contains Dequeued OAOO event as criteria and use the ECID from that message.

    4. Locate the composite instance by right-clicking soa-infra, selecting the Instances tab, entering the ECID in the search criteria, and clicking Search.

    If composite instances are seen, then continue to step 13.

    If composite instances are not seen, this may be due to the following issues:

    • There are issues with the subscription logic in the composite, possibly a mismatch in the event namespace or event name or error in the filter logic. See step 11.

    • There may be a SOA issue. See Step 12.

11. If a filter is being used, there may be an issue with the filter logic. Use Fusion Applications Control to review the event subscriptions and filters defined for the composite and the payload content.

    1. Log in to Fusion Applications Control in the domain where the composite with the subscription is deployed.

    2. Navigate to domain_name then,SOA.

    3. Right-click soa-infra (SOA_cluster_name), and choose Business Events.

    4. Click the Subscriptions tab.

    5. Review the information in the Event Name, Namespace, and XPath Filter columns of the Component Subscriptions table.

    Alternatively, the SOA server diagnostic.log file can be also checked for filter-related logging. The log level must be set to at least TRACE:1 (FINE) for the oracle.integration.platform.blocks.event.saq logger. The following is an example log message that has the following statement pattern: Filter [XPath Filter: ….] for subscriber "…" returned true/false.

    [2011-03-22T11:52:36.976-07:00] [soa_server1] [TRACE] [SOA-31011]
    [oracle.integration.platform.blocks.event.saq] [tid:
    [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
    (self-tuning)'] [userId: <anonymous>] [ecid:
    5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1] [SRC_
    CLASS:
    oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
    [WEBSERVICE_PORT.name: execute_pt] [APP: soa-infra] [composite_name:
    MediatorPubSub] [component_name: PublishEvent] [component_instance_id:
    8E0411C054B511E0AF455DABE1395E7B] [J2EE_MODULE.name: fabric] [SRC_METHOD:
    fineFilterResults] [WEBSERVICE.name: PublishEvent_ep] [J2EE_APP.name:
    soa-infra] [composite_instance_id: 90087] Filter [XPath Filter:
    /be:business-event/be:content/ns1:singleString = 'PO123'] for subscriber
    "default/MediatorPubSub!2.0*soa_
    58454bff-3b36-4e29-a883-949085d85719/SubsEvent" returned true
    

12. Check the SOA diagnostic log for stack traces related to the issue. For example, the SOA instance may have been rolled back due to an error.

13. By the time this step is reached, it has been confirmed that the business event has been delivered to the SOA cluster and a composite instance has been created. However, issues with the composite components may be encountered:

    • If a composite instance is seen, but not a BPEL instance, check if the BPEL process can be executed independently by a client, as opposed to the event subscription.

    • If a composite instance is seen, but not a BPEL instance, this symptom may indicate a stuck thread on a service call. In this scenario, the BPEL process service engine is hung on a service call and has not created the audit trail. To verify the behavior, take three thread dumps approximately 30 seconds apart on all servers in the cluster. See Table 13-1 for more information on taking thread dumps. If the same thread shows it is stuck after the three thread dumps (that is, after 1.5 minutes), the thread is stuck. In this case, wait for the call to time out, which then enables the recovery of the instance.

    • If the audit trail shows the following message:

      Invoked 1-way operation "initiate" on target service service_name
      

      There may be an issue with an incorrect driver type for the data sources. Note that the driver XA configuration should already be created by the SOA/provisioning template. If the driver type is changed or is not set properly, routing from Oracle Mediator does not occur. Instead, it tries to route and fails several times. This can be confirmed by instances in Fusion Applications Control. As per Oracle Fusion Applications standards, the SOADataSource and EDNDataSource data sources should use the XA driver type; that is:

      • EDNDataSource: oracle.jdbc.xa.client.OracleXADataSource

      • SOADataSource: oracle.jdbc.xa.client.OracleXADataSource

The following documentation provides additional information about the topics discussed in this section:

• For more information about setting logging levels for Oracle SOA Suite, see section Setting Logging Levels for Troubleshooting in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

• For more information about data sources, see section Configuring the SOA Infrastructure in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

Problem

After a business event is published, there is a long delay (minutes, hours, or even days) before the event subscriber is triggered and its composite instance is created.

This can occur, for example, with an ADF application invoking a SOA composite.

Solution

To resolve this problem, perform the following steps.

1. Enable EDN logging in Fusion Applications Control. The following server loggers specific to EDN are available for selection:

   • oracle.integration.platform.blocks.event

   • oracle.integration.platform.blocks.event.saq

   • oracle.integration.platform.blocks.event.jms

   Note that oracle.integration.platform.blocks.event.jms only appears if EDN is running in EDN-JMS mode, instead of the default EDN-DB mode.

2. Set the server loggers to one of the following levels:

   • TRACE:1 (FINE)

   • TRACE:16 (FINER)

   • TRACE:32 (FINEST)

   Selective tracing can be alternately used to selectively get a detailed, on-disk trace (for example, by user name, thereby eliminating trace output for other users). See Get Started with Troubleshooting and Logging Basics for Oracle SOA Suite for details.

   Detailed logging goes into the server's diagnostic.log file configured in Fusion Applications Control.

3. Set the log level for the loggers.

   1. Go to the navigation pane.

   2. Right-click soa-infra.

   3. Select Logs, and then Log Configuration.

   4. Expand oracle.integration, choose oracle.integration.platform and oracle.integration.platform.blocks, and then oracle.integration.platform.blocks.event.

   5. Set the loggers described in step 1 to an appropriate logging level.

      The following sample shows a portion of a server log file:

      [2011-03-22T11:52:37.038-07:00] [soa_server1] [TRACE] [SOA-31010]
      [oracle.integration.platform.blocks.event.saq] [tid:
      weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
      <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
      [SRC_CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
      PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B]
       [SRC_
      METHOD: fineDequeuedEvent] [composite_instance_id: 90087] Dequeued event,
      Subject: null [source type "J"]:[[
          <business-event
      xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
      xmlns="http://oracle.com/fabric/businessEvent">
            <name>ns:MyEvent</name>
            <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
      <source>default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/PublishEvent</source>
          <tracking>
      <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
            <compositeInstanceId>90087</compositeInstanceId>
      <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B</paren
      tCom
      ponentInstanceId>
          </tracking>
            <content>
               <inp1:singleString
      xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
            </content>
      </business-event>
         ]]
         [2011-03-22T11:52:37.042-07:00] [soa_server1] [TRACE] [SOA-31011]
      [oracle.integration.platform.blocks.event.saq] [tid:
      weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
      <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
      [SRC_CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
      PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B]
       [SRC_
      METHOD: fineFilterResults] [composite_instance_id: 90087] Filter [XPath
       Filter:
      /be:business-event/be:content/ns1:singleString = 'PO123'] for subscriber
      "default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/SubsEvent" returned true
      . . .
      . . .
      

4. Find out when the event is published, dequeued, and sent to the subscriber by EDN.

   Use the Java logging in step 1 to identify the timestamps of event publishing, dequeuing, and sending to the subscriber. From these timestamps, it can be identified, if the delay is on the publishing side or the EDN delivery side.

   To search for the event-related messages in the log messages.

   1. Right-click soa-infra.

   2. Select Logs then, View Log Messages.

   3. Specify the date range.

   4. To search for all events delivered to the SOA cluster, specify Message contains Dequeued event as the search criteria, and click Search.

      The subject associated with the event is displayed in the Message field and the business event namespace and local name, payload, ECID, and message details are displayed in the Supplemental Detail field.

   5. To search for a particular event by event name, click Add Fields.

   6. Select Supplemental Detail, and click Add.

   7. Specify that Supplemental Detail contains either the event namespace or the event local name as the search criteria.

   8. Click Search.

   9. Use additional information such as the time interval or data in the payload to identify the specific event message of interest.

      The following sample shows a portion of a server log file containing information about event publishing with a timestamp of 2011-03-22T11:52:36.968-07:00.

      [2011-03-22T11:52:36.968-07:00] [soa_server1] [TRACE] [SOA-31000]
      [oracle.integration.platform.blocks.event.saq] [tid:
      [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
      (self-tuning)'] [userId: <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1] [SRC_
      CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [WEBSERVICE_PORT.name: execute_pt] [APP: soa-infra] [composite_name:
      MediatorPubSub] [component_name: PublishEvent] [component_instance_id:
      8E0411C054B511E0AF455DABE1395E7B] [J2EE_MODULE.name: fabric] [SRC_METHOD:
      fineEventPublished] [WEBSERVICE.name: PublishEvent_ep] [J2EE_APP.name:
      soa-infra] [composite_instance_id: 90087] Received event: Subject: null 
      Sender: default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/PublishEvent
      Event:[[
      <business-event xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
      xmlns="http://oracle.com/fabric/businessEvent">
        <name>ns:MyEvent</name>
        <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
        <tracking>
       <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
           <compositeInstanceId>90087</compositeInstanceId>
      
      <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B</paren
      tComponentInstanceId>
        </tracking>
        <content>
           <inp1:singleString
      xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
        </content>
      </business-event>
      
      ]]
      

      The following sample shows a portion of a server log file containing information about an event dequeued by EDN with a timestamp of 2011-03-22T11:52:37.038-07:00.

      [2011-03-22T11:52:37.038-07:00] [soa_server1] [TRACE] [SOA-31010]
      [oracle.integration.platform.blocks.event.saq] [tid:
      weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@96bab0] [userId:
      <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000140]
      [SRC_CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [APP: soa-infra] [composite_name: MediatorPubSub] [component_name:
      PublishEvent] [component_instance_id: 8E0411C054B511E0AF455DABE1395E7B]
      [SRC_METHOD: fineDequeuedEvent] [composite_instance_id: 90087] Dequeued
      event, Subject: null [source type "J"]:[[
      <business-event xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
      xmlns="http://oracle.com/fabric/businessEvent">
        <name>ns:MyEvent</name>
        <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
      
      <source>default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/PublishEvent</source>
       <tracking>
        <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
            <compositeInstanceId>90087</compositeInstanceId>
        <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B
        </parentComponentInstanceId>
       </tracking>
         <content>
           <inp1:singleString
      xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
         </content>
      </business-event>
      
      ]]
      

      The following sample shows a portion of a server log file containing information of an event sent to a subscriber by EDN with a timestamp of 2011-03-22T11:52:37.150-07:00.

      [2011-03-22T11:52:37.150-07:00] [soa_server1] [TRACE] [SOA-31033]
      [oracle.integration.platform.blocks.event.saq] [tid:
      weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@b402db] [userId:
      <anonymous>] [ecid:
      5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83,0:1:100000141]
      [SRC_CLASS:
      oracle.integration.platform.blocks.event.saq.SAQBusinessEventBusMessages]
      [APP: soa-infra] [SRC_METHOD: fineSentOAOOEvent] Sent OAOO event [QName:
      "{http://schemas.oracle.com/events/edl/MyEventDefn}MyEvent" to target:
      "default/MediatorPubSub!2.0*soa_
      58454bff-3b36-4e29-a883-949085d85719/SubsEvent"]:[[
      <business-event xmlns:ns="http://schemas.oracle.com/events/edl/MyEventDefn"
      xmlns="http://oracle.com/fabric/businessEvent">
        <name>ns:MyEvent</name>
        <id>a7ae9d28-9530-4049-8385-e0ebfb0eea50</id>
      <source>default/MediatorPubSub!2.0*soa_
       58454bff-3b36-4e29-a883-949085d85719/default/MediatorPubSub!2.0*soa_
       58454bff-3b36-4e29-a883-949085d85719/PublishEvent</source>
      <tracking>
      <ecid>5fc0ca821d51e919:3d214296:12ebaa2c996:-8000-00000000000c2b83</ecid>
           <compositeInstanceId>90087</compositeInstanceId>
      
      <parentComponentInstanceId>mediator:8E0411C054B511E0AF455DABE1395E7B</paren
      tComponentInstanceId>
      </tracking>
        <content>
           <inp1:singleString
      xmlns:inp1="http://xmlns.oracle.com/singleString">PO123</inp1:singleString>
        </content>
      </business-event>
      ]]
      

5. If the delay comes from the event publishing side, then check the following:

   1. If the remote EDN API is used to publish events from, for example, an ADF component to the SOA server, and the following event connection factory lookup code is used:

      BusinessEventConnectionFactory BusinessEventConnectionFactorySupport.
      findRelevantBusinessEventConnectionFactory(boolean forceRetry)
      

      The forceRetry parameter may need to be set to true to cache the BusinessEventConnectionFactory object after it is found, and avoid forced retries of JNDI lookups that can cause a significant delay.

   2. If multiple events are published with a loop statement in a batch fashion, the above event connection factory lookup may need to be moved outside the loop to avoid a lookup per event.

6. If the delay comes from event dequeuing after the event is published, then check the following:

   Ensure That...	Description
   The SOA server is up running, and EDN has been configured with the correct amount of dequeuer threads through an EDNConfig MBean.	To examine the configuration of the EDNConfig MBean in Fusion Applications Control: Go to the navigation pane. Right-click soa-infra. Select SOA Infrastructure then, Administration and then, System MBean Browser. Expand Application Defined MBeans then, oracle.as.soainfra.config then, Server: soa_server-x then, EDNConfig and then, edn. Ensure that the NumberOfThreads attribute (for EDN dequeuer threads) is set to an appropriate number (the default is 3). Set the number of EDN dequeuer threads (the default is 3) to a value that matches the number of events published. If there are a large number of events published which cause event backlog, this can be verified by navigating to the following URL: http://host_name:port/soa-infra/events/edn-db-backlog. Note that the EDN dequeuer threads come from a thread pool that is provided by the SOA server and shared with other SOA processes. Therefore, when the number of EDN dequeuer threads is increased, it may potentially affect server performance for other SOA functionality.
   The backing store of EDN is functioning properly.	For example, if the backing store of EDN is Oracle AQ, and there are a significant amount of dead events stuck in the AQ that are not cleared, then event dequeuing by EDN may slow down significantly.

7. If the delay comes from event consumption by its subscriber after the event is dequeued, then check the following:

   1. Ensure that all the event subscribing components such as Oracle Mediator or a BPEL process can consume and process the event within the expected time frame. If not, investigate why there is a delay.

   2. If the event delivery is configured with an OAOO delivery guarantee, then EDN delivers the event to its subscriber in a global transaction context with built-in retry logics, should the transaction fail to commit. Retries may delay overall event processing. If the transaction fails to commit, ensure that the root cause of transaction failure is researched.

For more information about setting logging levels for Oracle SOA Suite, see section Setting Logging Levels for Troubleshooting in Administering Oracle SOA Suite and Oracle Business Process Management Suite.

Problem

Events are consumed by multiple revisions of the same composites.

Solution

By design, different composites can subscribe to the same event. However, it is an error if multiple revisions of the same composite subscribe to the same event. This occurs when multiple versions of the composite are active possibly as a result of a patch failure.

Perform the following steps to identify whether there are multiple, active revisions of the same composite and to retire the composite revision that should not be active:

Problem

If the Java debug level is set to TRACE:16 (FINER) or a lower value, log messages such as the following may be seen :

Began XA for OAOO
Rolled back XA for OAOO

Solution

These are normal messages of OAOO event delivery when there are no events waiting to be delivered. These are not error conditions. These messages can be turned off by setting the Java logging level to TRACE:1 (FINE) or a higher value.

Problem

An application transaction is not completing. For example, a purchase order status may remain processing. Checking the composite instance shows that the composite is stuck in the running state. In this case, the component is probably not running. Instead, it has likely faulted and may need recovery.

Faults may occur for various reasons:

• A BPEL activity faulted with an error (for example, a business error, security authorization error, or some other error).

• A BPEL activity invoked an external web service that was unavailable.

• A BPEL activity has already been terminated by the administrator using Fusion Applications Control.

• A BPEL activity invoked an asynchronous ADF service and the message is stuck in the AQ/JMS queue.

• A BPEL activity invoked an asynchronous ADF service, but because SOA was unavailable, the callback message did not arrive.

• A BPEL activity invoked a synchronous ADF service, which is taking a long time (or is hanging).

• A network error occurred.

Solution

1. Log in to Fusion Applications Control.

2. In the navigator pane, go to domain_name and then, SOA.

3. Click soa-infra (SOA_cluster_name).

4. Click the Instances tab.

5. Search for the composite instance, and click the instance ID.

   The Flow Trace page appears.

   If the instance is not visible (and the Audit Level is not set to Off in the SOA Infrastructure Common Properties page), this implies that the message is stuck outside of SOA. Check the ADF log (if invoked through an ADF service) to find out if it is stuck in the AQ/JMS queue.

   If the message has reached Oracle Mediator, but not instantiated the BPEL flow, the BPEL instance may have been rolled back from the start due to an error.

   If the BPEL flow exists, the Faults section of the Flow Trace page typically shows the faulted service that can trace the root cause.

6. In the Trace section, click the BPEL process.

7. Expand the BPEL audit trail to see the exact point at which the service faulted.

   This information is also available in the Faults tab of the BPEL flow trace. It also indicates whether the fault can be recovered.

8. Click the View Raw XML link.

   The same information is also available through this link, where the error can be seen. For example:

   ...
   <message>Faulted while invoking operation "modifyUserRoles" on provider
    "UserService". </message><details>
   ...
   <tns:message>JBO-27023: Failed to validate all rows in a
    transaction.</tns:message>
   <tns:severity>SEVERITY_ERROR</tns:severity>
   ...
   <message>The transaction was rolled back. The work performed for bpel instance
    "451042" was rolled back to the previous dehydration point, but the audit
    trail has been saved.
   The instance can be recovered from the recovery console by resubmitting the
    callback message or activity for execution.</message
   

   Since the instance was rolled back to its previous dehydration point, the status remains as Running.

9. In the Audit Trail and Faults tabs, make a note of the following:

   • Composite name (for example, HcmUsersSpmlComposite)

   • Component (for example, UpdateGuid BPEL process)

   • BPEL instance ID (for example, bpel:451042)

   This is all used in the recovery of the instance, if it is recoverable. Note that the audit trail window may mark the error as a nonrecoverable business fault, but the recoverability of the message can be found in the Recovery tab of the BPEL process service engine.

10. To attempt to recover the instance, right-click soa-infra (SOA_cluster_name), and choose Service Engines then, BPEL.

11. Click the Recovery tab.

12. From the Type list, select Activity.

13. Specify the composite and component names captured in step 9, and click Search.

14. Find the specific BPEL instance ID. Faults marked as Recoverable can be recovered.

15. Check the other recovery options in the Type list (for example, Invoke and Callback), if they exist.

    It is also possible to search for recoverable messages from the Faults and Rejected Messages tab of the SOA Infrastructure by clicking the message and selecting the appropriate action from the Recovery Actions list.

16. If the instance is not marked as recoverable, then reinvoking the service is not allowed (most probably because it is not idempotent). In some cases, diagnostic information may need to be provided to Oracle Support Services to resolve issues with nonrecoverable, nonidempotent transactions.

17. If the BPEL activity has invoked an asynchronous ADF service and the message is stuck in the AQ JMS queue, the server.log and server-diagnostic.log files can be viewed to see the logging of the message metadata logged by the JRF web services infrastructure.

    In addition, the ADF diagnostic logs are also available to debug, if needed.

    1. Use the ECID field to correlate and track ADF service logging corresponding with the SOA composite that invoked it.

    2. When viewing the log in Fusion Applications Control, click the Broaden Target Scope dropdown list and select the farm_name/domain_name (Oracle WebLogic Domain) to view messages across the domain.

    3. In the Log Messages page for the Oracle WebLogic Server domain, in the Selected Targets section, ensure that the search includes the ECID field with the value noted in step 17.a and the Component Name field is set to adf-bc.

    4. Search and view log records for the Oracle Application Development Framework (Oracle ADF) Business Component (BC) and the ECID and note any issues.

       For a specific ECID, find several root instances (top level clients). Drill down to the appropriate instance to find a specific fault.

    5. Observe if the Oracle ADF Business Components completed successfully or completed with an error. See section Viewing and Searching Log Files in the Administering Oracle Fusion Middleware.

       For more details on diagnosing Oracle ADF-BC asynchronous web service calls, see BPEL Process Received No Response from an ADF Asynchronous Service.

See the Administering Oracle SOA Suite and Oracle Business Process Management Suite for the various ways to recover transactions and messages:

• Managing SOA Composite Application Instances section to recover from the SOA composite application page in Fusion Applications Control.

• Managing BPEL Process Service Components and Engines section to recover from the BPEL process service component and BPEL process service engine message recovery pages.

Problem

A BPEL activity has invoked an asynchronous ADF service and not received a response. The message may be stuck in the request or response queues.

Solution

1. Check if the reply address is valid, the server is running, and the name of the server and port match with the server on which the BPEL process is running.

2. Verify that the policy on the callback receive activity matches with the policy advertised in the WSDL for the response port type/binding.

3. Look at the server-name_diagnostic.log file and check for all status messages of the request.

   1. Go to Fusion Applications Control.

   2. Right-click soa-infra.

   3. Choose Logs > View Log Messages.

   4. Click Target Log Files.

   5. Select the server diagnostic file to view, and click View Log File.

      Every log contains the message ID and other details, such as ECID, that can help isolate it from other messages. After finding the message ID (that is, MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33), search for the same ID in the log file. Messages such as the following should be seen for each successful asynchronous operation.

      If the following log is not available, this means the asynchronous operation was never called on this server:

      [2010-12-17T12:27:13.537-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: <anonymous>]
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0:1] 
      [WEBSERVICE_PORT.name: AsyncEjbPort] [APP: AsyncEjb] [J2EE_MODULE.name:
       AsyncEjb-ejb] [WEBSERVICE.name: AsyncEjbService] [J2EE_APP.name: AsyncEjb]
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33] Sending message
       to JMS queue* 
      "oracle.j2ee.ws.server.async.NonDefaultRequestQueue" for async processing
       of service "1d7551aa-73d9-4624-ad52-c15e35b5b25dRequest"
      

      If the following log is not available, this means the asynchronous operation was never called on this server:

      [2010-12-17T12:27:13.783-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: <anonymous>] 
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0:1] 
      [WEBSERVICE_PORT.name: AsyncEjbPort] [APP: AsyncEjb] [J2EE_MODULE.name:
       AsyncEjb-ejb] [WEBSERVICE.name: AsyncEjbService] [J2EE_APP.name: AsyncEjb]
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33] An asynchronous
       request message is received*
      and successfully recorded for service "AsyncEjbService" with a replyTo
      address as
      "http://adc2180314:7001/AsyncEjbCallback/AsyncEjbResponseImplService"
       
      [2010-12-17T12:27:13.986-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: weblogic] 
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0] [APP: AsyncEjb]  
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33]
       Startedasynchronous request processing for the service* 
      "AsyncEjbService" with the message selector
       "1d7551aa-73d9-4624-ad52-c15e35b5b25dRequest". Transaction enabled: "true"
       
      [2010-12-17T12:27:14.296-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: weblogic] 
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0] [APP: AsyncEjb]  
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33] Completed
       asynchronous request processing. A response will be sent to the client.*
       
      [2010-12-17T12:27:14.312-08:00] [AdminServer] [NOTIFICATION] []
       [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
       (self-tuning)'] [userId: weblogic] 
      [ecid: 0000Inq0wfa76EWLHyo2yf1D2wUW000000,0] [APP: AsyncEjb]  
      *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33]
       Startedasynchronous response processing for the service* 
      "AsyncEjbService" with the message selector
      "1d7551aa-73d9-4624-ad52-c15e35b5b25dResponse".
       
      [2010-12-17T12:27:15.018-08:00] [AdminServer] [NOTIFICATION] []
      [oracle.j2ee.ws.common.jaxws.JAXWSMessages] 
      [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default
      (self-tuning)'] [userId: weblogic] 
      [ecid: 77e271e3-afd5-48e8-a83d-9b326860f3f2-0000000000000018,0] [APP:
      AsyncEjb] *[MessageID: urn:uuid:ac1a4a81-39df-45b1-a741-e16e752d5d33]
      Completed asynchronous response processing successfully. Client must have
      received the response by now.*
      

   To monitor the traffic to the asynchronous ADF web services, monitor the Oracle AQ queues that are used for each family. The queue names for each family are shown in the following table:

   Family AQ	Name
   Oracle Fusion Customer Relationship Management Product	CRM_AsyncWS_Request and CRM_AsyncWS_Response
   Oracle Fusion Human Capital Management Product	HCM_AsyncWS_Request and HCM_AsyncWS_Response
   Oracle Fusion Financials Product	FIN_AsyncWS_Request and Fin_AsyncWS_Response
   Oracle Fusion Procurement Product	PRC_AsyncWS_Request and PRC_AsyncWS_Response
   Oracle Fusion Project Product	PRJ_AsyncWS_Request and PRJ_AsyncWS_Response
   Oracle Fusion Supply Chain Management Product	SCM_AsyncWS_Request and SCM_AsyncWS_Response
   Oracle Fusion Setup Product	COMMON_AsyncWS_Request and COMMON_AsyncWS_Response
   IC	IC_AsyncWS_Request and IC_AsyncWS_Response

   For more information about the different types of loggers to set, see section Setting Logging Levels for Troubleshooting in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

Problem

Long running synchronous calls to remote web services end with JTA transaction rolled-back errors. When executing a transaction making an asynchronous call (for example, to the SOA server), the application returns with an error. Server logs show JTA transaction timeouts, which can cause this behavior.

Solution

Check the JTA transaction timeout in Oracle WebLogic Administration Console.

1. Log in to Oracle WebLogic Server Administration Console.

2. In the Domain Structure, select Services then, JTA to check the timeout value.

   If the transaction is always timing out beyond 30 seconds and is a custom composite synchronous client invocation, then the design approach may need to be revisited. It may be best for the external web service to be invoked as an asynchronous transaction.

   Increasing the JTA for supporting long running synchronous transactions is simply an interim mechanism.

3. If synchronous client invocations to Oracle Fusion Applications code take a long time, check for any performance issues with the system and try to resolve them.

4. Check for appropriate values for the syncMaxWaitTime property and BPEL's EJB transaction timeout settings in relation to the JTA timeout settings and only then increase the value of the JTA timeout, if needed.

5. To view and change the BPEL EJB transaction timeout settings, perform the following steps:

   1. Log in to Oracle WebLogic Server Administration Console.

   2. In the Domain Structure, click Deployments.

   3. Expand soa-infra > EJBs.

   4. Update the following EJBs:

      - BPELActivityManagerBean

      - BPELDeliveryBean

      - BPELDispatcherBean

      - BPELEngineBean

      - BPELFinderBean

      - BPELInstanceManagerBean

      - BPELProcessManagerBean

      - BPELSensorValuesBean

      - BPELServerManagerBean

   5. Click Save.

   6. Restart Oracle WebLogic Server.

6. For asynchronous transactions, check the values for both the BPEL EJB transaction timeout and the JTA transaction timeout and adjust as needed.

The following documentation provides additional information about the topics discussed in this section:

• For information about changing the JTA transaction timeout setting, see section Resolving Connection Timeouts in theAdministering Oracle SOA Suite and Oracle Business Process Management Suite.

• For information about viewing and changing the syncMaxWaitTime property, see section Configuring BPEL Process Service Engine Properties in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

Problem

Messages primarily get lost for the following reasons:

1. The EDN message was not delivered.

2. The target asynchronous service did not respond.

3. The message was roll backed (though it was not lost; it is still seen in Fusion Applications Control).

Solution

Identify where the messages are supposed to be and diagnose the path, is what this issue typically requires.

• If the calling component is an ADF web service invoking a SOA composite, then check for any client errors in the ADF logs.

• Because ADF invokes SOA composites using EDN, check the JMS/AQ and XA data source configurations to ensure that events are being fired. For more information about EDN issues, see Business Event Subscriptions Are Not Firing.

• Check if the Oracle WSM Policy Manager security configurations allow the client to invoke the SOA composite.

• Verify that the transaction is not transient (nonpersistent).

  Oracle BPEL Process Manager uses the dehydration store database to maintain long-running, asynchronous processes and their current state information in a database while they wait for asynchronous callbacks. Storing the process in a database preserves the process and prevents any loss of state or reliability if a system shuts down or a network problem occurs. There are two types of processes in Oracle BPEL Process Manager. These processes affect the dehydration store database in different ways.

  • Transient processes: This process type does not incur any intermediate dehydration points during process execution. If there are unhandled faults or there is system downtime during process execution, the instances of a transient process do not leave a trace in the system. Instances of transient processes cannot be saved in-flight (whether they complete normally or abnormally). Transient processes are typically short-lived, request-response style processes. The synchronous process designed in JDeveloper is an example of a transient process.

  • Durable processes: This process type incurs one or more dehydration points in the database during execution because of the following activities:

    • Receive activity

    • OnMessage branch of a pick activity

    • OnAlarm branch of a pick activity

    • Wait activity

    Instances of durable processes can be saved in-flight (whether they complete normally or abnormally). These processes are typically long-living and initiated through a one-way invocation. Because of out-of-memory and system downtime issues, durable processes cannot be memory-optimized.

• If a composite instance is not visible in Fusion Applications Control and the SOA Infrastructure is running, check that the Audit Level is not set to Off in the SOA Infrastructure Common Properties page. This can be checked in Fusion Applications Control:

  1. Right-click soa-infra (SOA_cluster_name).

  2. Choose SOA Administration and then, Common Properties.

• Check the ADF logs to see if there was a bad/malformed message or incorrect SOAP headers (if invoking from an external web service). The transaction may have rolled back before being dehydrated due to an internal error. Again, the SOA logs can identify the cause of this issue.

• If the composite instance is available, check the bpel.config.oneWayDeliveryPolicy BPEL property value. The value can be checked in Fusion Applications Control:

  1. In the navigation pane, expand soa-infra (SOA_cluster_name).

  2. Expand the partition, and select the composite.

     The Dashboard page for the composite is displayed.

  3. In the upper right corner, click the Show XML Definition icon.

     The contents of composite.xml for that composite are displayed.

  If this is set to async.cache, messages may be lost. Set it to async.persist for reliable messages. This is typically specified in the BPEL process service component section of the composite.xml file, so this can be set for custom composites. If the value is not set in composite.xml, the value for oneWayDeliveryPolicy in the System MBean Browser in Fusion Applications Control is used. The following values are possible:

  • async.persist: Messages are persisted in the database hash map.

  • async.cache: Messages are stored in memory.

  • sync: Direct invocation occurs on the same thread.

  It is also possible that the transaction has invoked a target asynchronous service that has not responded back. In this case, the composite instance flow shows the call to the target asynchronous service.

• Check if the instance has rolled back and the message is in recovery.

  1. Log in to Fusion Applications Control.

  2. Right-click soa-infra (SOA_cluster_name), and choose Service Engines then, BPEL.

  3. Click the Recovery tab.

  This may occur if any external references receiving the message are not reachable (for example, an external web service, enterprise applications such as Siebel, and so on).

The following documentation provides additional information about the topics discussed in this section:

• For information about oneWayDeliveryPolicy property settings, see sections Deployment Descriptor Properties appendix and Transaction and Fault Propagation Semantics in BPEL Processes in the Developing SOA Applications with Oracle SOA Suite.

• For information about setting the oneWayDeliveryPolicy property in the System MBean Browser, see section Configuring BPEL Process Service Engine Properties in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

Problem

When a BPEL process flow errors out, it is retried with all its invocations. This is undesirable in some cases.

Solution

The property GlobalTxMaxRetry (default value is 3) specifies how many retries are performed if an error is identified as a retriable one. For example, after several web service invocations, if dehydration fails due to a data source error, then this is identified as a retriable error and all activities from the prior dehydration state are retried. If the activities being retried are not idempotent (that is, their state can change with each retry and is not guaranteed to give the same behavior), then multiple retries can be problematic.

To rectify this, customize the composite by specifically marking the nonidempotent activities with idempotentset set to false in the partner link settings section of the composite.xml file to prevent retries.

<property name="bpel.partnerLink.partner_link_name.idempotent">false</property>

The idempotentset property cannot be set in Fusion Applications Control.

GlobalTxMaxRetry can also be set to 0 in the Systems MBean Browser.

For information about the idempotent property, see the Tuning BPEL Properties Set Inside a Composite section of Tuning Performance.

For information about Oracle SOA Suite customizations in JDeveloper, see the Customizing and Extending SOA Components section in the Oracle Applications Cloud Customizing the Applications for Functional Administrators.

Problem

BPEL processes have an automatic recovery feature that attempts to automatically recover activities that are recoverable such as unresolved invoke/callback messages, activities not completed over a provided threshold time, and so on. However, the automatic recovery feature only tries to recover a few instances and only retries a fixed number of times. If some instances are not being automatically recovered, they are likely not being picked up because of the configuration of the automatic recovery parameters.

Solution

• Set the maximum number of messages to automatically recover.

  By default, the automatic recovery feature of Oracle BPEL Process Manager processes fifty messages to submit for each recovery attempt. This is controlled by the maxMessageRaiseSize property.

  1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

  2. Choose SOA Administration, then BPEL Properties, then More BPEL Configuration Properties, and then RecoveryConfig.

  3. Expand both RecurringScheduleConfig, then maxMessageRaiseSize and StartupScheduleConfig, and then maxMessageRaiseSize.

     The default value is 50 for each. A negative value causes all messages selected from the database to be submitted for recovery. A value of 0 causes no messages to be selected from the database (effectively disabling recovery). To recover more than fifty messages, set the property value to that value. Use this property to limit the impact of recovery on the server.

• Set the maximum number of automatic recovery attempts on a given message.

  The number of automatic recovery attempts can also be configured to submit in the same recoverable instance. The value provided, specifies the maximum number of times that invoke and callback messages are recovered. If the value is 0 (the default value), it recovers all messages. After the number of recovery attempts on a message exceeds the specified value, a message is marked as nonrecoverable.

  Follow these steps to configure automatic recovery attempts for invoke and callback messages in Fusion Applications Control.

  1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

  2. Choose SOA Administration, then BPEL Properties, and then More BPEL Configuration Properties.

  3. Select MaxRecoverAttempt, and enter a value in the Value field.

  4. Click Apply.

     It may not be desirable in all cases to use automatic recovery. If services are not idempotent, then corruption can occur. Moreover, the automatic recovery restores the composite to the last save point that can be immediately after an asynchronous invoke, wait, and so on. Therefore, it is important to understand the process behavior and what it does next before performing mass recoveries. Attempt mass automatic recovery only after the root cause of the composite failures has been fixed (for example, a service that was unavailable is now available, a database running out of space was fixed, and so on). Automatic recovery can also trigger an unexpected load during failure scenarios. This causes more threads to block on a remote server that can induce hangs in the SOA server in a cascading fashion.

     For more information, see section Some Composites Are Being Retried Multiple Times on Failure and section Configuring Automatic Recovery for Oracle BPEL Process Manager in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

Problem

The following error is displayed if either the endpoint is not available for a reference or a composite is deployed with an incorrect deployment plan file.

oracle.fabric.common.FabricException: Cannot read WSDL

Solution

Problem

Oracle Support Services has requested the composite to be uploaded to review the flow and associated configuration files.

Solution

Problem

Oracle Support Services has requested the configuration properties of various SOA components to be confirmed.

Solution

Problem

Solution

Problem

Solution

Problem

Solution

Problem

Solution

Problem

Solution

Problem

If the Worklist: Notifications and Approvals region in the Welcome dashboard on the Oracle Fusion Applications home page does not contain data, it is likely the connection from Oracle BPM Worklist to the SOA Managed Server is timing out.

Solution

1. Ensure the SOA Managed Servers are running in the CommonDomain domain with Fusion Applications Control:

   1. From the navigation pane, expand the farm, WebLogic Domain, CommonDomain, FS_SOACluster.

   2. Click FS_SOACluster to display the Oracle WebLogic Server Cluster Home page.

   3. In the Servers section, view the Status column of the SOA Managed Server, for example, soa_server1.

      If the Managed Server is not running, restart it.

2. Locate the HomePageServer_n-Diagnostic.log diagnostic log files on the CommonDomain domain. This log file is typically stored in the following locations:

   (UNIX) DOMAIN_HOME/servers/server_name/logs/HomePageServer_n-Diagnostic.log
   

   These log files may be stored in a different location if the log file was reconfigured.

3. Review the HomePageServer_n-Diagnostic.log diagnostic log for the following exception:

   [HomePageServer_1] [ERROR] []
   [oracle.soa.services.workflow.worklist] [tid: [ACTIVE].ExecuteThread: '24'
   for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: username]
   [ecid: ecidvalue] [APP: HomePageApp#V2.0] [URI:
   /homePage/faces/AtkHomePageWelcome] TimeoutException occurred for
   server='null'.[[
   oracle.bpel.services.common.concurrent.TimeoutException: Execution timeout
        service : FederetedWorklistService
   

4. Modify default timeout interval in the profile:

   1. Log in to the Setup UI application.

   2. From the Administration menu in the work area, choose Setup and Maintenance.

      The Overview page displays.

   3. Click the All Tasks tab.

   4. In the Name field, enter Profiles and click Search.

   5. In the Search Results table, in the Manage Administrator Profile Values table, click the Go to Task.

      Figure 13-2 Search Results Table

      
      

      
      Description of "Figure 13-2 Search Results Table"

      
      

      From the Manage Administrator Profile Values page, in the Profile Display Name field, enter Welcome Dashboard Worklist Timeout Interval, and then click Search.

   6. In the Search Results table, select ATK_HOME_PAGE_WORKLIST_TIMEOUT.

   7. In the ATK_HOME_PAGE_WORKLIST_TIMEOUT: Profile Values table, modify the Profile Value field.

      Increase the default timeout of 5 seconds to a larger value such as 10 seconds. A larger timeout value increases the amount of time that home page server waits for response from background SOA web services for all users and it might adversely affect home page performance when there are many concurrent users. Increasing this value should be a temporary change until the root cause for the delay in SOA federated worklist response time can be found.

   8. Click Save.

   9. Run a session, and check that the Worklist: Notifications and Approvals region in the Welcome dashboard on the Oracle Fusion Applications home page contain data.

For information about starting managed servers, see the Start Administration Servers and Managed Servers section.

Problem

The document is stuck pending approval and the logs show the following error:

"Unable to walk hierarchy for user username'

This occurs when the hierarchy for the given user has not been set up correctly. For example, assume there are approval rules set up so that the rule getting fired uses the supervisory hierarchy and it is expected to route to USER01 then, USER02 then, USER03 and so on up to USER10 (ten level hierarchy). If the supervisory/management hierarchy of USER01 has not been set up, then the approval task throws an error stating the inability to go through the hierarchy for USER01.

Solution

Since the user level hierarchy does not exist for the user_name, verify the Human Capital Management (HCM) setup by manually running the HCM service to which workflow-identity-config.xml is pointing. If it does not return any result, then configure the correct hierarchy for the user in HCM.

Run the HCM service, select the fetchManager(s) API (for supervisory), and provide the following payload, where <ns2:Id>DD7A1614BBFAA0F0A4511ACD96D2C88D</ns2:Id> is the GUID of the user whose manager is being searched. The supervisory hierarchy and position hierarchy services can be found under HcmCore. The name is HierarchyProviderService. The corresponding WSDLs are as follows:

• Supervisory hierarchy:

  http://host:port/hcmEmploymentCoreApprovalHierarchy/HierarchyProviderService?wsdl
  

• Position hierarchy:

  http://host:port/hcmTreesModel/HierarchyProviderService?wsdl
  

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>
<wsse:Security soap:mustUnderstand="1"
xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-sec
ext-1.0.xsd">
<wsse:UsernameToken>
<wsse:Username>FUSION_APPS_AMX_APPID</wsse:Username>
<wsse:Password
Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profi
le-1.0#PasswordText">Password</wsse:Password>
</wsse:UsernameToken>
</wsse:Security>
</soap:Header>
<soap:Body
xmlns:ns1="http://xmlns.oracle.com/apps/hcm/employment/core/approvalHierarchyServi
ce/types/">
<ns1:fetchManagers>
<ns1:principal
xmlns:ns2="http://xmlns.oracle.com/apps/hcm/employment/core/approvalHierarchyServi
ce/">
<ns2:Id>DD7A1614BBFAA0F0A4511ACD96D2C88D</ns2:Id>
</ns1:principal>
<ns1:level>2</ns1:level>
</ns1:fetchManagers>
</soap:Body>
</soap:Envelope> 

The following example shows the workflow-identity-config.xml file used in a development environment:

<?xml version='1.0' encoding='UTF-8'?>
<ISConfiguration xmlns="http://www.oracle.com/pcbpel/identityservice/isconfig/">
<configurations>
<configuration realmName="myrealm">
<provider name="JpsProvider" providerType="JPS" service="Identity">
<property name="jpsContextName" value="default"/>
<property name="IdentityServiceExtension" value="HCMIdentityServiceExtension"/>
</provider>
</configuration>
</configurations>
<property name="caseSensitive" value="false"/>
<property name="caseSensitiveGroups" value="true"/>
<serviceExtensions>
<serviceExtension name="HCMIdentityServiceExtension">
<serviceProvider
classname="oracle.bpel.services.identity.hierarchy.providers.hcm.HCMHierarchyProvi
der" type="supervisoryHierarchyProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmEmploymentCoreApprovalHierarchy/Hie
rarchyProviderService?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
<serviceProvider
classname="oracle.bpel.services.identity.hierarchy.providers.hcm.HCMHierarchyProvi
der" type="jobLevelHierarchyProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmEmploymentCoreApprovalHierarchy/Hie
rarchyProviderService?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
<serviceProvider
classname="oracle.bpel.services.identity.hierarchy.providers.hcm.HCMPositionHierar
chyProvider" type="positionHierarchyProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmTreesModel/HierarchyProviderService
?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
<serviceProvider
classname="oracle.bpel.services.identity.position.provider.hcm.PositionLookupServi
ceProvider" type="positionLookupProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmEmploymentCore/positionLookupServic
e?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
<serviceProvider
classname="oracle.bpel.services.identity.position.provider.hcm.PositionDisplayName
Provider" type="positionDisplayNameProvider">
<initializationParameter name="wsdlUrl"
value="http://example.com:20619/hcmTreesModel/HierarchyProviderService
?WSDL"/>
<initializationParameter name="csf-key-name" value="FUSION_APPS_AMX_APPID-KEY"/>
<initializationParameter name="http-read-timeout" value="30000"/>
<initializationParameter name="securityPolicyName" value="oracle/wss_username_
token_client_policy"/>
</serviceProvider>
</serviceExtension>
</serviceExtensions>
</ISConfiguration>

The workflow-identity-config.xml file is in the MDS repository. For information about how to export the workflow-identity-config.xml file from MDS, see Hierarchy List Builders Display an Error Due to Security Configuration Issues and Log Report Errors in Authentication or Authorization with HCM Services.

Problem

Hierarchy list builders (for example, supervisory, job-level, and position) used in rules are not working as expected and are throwing security configuration-related errors. For example:

SOAPFaultException: FailedAuthorization

Hierarchy services are protected services using Oracle Web Services Manager policies (for example, supervisory, job-level, and position services). Elevated privileges are used for authentication. These privileges must be configured in Fusion Applications Control and in configuration files.

Solution

To resolve this problem:

1. Check the MDS store to see if workflow-identity-config.xml and workflow-config.xml are set up with the correct information (pointing to HCM services and the JAZN name).

   1. Run the following script to connect to the WLST shell on Linux operating systems.

      $MW_HOME/oracle_common/common/bin/wlst.sh
      

   2. Run connect(). This takes to the prompt as shown in the following example:

      wls:/offline> connect()
      Please enter your username [weblogic] :weblogic
      Please enter your password [weblogic] :
      Please enter your server URL [t3://localhost:7001]
      :t3://example.com:9401
      

   3. Run the following command after connecting:

      exportMetadata(application='soa-infra',server='soa_server1',toLocation='any
      Location on
      server',docs='/soa/configuration/default/workflow-identity-config.xml')
      

      The file is stored in the soa/configuration/default directory in the location given in the toLocation attribute. After verifying, the file can be uploaded with the following command.

      importMetadata(application='soa-infra',server='soa_
      server1',fromLocation='any Location on
      server',docs='/soa/configuration/default/workflow-identity-config.xml')
      

2. Verify that the csf-keys are specified as initialization parameters.

   1. Ensure that the csf-key-name value is FUSION_APPS_AMX_APPID-KEY in workflow-identity-config.xml, as described in Document Is Stuck in Pending Approval and Logs Show Error "Unable to walk hierarchy for user user_name".

3. Check in Fusion Applications Control that the keys defined for the services have the correct values for authentication.

   1. Retrieve a credential with a given map and key by using scripting to invoke the MBean operation JpsCredentialMXBean.getPortableCredential(map, key).

      (map="oracle.wsm.security", key="FUSION_APPS_AMX_APPID-KEY")
      

      For more information about this MBean, see the Managing Credentials with OPSS Scripts section of Oracle Fusion Middleware Applications Security Guide.

Problem

This issue may be noticed by organizations using position hierarchy such as governments and universities. After users log in, they do not see any approvals in their inbox. This is caused by a failure to look up the position. This can be caused by network issues when accessing the HCM services. Sometimes due to load balancing or network setup, HCM services may not be accessible and Oracle BPM Worklist has errors such as the following:

WSDL port is not accessible

or

Could not find the host

Solution

Problem

SOA transactions are failing and logs indicate that the database is running out of space. The problem may occur with the SOA dehydration store or MDS store running out of space due to a high volume of transactions. In the latter example, the following errors are seen:

java.sql.SQLException: ORA-01653: unable to extend table SH_MDS.CUBE_INSTANCE by
16 in tablespace FUSION_TS_TOOLS 

This indicates that the tablespace is full and the database cannot extend it.

Solution

The following documentation provides additional information about the topics discussed in this section:

• For information tablespace settings, see the Resolving Message Failure Caused by Too Many Open Files section in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

• For information about creating a purging strategy, see the Managing Database Growth section in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

• For information about tuning parameters, see the Tuning Database Parameters section in the Tuning Performance.

Problem

Slow application performance and/or memory trashing. For example, it may take longer to load and serve pages or to complete composite transactions. Response time may seem slower compared to normal behavior.

There are various reasons for slow performance. It may be due to a large number of servers running on the same host, or there may be a large number of records/sessions/locking at the database. Thread contention can also be a reason for poor performance.

All configuration changes in Fusion Applications Control may be unavailable due to this problem.

Solution

1. Check the CPU utilization to see if it is saturated due to a heavy load or an excessive number of processes in relation to CPU capacity.

   If CPU utilization is at 100% during normal load hours (the target should be 70-80%), there is no capacity to handle a peak load and the hardware resources are insufficient. Add scale-out servers to handle the additional load.

2. Check applications using Fusion Applications Control to report on performance. Check the performance of ADF services and the invoke/response times in the BPEL process audit trail to identify if the issue is in Oracle Fusion Applications.

3. To maximize performance, it is recommended not to set the logging level higher than the default INFO level.

   For debugging purposes, the logging level must be set to the FINEST level. However, after issues are resolved, reset the logging level to the default level for best performance. It is also recommended to set Audit Level to Production in the SOA Infrastructure Common Properties page. This can be set in Fusion Applications Control as follows:

   1. Right-click soa-infra (SOA_cluster_name).

   2. Choose SOA Administration then, Common Properties.

   3. Set Audit Level to Production.

4. Purge periodically based on retention requirements to maintain the service level agreements (SLAs).

5. Check the memory/IO/paging/swapping/CPU usage load statistics using Top or Glance or another monitoring tool.

6. Optimize the JVM to avoid full garbage collection or out-of-memory errors.

   Frequent garbage collection can be either due to higher memory usage or memory leaks.

   1. Ensure that the sum of the maximum heap size of all the JVMs running on the system does not exceed the amount of available physical RAM to avoid operating system level paging activity.

   2. Use the JRockit mission control memory profiling tools to get thread dumps and memory snapshots, which helps Oracle Support Services debug any code issues.

7. Optimize threads to avoid contention. Get a thread dump to investigate and submit it to Oracle Support Services.

8. Check the JVM and thread dumps for methods invoking the database to identify if database performance is a bottleneck.

9. Run database Automatic Workload Repository (AWR) snapshots to identify causes of database performance issues.

10. Ensure that database statistics are updated at regular intervals and other tunable parameters for memory, tablespace, and partitions are used effectively to obtain maximum performance.

    Here are some common tuning recommendations:

    • Put indexes and tables in as physically separate disk areas if possible.

    • Never put rollback segments with data or index segments.

    • Separate highly active tables and indexes into their own tablespaces.

    • Partition high activity tables and indexes to help balance disk I/O and prevent hot disks.

    • Have processes in place to generate database table statistics at regular intervals.

11. Tune database tables to control the high watermark (HWM) contention of large objects. Tune database advanced queues (AQ) to control high watermark (HWM) contention and ensure consistent performance of producing and consuming messages from AQ.

    Specifically, the AQs to be aware of are as follows:

    Family AQ	Queue Name
    EDN	EDN_EVENT_QUEUE_TABLE and EDN_OAOO_DELIVERY_TABLE
    CRM	CRM_AsyncWS_Request and CRM_AsyncWS_Response
    HCM	HCM_AsyncWS_Request and HCM_AsyncWS_Response
    FIN	FIN_AsyncWS_Request and Fin_AsyncWS_Response
    PRC	PRC_AsyncWS_Request and PRC_AsyncWS_Response
    PRJ	PRJ_AsyncWS_Request and PRJ_AsyncWS_Response
    SCM	SCM_AsyncWS_Request and SCM_AsyncWS_Response
    COMMON	COMMON_AsyncWS_Request and COMMON_AsyncWS_Response
    IC	IC_AsyncWS_Request and IC_AsyncWS_Response
    Cross Family Business Event Subscriptions	ACR_XFAMILY_EVENT_Q and ACR_XFAMILY_EVENT_QT

12. Tune the BPEL and EDN thread counts to ensure optimal settings (for example, the Dispatcher Invoke Threads and Dispatcher Engine Threads properties on the BPEL Service Engine Properties page in Fusion Applications Control). If the thread configuration is too high, the servers run out of memory. If they are too low, the messages start backing up.

13. Tune the BPEL properties to reduce overhead (for example, disable the ValidateXML and StatsLastN (statistics gathering batch size) properties on the BPEL Service Engine Properties page in Fusion Applications Control) if they are not needed.

14. In case of integration with packaged applications (for example, Siebel), check if the issue lies with the legacy applications.

The following documentation provides additional information about the topics discussed in this section:

• For information about performance tuning the various components, see the Top Performance Areas and the SOA Suite Components section in the Tuning Performance.

• For information about setting properties on the BPEL Service Engine Properties page, see the Configuring BPEL Process Service Engine Properties section in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

• For information about identifying the tables where data growth may lead to performance issues, see the Identifying Tables Impacted By Instance Data Growth section in the Tuning Performance.

• For information about tuning, see the Tuning Database Parameters section in the Tuning Performance.

• For information about the use of the purge scripts, see the Managing Database Growth section in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

Problem

Fusion Applications Control pages are loading very slowly.

Solution

To improve Fusion Applications Control page load times on pages with instance and fault metrics, set the SOA Infrastructure Display Data Counts properties.

In addition to setting this property, it is also useful to scope queries by time where available (for example, only show items occurring in the last hour).

For more information, see the Optimizing the Loading of Pages with Instance and Fault Metrics section in the Administering Oracle SOA Suite and Oracle Business Process Management Suite.

Problem

JRockit writes log information (called flight recorder files) under /tmp when using JRockit Mission Control. When it rotates to a new flight recorder file, the JVM is suspended until it completes. If the old directories in use under /tmp are deleted (for example, a cleanup or custom maintenance script to run to clear /tmp when it is close to full to avoid server downtime), the domain may go into a suspended state because the JVM may wait for a directory that no longer exists.

Solution

To recover existing, suspended JVMs without restarting them, the directories that the JVM was trying to use must be recreated.

Problem

The SOA server (soa_infra) does not start.

Solution

There can be various reasons for server startup issues. A discovery-based approach to find the root cause is required. Check the server and diagnostic logs as a first attempt to diagnose the issue. In addition, check the following:

1. Check if the database is not available or there are not enough connections available. Some failures with SOA runtime can result from database outage/connectivity issues. Perform the following steps:

   1. Log in to Oracle WebLogic Server Administration Console.

   2. In the Domain Structure, view the status by selecting Services then, Data Sources then, SOADatasource then, Monitoring and then, Testing to test the data source. This can also be checked from a SQL prompt.

2. Check the list of ports used for port conflicts.

3. Check if the MDS repository is unable to load shared documents (incorrect MDS configuration or the database that holds the MDS schema is not reachable).

4. Check the Coherence configuration if using a SOA cluster (use of unicast versus multicast).

   This issue only applies after provisioning is complete. Clustering of SOA servers may fail if there are port conflicts (used by Coherence). For example, one SOA server may be picking up the deployment, but the other server is not. In some cases, this may only present the following error:

   [soa_server1] [ERROR] [] [Coherence] [tid: Logger@352821903 3.6.0.4] 
   [ecid: 46f620208907e045:63f295ec:12dd091ec2e:-8000-0000000000000003,1:27187] 
   [APP: soa-infra] 2011-01-28 23:06:19.463/414.816 Oracle Coherence GE 3.6.0.4 
   <Error> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
    (self-tuning)', member=n/a): 
   Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout
    during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster[[...
   

   In this example, Coherence timeouts are prominently available. However, sometimes it presents itself with an unrelated error, such as the following:

   Error creating bean with name 'SensorManager' defined in ServletContext
   resource
   

   The root cause of this is still primarily related to Coherence configuration.

5. Check if both the Administration Server and Managed Server ports are open and accessible.

6. Check if Managed Server startup failed because the Administration Server is not reachable.

7. Check network issues (for example, IP routing filtering/rules that may be causing issues).

8. Check Oracle WebLogic Server LDAP security corruption.

   The Managed Server may report that policies for the application System MBeans Browser already exist or do not exist.

For more information, see sections Setting the Frontend URL for the Administration Console and Setting Redirection Preferences and Setting the Frontend HTTP Host and Port in Enterprise Deployment Guide for Oracle SOA Suite.

Problem

The SOA server (soa_infra) is not responding to administrative commands. This problem can be due to the environment using a load balancer, and the default listener address or channel is local-only. Therefore, the connection cannot be made from the external client or server.

Solution

To resolve this problem, perform the following steps from the Oracle WebLogic Server Administration Console:

1. Ensure that the listener address is correctly defined and accessible.

   1. From the Domain Structure, expand Environment and select Servers.

   2. From the Servers table, click AdminServer(admin).

   3. From the Settings for AdminServer page, click the Protocols tab.

   4. Click the HTTP subtab.

   5. Set the Frontend Host property to the load balancer address.

2. Similarly, set the Frontend HTTP Host and Frontend HTTP Port properties for the SOA cluster:

   1. From the Domain Structure, expand Environment and select Clusters.

   2. From the Clusters table, click domain_SOACluster

   3. From the Settings for the domain_SOACluster page, click the HTTP subtab.

   4. Set the Frontend Host property to the load balancer address.

   5. Change the Frontend HTTP Host and Frontend HTTP Port properties.

For more information, see sections Setting the Frontend URL for the Administration Console and Setting Redirection Preferences and Setting the Frontend HTTP Host and Port section in the Enterprise Deployment Guide for Oracle SOA Suite.

Problem

Get a list of all composite state values from the *_SOAINFRA.COMPOSITE_INSTANCE state table (for diagnosis).

Solution

Table 13-2 shows the composite state values.

Problem

The SOA log files are getting deleted. This is typically due to the retention policy set in Fusion Applications Control.

Solution

Change the retention policy for the odl-handler to the appropriate value in Fusion Applications Control:

Problem

An internal key is displayed instead of a translated string in the human task title, human task outcome, email subject and body, and human task mapped attributes displayed in Oracle BPM Worklist.

Solution

To resolve this problem:

1. Check the SOA server-diagnostic.log for errors for each server in the cluster. If a resource bundle or the key is not found, a similar error stack that contains the composite name and resource bundle name is seen.

   2010-09-29T23:45:46.702-07:00 soa_server1 ERROR \[\]
    oracle.soa.services.workflow.soa tid: orabpel.invoke.pool-4.thread-7 userId:
    weblogic
   ecid:59ab739a26595dc1:-38d33932:12b614ddb44,0:1:100000026
   APP: soa-infra composite_name: RC3HumanTaskComposite component_name: Humantask1
    component_instance_id: composite_instance_id: 310003
   <WorkflowServiceEngine.request> Operation 'initiateTask' failed with exception
      'EJB Exception: : java.util.MissingResourceException:
   Can't find bundleforbase name
      oracle.apps.hcm.people.soa.resource.HcmPeopleTopSoaBundleDummy, locale
    en\[\[
   at
   java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1427
   )
   at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1250)
   at java.util.ResourceBundle.getBundle(ResourceBundle.java:952)
   

2. Confirm that the resource bundle has been deployed to the correct location in SOA MDS by first exporting the contents of SOA MDS and confirming that the resource bundle class is defined under /apps/resource. Follow these steps to export the metadata contents from Fusion Applications Control:

   1. From the navigation pane, expand SOA and select soa-infra.

   2. From the SOA Infrastructure menu, choose Administration then, MDS Configuration.

   3. Select Export metadata documents to an archive on the machine where the web browser is running.

   4. Click Export.

3. If the resource bundle is not defined in SOA MDS or is in the wrong location in SOA MDS, then an error during patching may have been encountered or there is a problem with the resource bundle definition.

   If the resource bundle is properly defined in SOA MDS and the string that is not translated is a mapped attribute that appears in the column heading of the worklist, then perform the followings steps to:

   • Set the System MBean Browser WorkflowCustomClasspathURL property to a value of "".

   • Apply the changes.

   • Set WorkflowCustomClasspathURL to "oramds:///apps/resource/".

   • Apply the changes in Fusion Applications Control.

   1. Log in to Fusion Applications Control in the domain where the JAR file was deployed.

   2. From the navigation pane, expand SOA and select soa-infra.

   3. From the SOA Infrastructure menu, choose SOA Administration then, System MBean Browser.

   4. Go to Application Defined MBeans then, oracle.as.soainfra.config then, Server: SOA_cluster_name then, WorkflowConfig and then, human-workflow.

   5. Remove the value in the Value column for the WorkflowCustomClasspathURL attribute.

   6. Click Apply.

   7. Enter oramds:///apps/resource/ in the Value column of the WorkflowCustomClasspathURL attribute.

   8. Click Apply.

   If the resource bundle is properly defined in SOA MDS and the string that is not translated is not a mapped attribute that appears in the column heading of the worklist, then stop and restart the composite that references the resource bundle by following these steps:

   1. Log in to Fusion Applications Control in the domain where the JAR file was deployed.

   2. From the navigation pane, expand SOA and select soa-infra.

   3. Click Shut Down.

   4. Click Yes in the confirmation window.

   5. Click Start Up.

   6. Click Yes in the confirmation window.

4. Retest the transaction flow.

Problem

When a BPEL process has partner links targeted to external servers, during start up of the SOA server or activation of the BPEL composite, the BPEL service engine recognizes that the target server is down and marks the BPEL composite as unavailable. Even after the target server becomes available, there is no way to make the BPEL process available without restarting the complete SOA server or redeploying the BPEL process. Until then, the BPEL composite is not listed in the navigation pane of Fusion Applications Control and is only visible through the Deployed Composites tab. Even though the composite is displayed here (sometimes as active), if the process is clicked on, this error is received:

The composite <Process Name>[ <Version> ] is not available. This could happen
because either the composite has been undeployed or EM has not yet discovered
this composite.

Solution

The issue is caused by using a concrete WSDL file in the BPEL definition. Using an abstract WSDL prevents the BPEL composite/process from becoming inactive or its state from becoming retired when the target server or service is down.

Oracle Fusion Applications develop composites with oramds:// references to WSDL files at design time for both the binding and the WSDL location. At deployment, the binding's concrete location (which is only used at runtime) is fixed.

Follow these steps to use an abstract WSDL by making a local copy of the target WSDL file in the BPEL project.

Problem

When Oracle SOA Suite is installed on Domain1 and ADF on Domain2 and if any remote calls are performed between them for any human workflow API use or for notifications, errors similar to the following are thrown if the domain trust is not set.

javax.servlet.ServletException:
javax.servlet.ServletException:java.lang.SecurityException: Security:090398Invalid
Subject:principals=CVBUYER01, AR_MANAGER_VISION_OPERATIONS_DATA, AR_ACCOUNTS_
RECEIVABLE_MANAGER_JOB, FBI_TRANSACTIONAL_BI_WORKER

Solution

Set the domain trust password on both domains (global trust).

Oracle Fusion Applications do not support RMI across domains. That is, all remote calls between ADF and SOA are always within the same domain. When cross domain communication is required, SOAP is used. The scenario described in this section may be encountered in the development environment. For example, the ADF task flow for human tasks may be running in the JDeveloper-integrated Oracle WebLogic Server and running the composite in an independent standalone SOA server. Both are in different domains.

Problem

The following error can be received when making customization modifications to an Oracle Mediator service component in a SOA application.

For example, assume the following steps are performed:

Solution

This error can be ignored. After this error occurs, everything still works correctly.

Problem

The following error can be received when making a customization modification to an Oracle Mediator service component in a SOA application.

For example, assume the following steps are performed:

Solution

If no alerts indicating that a WSDL file cannot be found or parsed are seen, then this exception in the log/console can be ignored.

Problem

Solution

This is only a warning message thrown by the compiler. Runtime works fine. Having a reference from SOA MDS in a composite makes the port and location values empty in the bindings. To resolve this warning message:

• Explicitly enter the corresponding details. For example, replace the binding.ws location value with the concrete WSDL file.

• Deploy the composite using the configuration plan, which replaces the port and oramds location with the concrete WSDL file.