C Troubleshooting Oracle SOA Suite and Oracle BPM Suite

This appendix describes how to troubleshoot issues you can encounter when using Oracle SOA Suite and Oracle BPM Suite, including logging level setup, parallel purge and table partitioning issues, connection and transaction timeout issues, runtime diagnostic issues, human workflow issues, Event Delivery Network (EDN) issues, performance issues, server startup best practices, and browser issues. References to additional troubleshooting information are also provided.

This appendix includes the following sections:

Setting Logging Levels for Troubleshooting

To simplify troubleshooting, it is recommended that you set logging levels to the TRACE:32 FINEST level in Oracle Enterprise Manager Fusion Middleware Control. This section describes loggers to which to set to this level.

To set logging levels for troubleshooting:

  1. See Configuring Log Files for instructions on accessing the Log Configuration page.
  2. From the Oracle Diagnostic Logging Level (Java Level) list, set the following parent loggers to the TRACE:32 FINEST level:
    • oracle.soa

    • oracle.fabric

    • oracle.integration

    • oracle.wsm (Setting this logger to the ERROR level may also be sufficient because this setting logs the required error messages.)

  3. If you want finer-grained control over logging, expand the parent loggers and set any of the following loggers:
    Component Runtime Loggers

    Human workflow/approval management extensions (AMX)/rules

    • oracle.soa.services.common

    • oracle.soa.services.identity

    • oracle.soa.services.notification

    • oracle.soa.services.rules

    • oracle.soa.services.rules.obrtrace

    • oracle.soa.services.workflow

    • oracle.soa.services.workflow.common

    • oracle.soa.services.workflow.evidence

    • oracle.soa.services.workflow.metadata

    • oracle.soa.services.workflow.persistency

    • oracle.soa.services.workflow.query

    • oracle.soa.services.workflow.report

    • oracle.soa.services.workflow.runtimeconfig

    • oracle.soa.services.workflow.soa

    • oracle.soa.services.workflow.task

    • oracle.soa.services.workflow.task.dispatch

    • oracle.soa.services.workflow.task.routing

    • oracle.soa.services.workflow.user

    • oracle.soa.services.workflow.verification

    • oracle.soa.services.workflow.performance

    • oracle.soa.services.workflow.worklist

    SOA Infrastructure

    • oracle.fabric.common.wsdl

    • oracle.integration.platform.blocks.deploy

    • oracle.integration.platform.blocks.soap

    • oracle.integration.platform.blocks.local

    • oracle.integration.platform.kernel

    • oracle.integration.plaform.blocks.mesh

    • oracle.integration.platform.common

    • oracle.integration.platform.instance

    • oracle.integration.platform.instance.activity

    • oracle.integration.platform.instance.store

    EDN

    • oracle.integration.platform.blocks.event

    • oracle.integration.platform.blocks.event.saq

    • oracle.integration.platform.blocks.event.jms

    Deployment

    oracle.integration

    Oracle Mediator

    • oracle.soa.mediator.common

    • oracle.soa.mediator.common.cache

    • oracle.soa.mediator.common.error

    • oracle.soa.mediator.common.error.recovery

    • oracle.soa.mediator.common.listener

    • oracle.soa.mediator.common.message

    • oracle.soa.mediator.common.persistence

    • oracle.soa.mediator.dispatch

    • oracle.soa.mediator.dispatch.db

    • oracle.soa.mediator.dispatch.resequencer.toplink

    • oracle.soa.mediator.monitor

    • oracle.soa.mediator.resequencer

    • oracle.soa.mediator.resequencer.besteffort

    • oracle.soa.mediator.resequencer.fifo

    • oracle.soa.mediator.resequencer.standard

    • oracle.soa.mediator.service

    • oracle.soa.mediator.service.common.functions

    • oracle.soa.mediator.service.filter

    • oracle.soa.mediator.service.transformation

    • oracle.soa.mediator.serviceEngine

    Oracle BPEL Process Manager

    • oracle.soa.bpel

    • oracle.soa.bpel.console

    • oracle.soa.bpel.engine

    • oracle.soa.bpel.engine.activation

    • oracle.soa.bpel.engine.agents

    • oracle.soa.bpel.engine.bpel

    • oracle.soa.bpel.engine.compiler

    • oracle.soa.bpel.engine.data

    • oracle.soa.bpel.engine.delivery

    • oracle.soa.bpel.engine.deployment

    • oracle.soa.bpel.engine.dispatch

    • oracle.soa.bpel.engine.sensor

    • oracle.soa.bpel.engine.translation

    • oracle.soa.bpel.engine.ws

    • oracle.soa.bpel.engine.xml

    • oracle.soa.bpel.entity

    • oracle.soa.bpel.jpa

    • oracle.soa.bpel.system

    Oracle BPM Suite

    oracle.bpm.bpmn.engine

    Oracle B2B

    • oracle.soa.b2b.apptransport

    • oracle.soa.b2b.engine

    • oracle.soa.b2b.transport

    • oracle.soa.b2b.ui

    • oracle.soa.b2b.repository

    Oracle adapters

    oracle.soa.adapter

    Oracle Web Services Manager (OWSM)

    oracle.wsm

  4. From the Oracle Diagnostic Logging Level (Java Level) list, change the logger level to one of the following settings:
    • TRACE:1 (FINE)

    • TRACE:16 (FINER)

    • TRACE:32 (FINEST) - Most verbose level (recommended for troubleshooting)

    The change takes effect within several minutes.

Log Files and Thread Dumps from All Managed Servers

Table C-1 describes the log files to view and thread dumps to obtain.

Table C-1 Log Files and Thread Dumps

Output Description

Server diagnostic log

View the following file:

$DOMAIN_HOME/servers/server_name/logs/server_name-diagnostic.log

For example, soa_server1-diagnostic.log, if server_name is soa_server1.

This is where the log output is available. By default, only the last 100 MB of the diagnostic logs are retained.

Server log

server_name.log (for example, soa_server1.log, if server_name is soa_server1)

Server console output

<stdout> is also useful, especially for deployment and patching issues.

Server thread dump

Enter the following at the operating system command prompt:

kill -3 managed_server_process_ID

You can also use Oracle WebLogic Server Administration Console.

  1. In the navigation tree of Oracle WebLogic Server Administration Console, select Environment > Servers.

  2. In the table, select the server.

  3. Select the Monitoring tab.

  4. Select the Threads tab.

  5. Click Dump Thread Stacks.

The output is in the console logs.

OWSM message log

The following log captures all SOAP messages on the wire.

$DOMAIN_HOME/servers/server_name/logs/owsm/msglogging/diagnostic.log

This log is not enabled by default. To enable this log:

  1. Go to Fusion Middleware Control > Weblogic Domain > Web Services > Policies.

  2. Select the security level for which to enable logging.

  3. Edit the policy to enable the log assertion.

Parallel Purging and Table Partitioning Issues

This section describes how to troubleshoot parallel purge and table partitioning issues. Audit level recommendations are also provided.

You can delete instances and rejected messages with the PL/SQL purge script or from the Auto Purge page in Oracle Enterprise Manager Fusion Middleware Control. For more information, see Managing Database Growth .

Executing the Parallel Purge Script

This section addresses situations in which the Oracle SOA Suite database has not been managed and space must be reclaimed. This section also describes how to identify the appropriate parameter settings for normal, daily execution.

When trying to reclaim space, the purge scripts must be executed frequently to delete substantially more SOA composite applications than are being created. This has implications for host and database resources that must be monitored to avoid lengthy wait queues.

There must be a clear goal on the number of SOA composite applications to delete. Otherwise, the purge script can be configured too aggressively. The following aspects are described:

Specifying the Degree of Parallel Value

When specifying the appropriate degree of parallel (DOP) value, hardware resources must be considered. The DOP identifies the number of jobs scheduled by the database and the number of processes concurrently executed on the host.

The parallel purge jobs are I/O intensive, which means that they are not entirely CPU-bound. However, the following formula is a starting point after which testing and monitoring are required to identify any increases:

DOP <= PARALLEL_THREADS_PER_CPU x CPU_COUNT

While purge jobs are run concurrently, they are actually executed with a 30-second incremental delay to avoid resource contention. The delay is not meant to avoid row contention because the purge jobs must not contend for the same SOA composite applications. This delay has implications for the elapsed time of the purge jobs. As an example, a DOP of 10 means that the tenth purge job sleeps for 270 seconds before starting (that is, (DOP - 1) * 30 sec).

If a high DOP number is specified, then the max_runtime parameter must be set appropriately.

Parsing SOA Composite Applications to Delete

The temptation to delete as many SOA composite applications as possible in each execution of the purge script is understandable when trying to reclaim space. However, the script must parse the data for the appropriate SOA composite application, which can monopolize the elapsed time of the purge.

The strategy must be to reduce the data by running the purge script often with modest date and max_count parameter settings. This reduction requires the parallel purge to be scheduled frequently, perhaps twice an hour.

The set of SOA composite applications to parse is identified by the following parameters:

  • min_creation_date

  • max_creation_date

  • retention_date

  • max_count

A modest max_count parameter setting is the most effective method for reducing parse elapsed times. Set the max_count parameter low, perhaps to 50K, to begin until an optimal value can be identified. As the tables start to reduce in size, the max_count parameter and date ranges can be increased.

Note:

This method of reduction is not always effective. On occasions, you may need to introduce parallel query to parse larger amounts of data.

Using Parallel Query Slaves

The parsing for candidate SOA composite applications to delete is primarily performed by several INSERT ... SELECT statements. The larger the set of SOA composite applications to parse, the more the database cost-based optimizer favors a full table scan. When large portions of a table must be traversed, a full table scan can be faster than index access because it performs multiblock I/O calls.

A parallel query can significantly improve full table scan performance by coordinating and distributing the work over multiple slave processes. Additionally, the PARALLEL hint on the INSERT portion executes in APPEND mode. This further improves performance through direct path inserts.

As with the scheduled purge jobs, the parallel query slaves are I/O intensive and not entirely CPU bound. However, the following formula is a good starting point when selecting a DOP for the PARALLEL hint:

DOP <= PARALLEL_THREADS_PER_CPU x CPU_COUNT

Note:

There is a DOP setting to identify the number of parallel purge jobs and a DOP setting to identify the number of parallel query slaves. These DOP settings both impact the amount of processes executed on the host, but not simultaneously. The INSERT ... SELECT statement executes first to parse the data followed by the scheduling of the parallel purge jobs.

For the INSERT... SELECT statement, the PARALLEL hint can be specified after the INSERT keyword, after the SELECT keyword, or both. Therefore, parallelism of the INSERT and SELECT operations are independent of each other. The appropriate DOP for the entire statement is identified by the following precedence rule:

  • PARALLEL hint on INSERT portion of the statement.

  • Set at session.

  • DOP on INSERT table.

  • Maximum DOP set on any table in the statement.

To execute parallel DML, the following statement must be executed within the script:

ALTER SESSION FORCE PARALLEL DML STATEMENT

For example, this statement has a DOP of 2 for the INSERT and 4 for the SELECT. However, given the precedence rules, the DOP for the entire statement is 2:

INSERT /*+ PARALLEL(TBL_INS,2) */ INTO TBL_INS
SELECT /*+ PARALLEL(TBL_SEL,4) */ * FROM TBL_SEL;

Note:

A parallel query can also be applied to the DELETE and SUBSELECT commands in the purge script, if required.

Debugging and Tracing Purging Operations

This section describes debugging and tracing issues.

Debugging Purging Operations

The purge scripts are instrumented with a DEBUG flag that provides detailed, timestamped information on each operation. This information can be evaluated to identify which tables are difficult to delete and may be better managed outside of the purge scripts through table partitioning.To set the DEBUG flag:

  1. Create an operating system directory in which to write the debug logs.
    mkdir -p /,,/debuglogs
    
  2. Connect to the database with the SYSDBA account to create the database directory.
    SQL> CREATE OR REPLACE DIRECTORY SOA_PURGE_DIR AS '/../DEBUGLOGS';
    
  3. Grant privileges to the Oracle SOA Suite schema owner:
    SQL> GRANT READ, WRITE ON DIRECTORY SOA_PURGE_DIR TO DEV_SOAINFRA;
    
  4. Connect as the Oracle SOA Suite schema owner and set the DEBUG flag.
    cd MW_HOME/SOA_ORACLE_HOME/rcu/integration/soainfra/sql/soa_purge/
    SQL> @DEBUG_ON.SQL;
    
  5. Execute the purge script:
    SQL> SET SEVEROUT ON
    SQL> EXECUTE_PURGE

    Note:

    As an alternative, use the SQL*Plus SPOOL command to capture debug information.

    The Automatic Workload Repository (AWR) reports for the period of the purge can also help to identify the SQL with the longest elapse and execution times.

Tracing Purging Operations

If SQL tracing is required to diagnose the performance of the parallel purge scripts, then the following is required:

  1. Edit procedures soa.delete_instances_in_parallel and soa.delete_insts_in_parallel_job to add the SQL trace (10046):
    SQL> EXECUTE IMMEDIATE 'ALTER SESSION SET EVENTS ''10046 TRACE NAME CONTEXT
    FOREVER,LEVEL 12''';

    Note:

    All quotes are single quotes.

    This creates a SQL trace for the purge session and the scheduled purge jobs (J000) in the database user dump destination.

    SQL> SHOW PARAMETER DUMP
    
  2. Identify the trace for the main database session by searching the trace files for string dbms_scheduler.create_job or delete_instances_in_parallel.
    grep -i dbms_scheduler.create_job *
    

    For example:

    TESTORA2_ora_3893.trc
    

    The number of scheduled jobs depends on the DOP. For example, a DOP of 4:

    • TESTORA2_j000_9585.trc

    • TESTORA2_j001_9587.trc

    • TESTORA2_j002_9604.trc

    • TESTORA2_j003_9606.trc

    If parallel query slaves have been customized into the purge script, then there are trace files (P000) for each. The parallel query trace files are less important because the query plans in the other trace file account for their execution. For example:

    PSOAORA2_p000_4284.trc
    
  3. Execute TKPROF on the trace files and sort by the elapsed time to identify the most costly SQL:
    TKPROF TESTORA2_ORA_3893.TRC ../OUT/ TESTORA2_ORA_3893.OUT WAITS=YES
    SYS=YES AGGREGATE=YES SORT=EXEELA
    

Oracle SOA Suite Table Partitioning

Oracle table partitioning addresses many concerns about the maintenance of large tables by decomposing them into smaller, more manageable segments called partitions. The SOA schema has been instrumented with partition keys to take advantage of the Oracle database range partitioning.

For more information about partitioning, see Partitioning Component Tables.

Referential Integrity and Equipartitioning

For performance reasons, the Oracle SOA Suite service components (excluding Oracle B2B) have no foreign key constraints to police referential integrity. Without these constraints, the relationship between master and detail tables must be protected to avoid dangling references in the detail tables.

Equipartitioning is a database partitioning feature that partitions master and detailed tables based on their foreign key constraint. This feature creates table partitions that group related master and detail rows in the same date range. This grouping ensures that no dangling references are created when partitions are dropped.

To mimic this feature, Oracle SOA Suite is instrumented to push the creation date of the business flow instances down into the component master tables (Oracle BPEL Process Manager, Oracle Mediator, and human workflow). Because the SOA Infrastructure is the top level component on which all other components depend, this method groups all dependent rows by date range.

For example, the timestamp in the CREATED_TIME column of the SCA_FLOW_INSTANCE table is pushed down into the following tables:

  • Oracle BPEL Process Manager master table CUBE_INSTANCE column CPST_INST_CREATED_TIME, which is then pushed down into dependent tables such as CUBE_SCOPE.CI_PARTITON_DATE.

  • Oracle Mediator master table MEDIATOR_INSTANCE column COMPOSITE_CREATION_DATE, which is then pushed down into dependent tables.

Note:

  • Dependent tables must be partitioned with their master and, in all cases, the top level SOA Infrastructure SCA_FLOW_INSTANCE table must be partitioned.

  • To complete equipartitioning, the table partitions must all share the same name and date range.

In summary, equipartitioning means that the associated dependent table rows are in a partition with the same partition key range as their master table rows. Therefore, the state of each detail row in the equipartition can be inferred from its associated master table row.

SOA Composite Application Range-Hash Partitions

Range partitions can have their rows hashed into a subpartition to implement SOA composite application range-hash partitioning. Hash subpartitions can benefit in distributing I/O, but it is not currently recommended when partitioning Oracle SOA Suite tables.

Hashing of keys is a method that works well with equality predicates (=, IN), and there are several tables with keys that are good candidates such as the Oracle BPEL Process Manager CIKEY table. However, range-hash partitions alone do not convey the uniqueness of the hash key. For the cost-based optimizer to identify uniqueness, the query must search all partitions of a table (a full table scan). Partition pruning helps by narrowing the range of partitions to search, but Oracle SOA Suite does not use this feature. (See Partition Pruning.) Therefore, to avoid a full table scan when the cost-based optimizer identifies uniqueness, indexes are searched that bypass the hash subpartition.

Interval Partitioning

Interval partitioning is an extension of range partitioning that instructs the database to automatically create partitions as required.

The verification scripts currently require the partition name to be the same across all tables in a group. The interval partitioning system generates partition names that differ for each partition. Therefore, this is not supported.

Global Hash Indexes

The use of global hash indexes is independent of Oracle SOA Suite table partitioning.

Oracle SOA Suite is constantly inserting rows, and many of the index keys are monotonically increasing. Indexes of a BTREE structure insert these keys and target only a few database blocks that can become very hot across an Oracle Real Application Clusters (Oracle RAC) setup. This is typically seen in the AWR reports as excessive buffer busy waits. To distribute the index keys randomly across many database blocks, use global hash indexes.

Partition Pruning

When the range partition keys are used as predicates, the optimizer can prune the number of partitions to search. Oracle SOA Suite and especially Oracle Enterprise Manager Fusion Middleware Control do not presently use partition keys in SQL queries and therefore do not take advantage of this performance feature.

With the requirements of equipartitioning, the partition keys are populated with the SOA composite application creation date that is not the creation date of the components. The components (Oracle BPEL Process Manager, Oracle Mediator, and human workflow) have their own creation dates that are used in the console queries:

Example for Oracle BPEL Process Manager:

  • Partition key: Table CUBE_INSTANCE column CPST_INST_CREATED_TIME

  • Creation date: Table CUBE_INSTANCE column CREATION_DATE

Purging Partitions

Circumstances may arise that require the partitioned tables to be purged, especially if the row migration scripts are used. This defeats the purpose of partitioning the table for maintenance, which can remove data in bulk through the ALTER TABLE ... DROP PARTITION command.

Attempting to purge the partitioned tables most likely causes the parsing performance problem. If partitions must be purged, then also shrink the partitions. For more information, see Understanding Space Management.

Reducing Audit Levels

This section provides tuning information about how to reduce the audit level for SOA composite applications and the data written to the Oracle SOA Suite schema.

Setting the Audit Level for Production Environments

The default Audit Level is set to Production at the SOA Infrastructure level. To limit database growth, and for optimal performance, Oracle recommends that you turn Off the setting at the SOA Infrastructure level, and instead turn on auditing at the composite level for composites that require debugging or monitoring. Your compliance requirements might also determine the granularity of your audit requirements.

The following image depicts setting the audit level for a composite from the composite home page:

Setting audit level from Composite Home Page.

Note:

You can change the audit level setting at the SOA Infrastructure level using the SOA Infrastructure Common Properties page in Oracle Enterprise Manager Fusion Middleware Control. See Configuring SOA Infrastructure Properties for information about turning off audit at the SOA Infrastructure level.

See Managing the State of an Application from the SOA Composite Application Home Page for more information about setting the audit levels at the composite level.

If you can disable auditing (provided the business requirements allow for it), there are substantial performance improvements. Changing Production to Off can result in improvements (without tuning the individual component audit settings). However, if you cannot disable auditing, Oracle recommends that you consider the following steps to reduce audit levels.

For synchronous BPEL components for which the audit trail is not required, use the following settings:

  • inMemoryOptimization = true

  • completionPersistPolicy = faulted

With these settings, the synchronous process only appears in the audit trail if the instance faults.

For information about setting inMemoryOptimization and completionPersistPolicy, see "Deployment Descriptor Properties" in Developing SOA Applications with Oracle SOA Suite.

Note:

Use these settings for synchronous BPEL processes and not for asynchronous BPEL processes.

Connection and Transaction Timeout Troubleshooting

This section describes how to troubleshoot connection and transaction timeout issues.

Resolving Connection Timeouts

You can receive a connection timeout error under circumstances such as the following:

  • You run a SOA composite application with a large payload that takes more than 30 seconds to process.

  • You are invoking a stress test using a large payload from the Test Web Service page of Oracle Enterprise Manager Fusion Middleware Control.

  • You are passing a large number of message files (one million) into a composite with a file adapter service.

To avoid these timeout errors, increase the transaction timeout property as follows:

  1. Log into Oracle WebLogic Administration Console.
  2. Click JTA.
  3. Change the value of Timeout Seconds (the default is 30).
  4. Click Save.
  5. Restart Oracle WebLogic Server.

Resolving Email Notification Timeouts

You can receive a connection timeout errors when a large number (for example, 200,000) of notification emails are sent.

To avoid these timeout errors:

  1. Increase the JTA Timeout Seconds value to 1200.
  2. Change the EJB timeout to 1800 for NotificationSender, NotificationServiceBean, and TaskNotificationSender EJBs. See Long Running_ Synchronous Calls To Remote Web Services Error Out or Asynchronous Transactions Return with an Error after a Long Time.
  3. Increase the DataSource XA Transaction Timeout value to 1330.
  4. In the database, change the Distributed_lock_timeout parameter value to 1400.

    Note:

    Important point to note here is that the DataSource XA Transaction Timeout has to be greater than the JTA transaction time out and the Distributed_lock_timeout has to be greater than the DataSource XA Transaction timeout value.
  5. Restart Oracle WebLogic Server after making these changes.

    Note:

    You can also increase the timeout values in deployment descriptor file and redploy the deployment descriptor file. For more information, Deployment Descriptor Schema and Document Type Definitions Reference and other deployment related sections in the Developing Enterprise JavaBeans, Version 2.1, for Oracle WebLogic Server guide.

Increasing Database Connection Values

You can receive the error message shown below because of slow connections to the database:

090304)): oracle.toplink.exceptions.DatabaseException 
Internal Exception: java.sql.SQLException: Internal error: Cannot obtain 
XAConnection weblogic.common.resourcepool.ResourceDeadException: Pool 
SOADataSource has been disabled because of hanging connection tests, cannot 
allocate resources to applications.

If this occurs, perform the following steps:

  1. Open the $DOMAIN_HOME/bin/setSOADomainEnv.sh file on Linux or the DOMAIN_HOME\bin\setSOADomainEnv.cmd file on Windows.
  2. Open the DOMAIN_HOME\bin\setSOADomainEnv.cmd file.
  3. Uncomment the lines shown in bold.
    # 8331492: Value of weblogic.resourcepool.max_test_wait_secs is 10 
    # seconds. It can be increased by uncommenting line below if your database 
    # connections are slow. See SOA documentation for more details. 
     EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
    -Dweblogic.resourcepool.max_test_wait_secs=30" 
    export EXTRA_JAVA_PROPERTIES 
    
  4. Save your changes and restart the managed Oracle WebLogic Server.

Updating the EJB Transaction Timeout Value in the Deployment Archive After SOA Infrastructure Failure

Updating the transaction timeout value for the FacadeFinderBean property in Oracle WebLogic Server Administration Console under Deployments > expanded SOA Infrastructure Application > FacadeFinderBean > Configuration tab can result in the error shown below after restarting the SOA Infrastructure:

java.lang.IllegalArgumentException: Cannot convert value of type [$Proxy223
implementing
 oracle.bpel.services.workflow.verification.IVerificationService,org.springframe
work.aop.SpringProxy,org.springframework.aop.framework.Advised]
to required type
[oracle.bpel.services.workflow.verification.IVerificationService] for
property 'verificationService': no matching editors or conversion strategy found
Message icon - Warning Errors were encountered while performing this operation.

The SOA Infrastructure status is also displayed as failed.

This error is not specific to FacadeFinderBean; it also applies to any EJB that is part of the SOA Infrastructure application.

To resolve this error, you must manually modify the transaction timeout setting in your deployment archive.

To update the transaction timeout setting:

  1. Open the fabric-ejb.jar file in your deployment archive.
  2. Increase the transaction timeout value in the META-INF/weblogic-ejb-jar.xml file to a larger value.
  3. Rejar the file.
  4. Restart the managed server that includes the SOA Infrastructure by following the instructions in Stopping and Starting the Managed Server and SOA Infrastructure .

    Note:

    This issue may also occur while updating any EJBs deployed as part of the SOA Infrastructure application. If this issue occurs, you must update the corresponding contained JAR file for those EJBs in a similar fashion.

Long Running, Synchronous Calls To Remote Web Services Error Out or Asynchronous Transactions Return with an Error after a Long Time

You may have long running, synchronous calls to remote web services that end with JTA transaction rolled-back errors. When executing a transaction making an asynchronous call (for example, to the SOA server), the application may return with an error. The server logs show JTA transaction timeouts, which can cause this behavior.

To check the JTA transaction timeout in the Oracle WebLogic Server Administration Console.

  1. Log in to Oracle WebLogic Server Administration Console.

  2. In the Domain Structure, select Services > JTA to check the timeout value.

    If the transaction is always timing out beyond 30 seconds and is a custom composite synchronous client invocation, then you may need to revisit the design approach. It may be best for the external web service to be invoked as an asynchronous transaction.

    Increasing the JTA for supporting long running, synchronous transactions is simply an interim mechanism. For information on changing the JTA transaction timeout setting, see Resolving Connection Timeouts.

  3. If synchronous client invocations take a long time, check for any performance issues with the system and try to resolve them.

  4. Check for appropriate values for the syncMaxWaitTime property and BPEL's EJB transaction timeout settings in relation to the JTA timeout settings and only then increase the value of the JTA timeout, if needed.

    For information on viewing and changing the syncMaxWaitTime property, see Configuring BPEL Process Service Engine Properties.

  5. To view and change the BPEL EJB transaction timeout settings, perform the following steps:

    1. Log in to Oracle WebLogic Server Administration Console.

    2. In the Domain Structure, click Deployments.

    3. Expand soa-infra > EJBs.

    4. Update the following EJBs:

      - BPELActivityManagerBean

      - BPELDeliveryBean

      - BPELDispatcherBean

      - BPELEngineBean

      - BPELFinderBean

      - BPELInstanceManagerBean

      - BPELProcessManagerBean

      - BPELSensorValuesBean

      - BPELServerManagerBean

    5. Click Save.

    6. Restart Oracle WebLogic Server.

  6. For asynchronous transactions, check the values for both the BPEL EJB transaction timeout and the JTA transaction timeout and adjust, as needed.

Increasing the HTTP POST Timeout Value to Resolve Broken Pipe Errors

If you receive the broken pipe error as shown below, increase the HTTP POST timeout value in Oracle WebLogic Server Administration Console.

<BEA-000000> <got FabricInvocationException, Cikey=73410002, FlowId=10001, Current
 Activity Key=73460002-BpInv0-BpSeq0.3-3, Current Activity Label=Invoke_1,
 ComponentDN=default/TestClientWSMEDWSBPELFAProj!2.0*soa_
b2e412f3-1167-4a9a-bbe0-73795f0743d4/BPELProcess1
java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)

To increase the HTTP POST timeout value:

  1. Log in to Oracle WebLogic Server Administration Console.
  2. In the Domain Structure, expand Environment > Servers.
  3. In the Name column, click the SOA server.
  4. At the top of the page, click the Protocols tab.
  5. In the Post Timeout field, increase the value to a maximum of 120.

Resolving Exception Errors When Processing Large Documents

If you receive the error shown below when processing large attachments, increase EJB property timeout values.

java.sql.SQLException: Unexpected exception while enlisting XAConnection
java.sql.SQLException: Transaction rolled back: Transaction timed out after 501
 seconds

To increase EJB property timeout values:

  1. Log in to Oracle WebLogic Server Administration Console.
  2. In the Domain Structure, click Deployments.
  3. Expand soa-infra > EJBs.
  4. At the bottom of the page, expand the EJBs section.
  5. Increase the time out value:
    • BpelEngineBean

    • BpelDeliveryBean

    • CompositeMetaDataServiceBean

Runtime Diagnostics Troubleshooting

This section describes how to troubleshoot runtime issues.

Unavailability of Work Manager Threads for Incoming Processing

The exception message shown below occurs because there are a fixed number of work manager threads allocated for incoming processing, and Oracle SOA Suite has reached the thread limit. Oracle SOA Suite gracefully backs out of processing to prevent a resource bottleneck that may eventually lead to a stuck thread scenario. If you receive this error, you must resubmit your request.

<Mar 11, 2014 2:10:16 PM PDT> <Error> <HTTP> <BEA-101017>
<[ServletContext@207971169[app:soa-infra module:/soa-infra path:null
spec-version:3.0]] Root cause of ServletException.
java.lang.Exception: Unable to acquire the workmanager thread resource in
order to process requests directed at compsoite
default/POProcessing!1.0*soa_7a9c6320-c26e-4df2-b29f-7b80782d3693
        at
oracle.integration.platform.blocks.deploy.CompositeLazyLoader.load(CompositeLa
zyLoader.java:112)
        at 
. . .
. . .

Oracle SOA Suite Runtime Failure with a "Cannot read WSDL" Error

The following error is displayed if either the endpoint is not available for a reference or a composite is deployed with an incorrect deployment plan file.

oracle.fabric.common.FabricException: Cannot read WSDL

To diagnose this issue:

  1. Ensure that the endpoint for the reference is up and running.

    The WSDL or endpoint is stated in the error. The service can then be looked up from Oracle Enterprise Manager Fusion Middleware Control to check if it is active. Once the service is active and the endpoint is reachable, search for the SOA instance and retry it through the Recovery tab of the BPEL process service engine in Oracle Enterprise Manager Fusion Middleware Control.

  2. For extensions/customizations, ensure that the correct URL is updated in the deployment configuration plan.

    For information about deployment plans, including examples of using sca_extractPlan to extract plans, see the Section "Customizing Your Application for the Target Environment Before Deployment" of Developing SOA Applications with Oracle SOA Suite.

Automatic Recovery of BPEL Instances is Not Recovering a Specific Instance

BPEL processes have an automatic recovery feature that attempts to automatically recover activities that are recoverable such as unresolved invoke and callback messages, activities not completed over a provided threshold time, and so on. However, the automatic recovery feature only tries to recover a few instances and only retries a fixed number of times. If some instances are not being automatically recovered, they are likely not being picked up because of the configuration of the automatic recovery parameters.

To resolve this issue:

  • Set the maximum number of messages to automatically recover.

    By default, the automatic recovery feature of Oracle BPEL Process Manager processes 50 messages to submit for each recovery attempt. This is controlled by the maxMessageRaiseSize property.

    1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

    2. Select SOA Administration > BPEL Properties > More BPEL Configuration Properties > RecoveryConfig.

    3. Expand both RecurringScheduleConfig > maxMessageRaiseSize and StartupScheduleConfig > maxMessageRaiseSize.

      The default value is 50 for each. A negative value causes all messages selected from the database to be submitted for recovery. A value of 0 causes no messages to be selected from the database (effectively disabling recovery). To recover more than 50 messages, set the property value to that value. Use this property to limit the impact of recovery on the server.

  • Set the maximum number of automatic recovery attempts on a given message.

    You can also configure the number of automatic recovery attempts to submit in the same recoverable instance. The value you provide specifies the maximum number of times that invoke and callback messages are recovered. If the value is 0 (the default value), it recovers all messages. Once the number of recovery attempts on a message exceeds the specified value, a message is marked as nonrecoverable.

    To configure automatic recovery attempts for invoke and callback messages in Oracle Enterprise Manager Fusion Middleware Control:

    1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

    2. Select SOA Administration > BPEL Properties > More BPEL Configuration Properties.

    3. Select MaxRecoverAttempt, and enter a value in the Value field.

    4. Click Apply.

      It may not be desirable in all cases to use automatic recovery. If services are not idempotent, then corruption can occur. Moreover, the automatic recovery restores the composite to the last save point that can be immediately after an asynchronous invoke, wait, and so on. Therefore, it is important to understand the process behavior and what it does next before performing mass recoveries. Attempt mass automatic recovery only after the root cause of the composite failures is fixed (for example, a service that was unavailable is now available, a database running out of space was fixed, and so on). Automatic recovery can also trigger an unexpected load during failure scenarios. This causes more threads to block on a remote server that can induce hangs in the SOA server in a cascading fashion.

      For more information, see Configuring Automatic Recovery for Oracle BPEL Process Manager.

Some Composites Are Retried Multiple Times on Failure

When a BPEL process flow errors out, it is retried with all its invocations. This is undesirable in some cases.

The property GlobalTxMaxRetry (default value is 3) specifies how many retries are performed if an error is identified as retriable. For example, after several web service invocations, if dehydration fails due to a data source error, then this is identified as a retriable error and all activities from the prior dehydration state are retried. If the activities being retried are not idempotent (that is, their state can change with each retry and is not guaranteed to give the same behavior), then multiple retries can be problematic.

To rectify this situation, customize the composite by specifically marking the nonidempotent activities with idempotent set to false in the partner link settings section of the composite.xml file to prevent retries.

<property name="bpel.partnerLink.partner_link_name.idempotent">false</property>

You cannot set the idempotent property in Oracle Enterprise Manager Fusion Middleware Control.

You can also set GlobalTxMaxRetry to 0 in the Systems MBean Browser.

To set GlobalTxMaxRetry:

  1. Right-click soa-infra (SOA_cluster_name).
  2. Select SOA Administration > Common Properties.
  3. Click More SOA Infra Advanced Configuration Properties.
  4. Click GlobalTxMaxRetry.
  5. In the Value field, enter an appropriate value.
  6. Click Apply.

For more information about the idempotent property, see "Managing Idempotence at the Partner Link Operation Level" section of Developing SOA Applications with Oracle SOA Suite.

Application Transaction Does Not Complete and the Underlying Composite is Stuck in a Running State

Assume an application transaction is not completing. For example, a purchase order status may remain processing. Checking the business flow instance shows that the composite is stuck in the running state. In this case, the component is probably not running. Instead, it has likely faulted and may need recovery.

Faults may occur for various reasons:

  • A BPEL activity faulted with an error (for example, a business error, security authorization error, or some other error).

  • A BPEL activity invoked an external web service that was unavailable.

  • A BPEL activity has already been terminated by the administrator using Oracle Enterprise Manager Fusion Middleware Control.

  • A BPEL activity invoked an asynchronous ADF service and the message is stuck in the AQ JMS queue.

  • A BPEL activity invoked an asynchronous ADF service, but because Oracle SOA Suite was unavailable, the callback message did not arrive.

  • A BPEL activity invoked a synchronous ADF service, which is taking a long time (or is hanging).

  • A network error occurred.

To diagnose this issue:

  1. Log in to Oracle Enterprise Manager Fusion Middleware Control.

  2. In the navigator pane, go to domain_name > SOA.

  3. Click soa-infra (SOA_cluster_name).

  4. Click the Flow Instances tab.

  5. Search for the business flow instance, and click the flow ID.

    The Flow Trace page appears.

    If the instance is not visible (and the Audit Level is not set to Off on the SOA Infrastructure Common Properties page), this implies that the message is stuck outside of Oracle SOA Suite.

    If the message has reached Oracle Mediator, but not instantiated the BPEL flow, the BPEL instance may have been rolled back from the start due to an error.

    If the BPEL flow exists, the Faults section of the Flow Trace page typically shows the faulted service that can trace the root cause.

  6. In the Trace section, click the BPEL process.

  7. Expand the BPEL audit trail to see the exact point at which the service faulted.

    This information is also available in the Faults tab of the BPEL flow trace. It also indicates whether the fault can be recovered.

  8. Click the View Raw XML link.

    The same information is also available through this link, where you can see the error. For example:

    ...
    <message>Faulted while invoking operation "modifyUserRoles" on provider
     "UserService". </message><details>
    ...
    <tns:message>JBO-27023: Failed to validate all rows in a
     transaction.</tns:message>
    <tns:severity>SEVERITY_ERROR</tns:severity>
    ...
    <message>The transaction was rolled back. The work performed for bpel instance
     "451042" was rolled back to the previous dehydration point, but the audit
     trail has been saved.
    You can recover the instance from the recovery console by resubmitting the
     callback message or activity for execution.</message
    

    Since the instance was rolled back to its previous dehydration point, the status remains as Running.

  9. In the Audit Trail tab, make a note of the following:

    • Composite name (for example, POComposite)

    • Component (for example, UpdateGuid BPEL process)

    • BPEL instance ID (for example, bpel:451042)

    This is all used in the recovery of the instance, if it is recoverable. The Audit Trail may mark the error as a nonrecoverable business fault, but the recoverability of the message can be found in the Recovery tab of the BPEL process service engine.

  10. To attempt to recover the instance, right-click soa-infra (SOA_cluster_name), and select Service Engines > BPEL.

  11. Click the Recovery tab.

  12. From the Type list, select Activity.

  13. Specify the composite and component names captured in step 9, and click Search.

  14. Find the specific BPEL instance ID. You can recover faults marked as Recoverable.

  15. Check the other recovery options in the Type list (for example, Invoke and Callback), if they exist.

    Note:

    You can also search for recoverable messages from the Flow Instances page of the SOA Infrastructure.

  16. If the instance is not marked as recoverable, then reinvoking the service is not allowed (most probably because it is not idempotent). In some cases, you may need to provide diagnostic information to Oracle Support Services to resolve issues with nonrecoverable, nonidempotent transactions.

  17. If the BPEL activity has invoked an asynchronous ADF service and the message is stuck in the AQ JMS queue, you can view the server.log and server-diagnostic.log files to see the logging of the message metadata logged by the JRF web services infrastructure.

    In addition, the ADF diagnostic logs are also available to debug, if needed.

    1. Use the ECID field to correlate and track ADF service logging corresponding with the SOA composite application that invoked it.

    2. When viewing the log in Oracle Enterprise Manager Fusion Middleware Control, click the Broaden Target Scope dropdown list and select the farm_name/domain_name (Oracle WebLogic domain) to view messages across the domain.

    3. In the Selected Targets section of the Log Messages page for the Oracle WebLogic Server domain, ensure that the search includes the ECID field with the value noted in step 17.a.

    4. Search and view log records for the execution content ID (ECID) and note any issues.

      For a specific ECID, you find several root instances (top level clients). You must drill down to the appropriate instance to find a specific fault.

    5. Observe if the components completed successfully or completed with an error. See the "Viewing and Searching Log Files" section in the Administering Oracle Fusion Middleware.

See the following chapters for the various ways to recover transactions and messages:

Errors During Analytics Measurement Event Processing

When errors occur during processing of BPMN, Human Workflow, CASE analytics measurement events for any reason like target BAM is down or the database is not available and so on, (i) analytics message processing related errors are seen in the logs and (ii) BPM_MEASUREMENT_ACTION_EXCEPS table also has one or more records of these errors. You can republish these events by using any of the two methods.

Using Measurement Action Service client

Use republishErroredEvents operation to republish errored measurement events of the past 24 hours. This operation is available as part of AnalyticsConfig mbean To republish the error events, perform the following activities. Enter password when prompted..

java    [CLASSPATH]   [PROPERTIES]  oracle.bpm.metrics.action.MeasurementActionRecoveryClient
CLASSPATH:
-classpath $MW_HOME/soa/soa/modules/oracle.bpm.runtime_<version number of instance>/oracle.bpm.analytics.metrics.interface.jar:
           $MW_HOME/soa/wlserver/server/lib/weblogic.jar

Enter the following required properties.

-Dhost=<host> (soa server host name)
-Dport=<port> (soa server port)
-Duser=<admin user> (soa server admin user name)
-DcompositeName=<soa composite name> (soa server host name)

Enter the following optional properties depending on your need.

-Drealm = <hostrealm Name>  (default is  jazn.com)
-Dprotocol=<protocol>  (jndi protocol. Default is t3)
-DinitialContextFactory = <jndi initial context factory> (Default is weblogic.jndi.WLInitialContextFactory)
-DmeasurementAction=<action name>  (measurement action name.  Default is ProcessMetrics (BAM 12c))
-DrangeStartDate=mm/dd/yyyy  (Range start date . Default is 24 hours before rangeEndDate )
-DrangeEndDate=mm/dd/yyyy (Range end date. Default is current time)
-Ddebug=true/false (if true, prints the exception stacktraces in case of any error. default is false)

Using Oracle Enterprise Manager AnalyticsConfig MBean - republishErroredEvents operation

republishErroredEvents operation is also available as part of AnalyticsConfig mbean (mbean object name = oracle.as.soainfra.config:name=analytics,type=AnalyticsConfig,Application=soa-infra). Oracle Enterprise Manager administrator can access and republsih errored messages of the last 24 hours.

  1. Log into Oracle Enterprise Manager

  2. Expand oracle.as.soainfra.config, Server:AdminServer, AnalyticsConfig and select analytics.

  3. Open Operations tab in the right pane.

  4. Click on republishErroredEvents.

  5. Provide the necessary details as described above.

  6. Click Invoke.

Human Workflow Troubleshooting

This section describes how to troubleshoot human workflow issues.

Unavailability of Human Workflow Service Engine

During SOA server startup, you can receive the exception shown below. This exception occurs because the SOA server is still initializing. Wait for SOA server initialization to complete.

ORABPEL-30132

exception.code:30132
exception.type: ERROR
exception.severity: 2
exception.name: Workflow Service Engine not ready. SOA server is still
starting.
exception.description: Could not obtain instance of Workflow Service Engine
as the SOA server is still initializing.
exception.fix: Wait for SOA server to finish starting up, and retry.

For information about monitoring SOA server initialization status in Oracle Enterprise Manager Fusion Middleware Control, see Viewing the Overall Runtime Health of the SOA Infrastructure.

Task Assignment/Routing/Escalation Issues

Table C-2 describes symptoms, possible causes, and possible solutions for task assignment/routing/escalation issues.

Table C-2 Troubleshooting Task Assignment/Routing/Escalation Issues

Symptom Possible Cause Possible Solution

The task completes without any assignment occurring.

The most common problem is that task assignees are specified using XPath expressions, and the expression does not evaluate to any nodes. Other problems can include incorrect skip conditions for participants.

  1. Correct any issues with the XPath expressions.

  2. Ensure that you get some results for the XPath expression for the given data.

  3. Check the skip conditions specified with the Specify skip rule checkbox for task participants in the Human Task Editor.

The business rules do not return any list builders.

When participants of a task are specified using business rules, it is expected that business rules return at least one list builder. If business rules determine that no participants are needed, the function ignoreParticipant(...) must be used. If modeled correctly and you still see this error, it is likely that none of the rules fired. See the symptom entitled '"A human workflow task chooses the incorrect user if many rules are defined or it errors with the following message:" in this table.

  1. Use the ignoreParticipant(..) function to model your rules.

  2. Ensure the rules are modeled correctly so that at least one rule is fired.

The business rules return list builders of different types.

When participants in a task are specified using business rules, it is expected that business rules return list builders of the same type.

Correct your rules.

A human workflow task chooses the incorrect user if many rules are defined or it errors with the following message:

Ruleset returned lists with
different list builder

At runtime, when a human workflow task tries to fetch the list of users, it may error out with the following error:

Ruleset returned lists with different list builder

This error is displayed in the Task Detail comments field. Alternately, the task may select a user or approver, which may not appear to be the correct or expected one. This is primarily caused by having overlapping rules. When the participants of a task are specified using business rules, it is expected that business rules return list builders of the same type.

Moreover, only one rule from a ruleset must be applicable for a transaction. In case many rules are true, the actions associated with the applicable rule with the highest priority get executed. In case multiple applicable rules have the same priority, then the first rule in the list is picked and its actions executed.

Avoid writing overlapping rules. Constraints from different list builders are different and cannot be mixed. If multiple rules get triggered with a different list builder, this error occurs. In addition, only one set of constraints is honored.

Check that all rules in the ruleset have priorities defined so that multiple rules with the same priority are not applicable for the same transaction.

For more details., see Developing Business Processes with Oracle Business Process Management Studio.

Parallel assignees have to approve or reject the task even though the parallel completion criteria is met.

In the Add Participant Type dialog for a parallel participant, you can configure the human task during runtime to wait for all parallel participants to complete or to complete when criteria are met.

Make the correct selection for completion in the Add Participant Type dialog.

The task is assigned to the group/role when the expectation is that it goes to every user in the group/role individually.

When a group or a role is used as a task assignee, the task is assigned to the group or role directly. Task runtime does not assign it separately. One of the users in the group/role has to claim the task and work on it. When used with a parallel or serial participant, often times it is expected that this resolution to users is automatic, which it is not.

To assign separately to the members of the group or role, use the XPath functions ids:getUsersInGroup and ids:getUsersInAppRole.

A task errors out when invoking the decision service for the evaluation of routing rules or rule-based participants.

Payload validation is enabled on the SOA Infrastructure instance.

Deselect the Payload Validation checkbox for the instance. For more information, see Configuring SOA Infrastructure Properties.

Task Action Issues

Table C-3 describes symptoms, possible causes, and possible solutions for task action issues.

Table C-3 Troubleshooting Task Action Issues

Symptom Possible Cause Possible Solution

A user is not allowed to perform an action on a task.

The most common problem is that the user does not have permissions to perform that action on that task at that point in time.

Find out if the user can be an assignee, owner, or creator of the task, or if they are an administrator. If the user should have been allowed to perform the action, check the server log file for a detailed log message, which includes information such as the task state, task assignees, user who acquired it, permitted actions, roles played by this user for the given task, and so on.

A task is locked and cannot be updated in Oracle BPM Worklist. An ORA-30279 error is displayed (with an underlying ORA SQL exception of ORA-20001 also displayed).

Two users are trying to update the same task concurrently.

Wait for the other user to finish.

Notification Issues

Table C-4 describes symptoms, possible causes, and possible solutions for notification issues.

Table C-4 Troubleshooting Notifications Issues

Symptom Possible Cause Possible Solution

You may receive the following error message:

[2012-07-05T09:13:05.246-04:
00] [soa_server1] [ERROR] []
[oracle.soa.services.workflow
.common] [tid:
[ACTIVE].ExecuteThread: '6'
for queue:
'weblogic.kernel.Default
(self-tuning)'] [userId:
<anonymous>] [ecid:
dc2ff0568380b04f:23824bc7:13
8570a4e67:-8000-000000000000
1716,0] [APP: soa-infra] <.>
[[ORABPEL-0
at
oracle.bpel.services.workflo
w.task.notification.TaskNoti
fications.notifyForTask(Task
Notifications.java:481)
. . .
. . .

Check if you have apostrophes in the body of an email notification message.

<%string('À titre de
responsable fonctionnel, une
demande d'approbation 
. . .

Escape each apostrophe by adding double apostrophes in your string. For example, if you have 'd in your message, escape it as follows:

<%string('d'd')%>

to:

<%string('d''d')%>

The email notification is not being sent out.

Incorrect outgoing (SMTP) server settings are used in the email driver configuration.

Check the Simple Mail Transfer Protocol (SMTP) port/SMTP host/user name/password/email values.

Tip: Validate the values by using them in any email client for connecting to the SMTP server.

Perform the following steps to verify the settings in Oracle Enterprise Manager Fusion Middleware Control:

  1. In the navigator, right-click User Messaging Service.

  2. Select usermessagingdriver-email > Email Driver Properties.

  3. Check the settings, in particular the settings for these properties:

    - OutgoingMailServer

    - OutgoingMailServerPort

The task email notification is not being sent out.

Notification Mode is set to NONE on the Workflow Notification Properties page in Oracle Enterprise Manager Fusion Middleware Control.

Change this setting to Email or All. For information, see Configuring Human Workflow Notification Properties.

Notifications are not being sent because of the following error message:

No matching drivers found for
sender address = <address>

The User Messaging Service driver for the appropriate channel is configured with a specific list of sender addresses, and the message sent by the application has set a nonmatching sender address.

Note: The User Messaging Service server matches the outbound message's sender address, if set, against the available drivers' sender addresses to find a matching driver to use for delivering the message. If a driver has set one or more sender addresses, then the User Messaging Service server only sends messages with the matching sender address to it.

Perform the following tasks:

  • Check the following settings for the appropriate driver in Oracle Enterprise Manager Fusion Middleware Control:

    SenderAddresses

    SenderAddresses format is a comma-delimited list of DeliveryType:Address. For example:

    EMAIL:sender@example.com,

    EMAIL:sender@example2.com

  • Leave this property blank if you want this driver to service outbound messages for all sender addresses for this channel (delivery type).

  • If there are multiple driver instances deployed for the same channel (delivery type) with different configurations, use SenderAddresses to differentiate the driver instances. For example, set one instance with a value in SenderAddresses to only service outbound messages with that matching sender address. The other instance can keep the SenderAddresses blank to service all outbound messages that do not specify any sender address or one that does not match the first driver instance.

  • SenderAddresses that are configured with incorrect syntax (such as missing DeliveryType:) are ignored by the User Messaging Service server for driver selection.

The notifications are sent, but are not actionable.

The Actionable Address field is not configured on the Workflow Notification Properties page in Oracle Enterprise Manager Fusion Middleware Control.

In Oracle Enterprise Manager Fusion Middleware Control, configure the Actionable Address field with a valid email address. For information, see Configuring Human Workflow Notification Properties.

Ensure that the same email address is used when configuring the incoming server setting in the User Messaging Service email driver. For information, see Configuring Human Workflow Notification Properties.

Notifications are sent, but are not actionable.

The human workflow task is not set to send actionable notifications.

In the Human Task Editor (you can double click the .task file in Oracle JDeveloper to start the editor), expand the Notification section, click the Advanced tab, and select the Make notification actionable checkbox.

Actionable notifications are sent, but no action is taken after responding.

The Actionable Address field is incorrect.

Check the IMAP/POP3 server/port values. Ensure the Actionable Address field is used in the email driver configuration.

Tip: Validate the values by using them in any email client for connecting to the IMAP/POP3 server.

Actionable notifications are sent, but no action is taken after responding.

The nondefault email client is configured for receiving notifications.

When the user clicks the approval link, the default mail client page opens, which may send emails to a different email server. Configure the default email client to receive actionable notifications.

Enter the correct value in the Actionable Email Account field of the Workflow Task Service Properties page as the incoming, actionable email account to use. The default account name is Default.

For information, see Configuring Human Workflow Task Service Properties.

Human workflow actionable notifications are sent, but no action is taken after responding.

Incoming mail server settings for the email driver are incorrect.

Check the following email driver settings in Oracle Enterprise Manager Fusion Middleware Control:

  • MailAccessProtocol (IMAP or POP3 in uppercase)

  • ReceiveFolder

  • IncomingMailServer

  • IncomingMailServerPort

  • IncomingMailServerSSL

  • IncomingUserIDs

  • IncomingUserPasswords

  • ImapAuthPlainDisable

Actionable notifications are sent but no action is taken after responding.

An email client is configured with the same account used in the email driver.

The mail may be downloaded and marked as read or deleted by the email client before the human workflow notification service can download and process the mail. Remove that account from the email client.

User notifications or human workflow notifications are sent through the correct delivery type (email, SMS, and so on), but to the wrong address.

A self-provisioned messaging channel was created by the user in User Messaging Preferences for use in a BPEL process user notification or human workflow notification.

Note: The User Messaging Preferences user interface enables you to create your own messaging channel for various use cases. However, do not use these channels for BPEL process user notification and human workflow.

Do not use a self-provisioned messaging channel for BPEL process user notification and human workflow use cases (that is, do not set as default channel and do not use in a messaging filter for such use cases). BPEL process user notification and human workflow use User Messaging Preferences only for the delivery type preference. The the actual address is retrieved from the user profile in the identity management system.

Note: Addresses from the user profile in the identity management system are available through User Messaging Preferences using predefined channel names, such as business email, business mobile, business phone, and instant messaging. Use these predefined messaging channels instead for bpel process user notification and human workflow use cases.

The Oracle BPM Worklist link appears in email notifications.

This is the default behavior. By default, email notifications point to Oracle BPM Worklist.

Perform the following steps:

  1. In the Notification section of the Human Task Editor, click the Advanced tab.

  2. Deselect the Show worklist URL in notifications checkbox.

Performance is slow for group notifications.

The group notification performance depends on the number of members in the group (size of group).

  1. Provide an email ID for the group in LDAP. In this case, human workflow sends one email to the group email ID, instead of individual emails to each group member.

  2. In the Notification section of the Human Task Editor, click the Advanced tab.

  3. If you want to send one email to all group members, select Send one email containing all user addresses from the Group notification configuration list. This action enables all members to see the to list and common content is sent to all members (without considering locale, and so on).

  4. If you want to send individual emails, but reuse content between members, select Send individual emails (the default selection) from the Group notification configuration list and unselect Use separate task forms based on locale. This action enables group members to receive individual mails in their locale. Task forms generated for creating notification content are reused between members in the same locale.

Task View Issues

Table C-5 describes symptoms, possible causes, and possible solutions for task view issues.

Table C-5 Troubleshooting Task View Issues

Symptom Possible Cause Possible Solution

Showing custom (mapped attribute) columns in a view.

Attribute mappings are created for specific task types. The view must be associated with one or more task types to use mapped attributes.

In Oracle BPM Worklist (view/create/edit UI), specify a task type for the view in the Definition tab. The attribute labels used in the mappings for that task type are now available as columns that can be used in the view in the Display tab.

It is possible to associate a view with multiple task types. Multiple task types can be selected from the Task Type browser. If multiple task types are selected, then the attribute labels for all those task types are available for use in the view.

View grantees can view and edit tasks belonging to the view owner.

The view is shared as data. This type of sharing allows grantees to use the view as if they are the view owner, and can see and act on the view owner's task.

In the Definition tab of Oracle BPM Worklist (view/edit UI), ensure that Share View is set to Definition only, which enables grantees to use the view against their own tasks. Setting Share View to Data enables grantees to use the view against the view owner's data.

Creating a new standard view.

Only users with administration privileges can create standard views.

  1. Ensure the logged-in user has administration privileges.

  2. Define the view as usual, using Oracle BPM Worklist (create/edit UI).

  3. Check the add to standard views checkbox.

    The view is created as a standard view.

Internationalizing a standard view name.

The value specified in the name field for standard views can be used as a resource key to look up a display name from the WorkflowLabels resource bundle.

Add a new resource key to the WorkflowLabels resource bundle. The key is the name you used for the view, prefixed by STD_VIEW. (Note the required trailing period.)

Migrating views and standard views you have created on one instance to another SOA server.

You must use the test-to-production utility.

The test-to-production utility enables you to export user views and standard views as an XML file, and to import the views from the XML file into another instance. For information about this utility, see Moving Human Workflow Data from a Test to a Production Environment.

Task Attribute Mapping Issues

Table C-6 describes symptoms, possible causes, and possible solutions for task attribute mapping issues.

Table C-6 Troubleshooting Task Attribute Mapping Issues

Symptom Possible Cause Possible Solution

No payload attributes are available for mapping to a public attribute in Oracle BPM Worklist.

Oracle BPM Worklist only supports creation of mappings to simple payload attributes. Only simple attributes from the task payload are displayed for creating mappings in Oracle BPM Worklist.

  • Add simple attributes to the payload.

  • Create protected mappings at design time. Mappings can be made to be complex payload attributes using XPath expressions for protected attribute mappings at design time in Oracle JDeveloper.

  • Use the RuntimeConfigService API to create public mappings using XPath expressions.

You cannot create mappings for the protected attribute label in Oracle BPM Worklist.

Protected mappings can only be created as part of the task definition at design time. Protected mappings cannot be created or updated at runtime using Oracle BPM Worklist, or the RuntimeConfigService API.

  • Use a public attribute label for the mapping.

  • Create the mapping to the protected attribute label in the task definition at design time.

You cannot see any attribute labels for which to create mappings in Oracle JDeveloper.

Design-time mappings can only be created for protected attribute labels. Ensure that protected attribute labels have been created in the SOA instance to which you are connected.

  1. Log in to Oracle BPM Worklist as an administrator.

  2. Go to Administration > Protected Flexfields.

  3. Check that protected attribute labels exist.

  4. Create any protected attribute labels as required.

Internationalizing the name of an attribute label.

You can use the attribute label name as a resource key to look up a display name from the WorkflowLabels resource bundle.

Add a new resource key to the WorkflowLabels resource bundle. The key is the name you used for the label, prefixed by FLEX_LABEL. (note the trailing period).

Migrating attribute labels and mappings from one server to another.

Use the test-to-production utility.

The test-to-production utility enables you to export public attribute labels, public attribute mappings, and protected attribute labels as an XML file, and to import the labels and mappings from the XML file into another instance.

For more information, see Moving Human Workflow Data from a Test to a Production Environment.

Task Report Issues

Table C-7 describes symptoms, possible causes, and possible solutions for task report issues.

Table C-7 Troubleshooting Task Report Issues

Symptom Possible Cause Possible Solution

You receive the following error:

Null Pointer Exception when
running Task Productivity
Report

This is caused by an issue with the handling of dates when the worklist client locale and server default locale are different.

The workaround is to change the locale for the worklist client to be the same as the server, or to run a report without specifying dates.

Task History Issues

Table C-8 describes symptoms, possible causes, and possible solutions for task history issues.

Table C-8 Troubleshooting Task History Issues

Symptom Possible Cause Possible Solution

The Add Participant button is disabled.

A current or past participant is selected in the history table.

This is designed behavior. Adding adhoc participants is not allowed with the current or past participant. The current participant means the task is with that participant at that point in time.

All the added adhoc participants disappeared after a page refresh.

You may not have saved your modifications to the history table.

Ensure that you save your changes. Otherwise, all changes disappear. If you think you have saved your changes and the changes still disappear, file a bug.

Do not see future approvers in the history table.

The Future Approvers checkbox may not be selected.

Select the Future Approvers checkbox in Oracle BPM Worklist (configuration in the task sequence table).

You see the message in the history table about the correlation ID not being passed or any exception related to the correlation ID.

If the task is uninitiated, the correlation ID may not have been passed.

Ensure that you pass the correlation ID to the uninitiated task.

The edit toolbar is disabled or is not displayed.

The user may not have privileges to edit the participants.

  1. In the Assignment tab of the Human Task Editor, click the Task will go from starting to final participant icon in the upper right corner.

    The Configure Assignment dialog is displayed.

  2. Select the Allow participants to edit new participants and Allow initiator to add participants checkboxes.

You receive the following error:

<Warning>
<oracle.adf.controller.intern
al.metadata.MetadataService>
 <BEA-000000><ADFc:
 /META-INF/adfc-config.xml: > 
  <Warning>
<oracle.adf.controller.intern
al.metadata.MetadataService><
ADFC-52024> <ADFc: Duplicate
 managed bean definition for
 'aleCompBindings' detected.> 

Shared library oracle.soa.worklist.webapp is referenced in weblogic.xml and the JAR files adflibWorklistComponents.jar and adflibTasklistTaskflow.jar are packaged in the web application.

These JARs ideally should not be packaged inside the web application. They should only be referenced as a shared library. Do not package these JARs in the web application.

You receive the following error:

<Error> <Deployer>
 <BEA-149265> <Failure
 occurred in the execution of
 deployment request with ID
 '1297964056778' for task
 '3'. Error is:
 'weblogic.management.Deployme
ntException:
[J2EE:160149]Error while
processing library
references. Unresolved
application library
references, defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false].'
weblogic.management.Deploymen
tException:
[J2EE:160149]Error while
processing library
references. Unresolved
application library
references, defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false].

Shared library oracle.soa.workflow.wc is referenced in weblogic.xml, but not available on the server.

Ensure that this shared library is deployed on the server to which you are deploying your application. It may happen that the shared library is deployed, but not targeted, for that server.

You receive the following error:

java.lang.IllegalStateExcepti
on: Attempt to validate an
already invalid RegionSite: 

This is a generic exception that sometimes is displayed in the server logs (for example, AdminServer.log).

See the real exception in the diagnostic logs (for example, AdminServer-diagnostic.log) and provide that exception with the bug you file.

You receive the following error:

[AdminServer] [NOTIFICATION]
 [J2EE JSP-00008]
 [oracle.j2ee.jsp] [tid:
 [ACTIVE].ExecuteThread: '15'
 for queue:
 'weblogic.kernel.Default
 (self-tuning)'] [userId:
 weblogic] [ecid:
17011f2a001d6b0e:7e22d6ce:12e
3444eb1b:-8000-0000000000002f
0a,0] [APP: FederatedApp_
application1] unable to
 dispatch JSP page: The
 following exception
 occurred:.[[
java.lang.RuntimeException:
 Cannot find FacesContext
        at
javax.faces.webapp.UIComponen
tClassicTagBase.getFacesConte
xt(UIComponentClassicTagBase.
java:2122) 

This is a common mistake and is not related to any components you are using. You forget to put faces in the URL. For example:

http://server:port/FederatedApp
/test.jspx

Put faces in the URL as follows:

http://server:port/FederatedApp/fac
es/test.jspx

Task Form/Action Issues

Table C-9 describes symptoms, possible causes, and possible solutions for task form/ action issues.

Table C-9 Troubleshooting Task Form/ Action Issues

Symptom Possible Cause Possible Solution

The task form application does not have an empty JSPX page.

N/A

The task forms are now invoked using an ADF task flow and control is returned to the module that initiated the task form task flow when the task flow completes. Therefore, no empty JSPX is needed.

The task form does not load in Microsoft Internet Explorer.

Microsoft Internet Explorer has a URL length limit.

Your task form URL length is too long.

Deployment fails with a class not found exception.

The shared library entry is missing from weblogic.xml.

If you see the following error:

Caused By:
 java.lang.ClassNotFoundException:
oracle.bpel.services.datacontrol.ty
pes.Number 

during deployment of a task form, then it is likely due to the missing shared library in weblogic.xml. Add the following element in weblogic.xml:

<library-ref>
 <library-name>oracle.soa.worklist.w
 ebapp</library-name>
 <specification-version>11.1.1</spec
 ification-version>
</library-ref> 

Deployment/access of task form fails when the hostname is used.

The DNS entry is missing.

If you are using a server with DHCP, the DNS entry may be missing for the host. Therefore, deployment/access using the IP address may succeed, but deployment/access using a hostname may fail. Update your client computer by manually adding the host/IP address:

  • On Windows operating systems, this is typically is in %windir%\drivers\etc\lmhosts.

  • On Linux/UNIX, this is typically in /etc/hosts.

Task form URL protocol (HTTP or HTTPS).

You are unable to access the task form through HTTPS or HTTP.

  • DefaultToDoTaskForm: The worklist accesses the default to-do task form using the port/protocol returned by the getServerInfo API from the runtime config service. This API uses the frontend host setup for the cluster or managed server. It gives preference to the HTTPS protocol if it is enabled. It is dynamic and can be changed using Oracle Enterprise Manager Fusion Middleware Control for default to-do tasks as the other custom task form.

  • CustomTaskForm: The custom task form gets the HTTP and HTTPS information from the setup. It uses the JRF API to get the information. This gets the frontend host information from the cluster and then the managed server if the cluster frontend host is not set up. In case the managed server frontend host is not set up, it uses a regular port. If the user enables the HTTPS port during deployment, the task form stores this information in the database. After this occurs, if the user disables the HTTPS port using Oracle WebLogic Server Administration Console, then the user must remove the HTTPS port for the task form using Oracle Enterprise Manager Fusion Middleware Control.

Note:

If you receive a java.lang.OutOfMemoryError: PermGen space error when deploying multiple task forms, you may need to increase PermGen memory. For more information, see PermGen Memory Requirements for Multiple ADF Task Form Deployments.

Task Comments/Attachment Issues

Table C-10 describes symptoms, possible causes, and possible solutions for task comments/attachment issues.

Table C-10 Troubleshooting Task Comments/Attachment Issues

Symptom Possible Cause Possible Solution

The file is not getting uploaded.

The file is too big

By default, ADF has a size limit of 2000 KB for each request. Add the following parameters in web.xml to adjust the file size and temporary storage for uploaded files:

<context-param>
  <!-- Maximum memory per request
(in bytes) -->
<param-name>oracle.adf.view.faces.U
PLOAD_MAX_MEMORY</param-name>
  <!-- Use 500K -->
  <param-value>512000</param-value>
</context-param>
<context-param>
  <!-- Maximum disk space per
 request (in bytes) -->
<param-name>oracle.adf.view.faces.U
PLOAD_MAX_DISK_SPACE</param-name>
  <!-- Use 5,000K -->
 <param-value>5120000</param-value>
</context-param>
<context-param>
  <!-- directory to store temporary
 files -->
<param-name>oracle.adf.view.faces.U
PLOAD_TEMP_DIR</param-name>
  <!-- Use an ADFUploads
 subdirectory of /tmp -->
<param-value>/tmp/ADFUploads/</para
m-value>
</context-param>

The file uploaded in the task details application is not visible in the same task flow.

After uploading a file, the attachment link generated in the task form is invalid. Clicking this link returns an empty stream.

When you upload a file, you see the attachment link in the table. However, this link does not work. You must reload the task details to view the file.

Adding file attachments creates a new task version, but adding a URL attachment does not create a new version.

Inconsistent behavior of the URL attachment and file attachment.

When a file is uploaded, the task is saved because the file is uploaded to persistency storage. This creates a new task version. The URL attachments only update the local task object in the user interface application. Therefore, no task version is created.

Design Time at Runtime Issues

Table C-11 describes symptoms, possible causes, and possible solutions for design time at runtime issues. Two design time at runtime tools are available for use:

  • Oracle SOA Composer

  • Task Configuration tab of Oracle BPM Worklist

Table C-11 Troubleshooting Design Time at Runtime UI Issues

Symptom Possible Cause Possible Solution

Modifications made to a task in a design time at runtime tool do not appear for the task.

The task was instantiated before you actually edited it using a design time at runtime tool.

Design time at runtime updates go into effect only for instances created after the changes, and not for those that were created before the change.

Therefore, if you edit a task using a design time at runtime tool, and then instantiate a new task, the new instance of the task has the changes you made.

Modifications made to a task in a design time at runtime tool do not appear for the task.

The changes made were probably not committed to the MDS repository.

The Save button just saves the changes made in a design time at runtime tool to the sandbox. To see these changes in action, click Commit to send them to the MDS repository.

Human Workflow API (Including SOAP/EJB) Usage Issues

Table C-12 describes symptoms, possible causes, and possible solutions for human workflow API (including SOAP/EJB) usage issues.

Table C-12 Troubleshooting Human Workflow API Usage Issues

Symptom Possible Cause Possible Solution

Location of the JavaDoc for human workflow APIs.

N/A

See Workflow Services Java API Reference for Oracle SOA Suite, which is available in the documentation library.

Understanding the API usage.

N/A

  1. Refer to the API documentation.

Using .net to access the APIs.

N/A

It is possible to write a .net client that accesses the SOAP web service APIs.

You receive Class not found errors when attempting to use the Java API clients.

Not all required JAR files are in the client class path.

-

Creating a routing slip for simple patterns to use with a simple approval task or to dynamically route a task during task initiation.

N/A

See oracle.bpel.services.workflow.task.impl.SimpleApprovalTaskUtil.

Newly created task does not appear in Task UI

createTask activity requires SimpleWorkflowApp. This did not get deployed during SOA server setup.

Deploy Simple Approval composite before you create a task using Create Task global activity.

Oracle JDeveloper Data Control / Form Generation Issues

Table C-13 describes symptoms, possible causes, and possible solutions for Oracle JDeveloper data control/form generation issues.

Table C-13 Troubleshooting Oracle JDeveloper Data Control / Form Generation Issues

Symptom Possible Cause Possible Solution

Empty1.jspx is not generated when creating the task form.

Created an initiator task based on an XSD element and tried to autogenerate the task form.

This file is not required except for the BPM initiator task. If you encounter it, create an Empty1.jspx file in the same directory as referenced by adfc-config.xml and put the following content in it:

<?xml version='1.0'
 encoding='UTF-8'?><jsp:root
xmlns:jsp="http://java.sun.com/JSP/
Page" version="2.1"xmlns:f="http://java.sun.com/jsf/core"
xmlns:h="http://java.sun.com/jsf/html"
xmlns:af="http://xmlns.oracle.com/a
df/faces/rich">  <jsp:directive.page
contentType="text/html;charset=UTF-8"/>
  <f:view>
    <af:document id="d1">
      <af:form id="f1"></af:form>
    </af:document>
  </f:view>
</jsp:root>

Human Workflow Service/ System MBean Browser Issues

Table C-14 describes symptoms, possible causes, and possible solutions for human workflow service/System MBean Browser issues.

Table C-14 Troubleshooting Human Workflow Service/ System MBean Browser Issues

Symptom Possible Cause Possible Solution

Setting commonly used human workflow configuration parameters.

N/A

Use the Workflow Task Service and Workflow Notification pages of Oracle Enterprise Manager Fusion Middleware Control:

  1. In the navigator, right-click soa-infra.

  2. Select SOA Administration > Workflow Config > Mailer tab to access notification properties.

  3. Select SOA Administration > Workflow Config > Task tab to access task service properties.

For more information, see Configuring Human Workflow Notification Properties and Configuring Human Workflow Task Service Properties.

Setting human workflow configuration parameters not available in the Oracle Enterprise Manager Fusion Middleware Control properties pages.

N/A

Use the System MBean Browser in Oracle Enterprise Manager Fusion Middleware Control:

  1. In the navigator, right-click soa-infra.

  2. Select SOA Infrastructure >Administration > System MBean Browser.

  3. Select Application Defined MBeans > oracle.as.soainfra.config > server > WorkflowConfig > human-workflow.

  4. Set simple parameters by editing the appropriate field in the Attributes tab, and clicking Apply.

  5. Make more complex parameter changes (for example, adding a new locale) by switching to the Operations tab, selecting the appropriate operation, entering required fields, and clicking Invoke.

The System MBean Browser does not reflect my changes after editing the human workflow configuration MBeans.

The System MBean Browser is showing a previously cached version of beans.

Click the refresh cached tree data button in the System MBean Browser.

Human workflow services are not locating resource bundles or classes located at the workflow customizations class path URL.

The protocol is not specified in the URL, or the URL is missing a trailing forward slash (/).

Ensure that the configured URL is formatted correctly, and specifies a protocol. If the class path points to a directory (rather than a JAR file), it is important that the URL has a trailing forward slash character. For example:

file:///home/wstallar/wfcustomizati
ons/

Manually setting the URL used for displaying task details for a particular task component.

N/A

Use the Administration page in Oracle Enterprise Manager Fusion Middleware Control for the human task service component.

See Managing the URI of the Human Workflow Service Component Task Details Application for instructions.

You can edit or delete existing task display URL entries, and add new entries. For task display URLs used from Oracle BPM Worklist, the application name must be set to worklist.

AMX Extension Issues

Table C-15 describes symptoms, possible causes, and possible solutions for AMX extension issues.

Table C-15 Troubleshooting AMX Extension Issues

Symptom Possible Cause Possible Solution

The dynamic approval group class is not found.

The class file is not accessible in the Oracle SOA Suite class path.

To make the dynamic approval group class accessible, the class file must be placed in the following directory:

$FMW_HOME/SOA_HOME/soa/modules/ora
cle.soa.ext_11.1.1/classes

This directory is part of the SOA class path.

The Oracle WebLogic Server must be restarted.

During design time at runtime, while defining a rule based on the Approval Group list builder, a message keeps appearing indicating that the group does not exist.

The Approval Group name is not enclosed in quotes (" ").

Enclose the name in quotes (for example, "Sample Approval Group Name").

In a ruleset, many rules defined are applicable for a transaction. It appears that the correct constraints are not getting applied; therefore, the generated approver list is not correct.

Only one rule from a ruleset must be applicable for a transaction.

In case many rules are true, the actions associated with the applicable rule with the highest priority are executed.

In case multiple applicable rules have the same priority, the first rule in the list is picked and its actions are executed.

Check that all rules in the ruleset have priorities defined so that multiple rules with the same priority are not applicable for the same transaction.

Oracle BPM Worklist/Task Region Issues

Table C-16 describes symptoms, possible causes, and possible solutions for Oracle BPM Worklist/task region issues.

Table C-16 Troubleshooting Oracle BPM Worklist/Task Region Issues

Symptom Possible Cause Possible Solution

You receive the following exception message in the logs:

<Warning>
<oracle.adf.controller.internal.
metadata.MetadataService>
<BEA-000000><ADFc:
/META-INF/adfc-config.xml: >
<Warning>
<oracle.adf.controller.internal.
metadata.MetadataService><ADFC-5
2024> <ADFc: Duplicate managed
bean definition for
'aleCompBindings' detected.> 

The shared library oracle.soa.worklist.webapp is referenced in the weblogic.xml file and also the JAR files adflibWorklistComponents.jar and adflibTasklistTaskflow.jar are packaged in the web application.

These JARs should not be packaged inside the web application. They should only be referenced as a shared library. Do not package these JARs in the web application.

You receive the following exception message in the logs:

Duplicate default server in
client configuration.
Configuration needs to have only
one default server in client
configuration.
Specify one default server in
client configuration. 

Two default servers are specified in the client configuration file or in the JAXB object passed to the task flow.

Mark only one server as the default in the client configuration file or in the JAXB object passed.

You receive the following exception message in the logs:

The default server is not
specified 

The default server is not specified in the client configuration file or in the JAXB object passed to the task flow.

Ensure that the default server is marked in the client configuration file or the JAXB object.

You receive the following exception message in the logs:

Invalid display column.
The display column COLUMN NAME
is not a valid Task column.
Specify a valid column name.

The column name passed to the task flow parameter displayColumnsList is not correct.

Ensure that you pass the correct column name to the task flow parameter.

You receive the following exception message in the logs:

java.lang.IllegalStateException:
Attempt to validate an already
invalid RegionSite: 

This is a generic exception that sometimes appears in server logs (for example, AdminServer.log).

See the real exception in the diagnostic logs (for example, AdminServer-diagnostic.log) and provide that exception with the bug you can file.

You receive the following exception message in the logs:

Caused by:
oracle.adf.controller.Controller
Exception: ADFC-02001: The ADF
Controller cannot find
'/WEB-INF/taskList-task-flow-def
inition.xml'

The Oracle BPM Worklist JARs are not provided in the class path, either by referring to the shared library oracle.soa.worklist.webapp or by packaging those in the web application.

Ensure either the JARs are referred through the shared library or packaged inside the application.

Filters for the task list are removed when the task list is refreshed.

Because an inbox is not a persisted view, filters set on it are removed when rendering the page again or refreshing the task list.

Instead of setting filters on the task list, create a user view with the required set of filters and pass the viewId of that view as the value of parameter ViewFilter. This makes that view the default view of the task list.

You have set the taskTypesFilterList parameter, but instead of seeing assigned tasks, you are seeing all tasks.

If you specified the taskTypesFilterList parameter, then you forgot to specify the attributesFilterList parameter.

You have to use both parameters with the AND operator. For example:

<parameter
id="taskTypesFilterList"
value="http://xmlns.oracle.com/H
elpDeskRequestSOAApp/HelpDeskReq
uestComposite/HelpDeskRequestHum
anTask,[
http://xmlns.oracle.com/Vacation
RequestApp/VacationRequest/Vacat
ionRequestTask]"/>

<parameter id=
"attributesFilterOperator"
value="and"/>
 <parameter id= "attributesFilterList"
 value="state=ASSIGNED"/>

You receive the following exception message in the logs:

[AdminServer] [NOTIFICATION]
 [J2EE JSP-00008]
 [oracle.j2ee.jsp] [tid:
 [ACTIVE].ExecuteThread: '15'
 for queue:
 'weblogic.kernel.Default
 (self-tuning)'] [userId:
 weblogic] [ecid:
17011f2a001d6b0e:7e22d6ce:12e344
4eb1b:-8000-0000000000002f0a,0]
 [APP: FederatedApp_
application1] unable to dispatch
 JSP page: The following
 exception occurred:.[[
java.lang.RuntimeException:
 Cannot find FacesContext 
at
javax.faces.webapp.UIComponentCl
assicTagBase.getFacesContext(UIC
omponentClassicTagBase.java:2122)

This is a common mistake that is generic in nature and is not related to any components you are using. You forgot to put faces in the URL. For example:

http://server:port/FederatedApp/test.jspx

Put faces in the URL as follows:

http://server:port/FederatedApp/
faces/test.jspx

You receive the following exception message in the logs:

[AdminServer] [TRACE] [] []
[tid: [ACTIVE].ExecuteThread:
'5' for queue:
'weblogic.kernel.Default
(self-tuning)'] [userId:
weblogic] [ecid:
17011f2a001d6b0e:7e22d6ce:12e344
4eb1b:-8000-0000000000001d39,0]
[SRC_CLASS:
oracle.bpel.services.workflow.cl
ient.config.ClientConfigurationU
til] [APP: FederatedApp_
application1] [SRC_METHOD:
getClientConfiguration]
WorkflowServiceClientContext:
Cannot find client
configuration file: wf_client_
config.xml 

There are three possible causes for this issue:

  1. The client configuration file is not provided in the class path.

  2. The JAXB object is not passed to the task flow.

  3. If one of the above is provided, the port number for the remote/SOAP client is incorrect.

Ensure either the client configuration file wf_client_config.xml is provided in the class path or the JAXB object is passed to the task flow. If it is already done, ensure that the port number refers to the SOA server.

You receive the following exception message in the logs:

<Error> <Deployer> <BEA-149265>
<Failure occurred in the
execution of deployment request
with ID '1297964056778' for
task '3'. Error is:
'weblogic.management.DeploymentE
xception: [J2EE:160149]Error
while processing library
references. Unresolved
application library references,
defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false].'
weblogic.management.DeploymentEx
ception: [J2EE:160149]Error
while processing library
references. Unresolved
application library references,
defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false]. 

Shared library oracle.soa.workflow.wc is referenced in weblogic.xml, but is not available on the server.

Ensure that this shared library is deployed on the server on which you are deploying your application. The shared library may be deployed, but not targeted, for that server.

You cannot see the mapped attributes mapped columns.

Note: Starting with Release 11g R1 (11.1.1.4), Oracle BPM Worklist flex fields are now known as mapped attributes.

The right set of parameters is not being passed to the task list task flow.

The correct set of parameters to be passed is as follows:

<parameter id= "displayColumnsList"
value="assignees,creator,
assignedDate,state,
textAttribute1,textAttribute2"/>

textAttribute1,textAttribute2 is the correct way to provide a value for the column name. Providing 'label name'(Name of the mapping) associated with these values does not work.

You must specifically pass the fully qualified value to parameter taskTypesFilterList. Otherwise, the column creation does not work.

For example:

<parameter
 id="taskTypesFilterList"
value="http://xmlns.oracle.com/Hel
pDeskRequestSOAApp/HelpDeskRequest
Composite/HelpDeskRequestHumanTask
"/>

Test-to-Production Issues

Table C-17 through Table C-20 describe symptoms, possible causes, and possible solutions for test-to-production issues.

Table C-17 Troubleshooting Test-to-Production Issues

Symptom Possible Cause Possible Solution

Finding the default realm name for a SOA server.

N/A

The defaultRealmName can be obtained from the identity configuration service. You can get this from the SOAP service test page.

  1. From a browser, go to the following URL:

    http:host:port/integration/servic
    es/IdentityService/configuration
  2. Select getDefaultRealmName from the Operation drop-down menu.

  3. Click the Invoke button.

This retrieves the default realm name. Here is a sample answer from an invocation:

<env:Envelope
xmlns:env="http://schemas.xmlsoap.org/soap
/envelope/">
 <env:Header/>
 <env:Body>
  <realmName
xmlns="http://xmlns.oracle.com/bpel/servi
ces/IdentityService">jazn.com</realmName>
 </env:Body>
</env:Envelope>

Table C-18 Troubleshooting Test-to-Production Issues

Symptom Possible Cause Possible Solution

The following user authentication error (in this example, the user is FMW_USERID and the identity context is jazn.com) is displayed:

[java] Error in workflow service Web
service operation invocation. The
error is ORA-30501:Error in
authenticating user.
[java] Error in authenticating and
creating a workflow context for
user jazn.com/FMW_USERID. [java]
Verify that the user credentials
and identity service configurations
are correct.

This occurs if the given user is not seeded and available in the LDAP provider.

To identify if the user is seeded properly, try to log in to Oracle BPM Worklist from a browser as this user. If the user can log in to Oracle BPM Worklist, that means the user is seeded.

If Oracle Internet Directory or another LDAP provider is used, ensure the configuration of the LDAP provider is completed correctly. Otherwise, you cannot get past this error.

While importing task payload mapped attribute mappings (previously known as flex fields) into the target SOA server, you may encounter the following error in the console logs:

[java] Caused by:
java.sql.SQLIntegrityConstrain
tViolationException: ORA-02291:
integrity constraint (UAT_
SOAINFRA.SYS_C0018364) violated -
parent key not found* * 

The importing of task payload mapped attribute mappings into the target SOA server is a two-step process.

Even before the import of task payload mapped attribute mappings into the target SOA server operation is attempted, there is a prerequisite step that must be performed. This is the import of attribute labels into the target SOA server operation.

To be successful, perform the following operations (in the correct order) with the human workflow test-to-production migration tool:

  1. Import attribute labels into the target SOA server.

  2. Import payload mappings into the target SOA server.

See Moving Human Workflow Data from a Test to a Production Environment for more details.

Table C-19 Troubleshooting Test-to-Production Issues

Symptom Possible Cause Possible Solution

Assume you encounter the following error during rule migration:

[java]  Error encountered during
migration. [java] Exception in thread
"main"[java] 
UserConfigDataMigrationException:[java]
faultString:Invalid parameters for RULE.
[java] Invalid user and group: both
parameters can not have null
values.[java] To migrate User Rules,
provide the 'user' parameter only.
[java]  To migrate Group Rules, provide
the 'group' parameter only.[java]
oracle.bpel.services.workflow.util.
tools.wfUserConfigDataMigrator.
UserConfigDataMigrationE xception [java]
at oracle.bpel.services.workflow.
util.tools.wfUserConfigDataMigrator.
implhwfMigrator.parseParametersNode

There are two properties in the migration.properties file that are of interest for this error.

  • user

  • group

During any rule migration (whether export or import) operation, at most one of them (user or group) should have a value. That is, both user and group cannot have null or empty values.

Set values for at most one of them. To perform user rule migration, set the user parameter alone. To perform group rule migration, set the group parameter alone.

Assume you encounter the following error during rule migration:

[java]  Error encountered during
migration.[java] Exception in
thread "main"[java] 
UserConfigDataMigrationExcepti
on: [java]  faultString:
Invalid parameters for RULE.
[java]  Invalid user and group: both
 parameters can not have values.
[java]  To migrate User Rules,
 provide the 'user'parameter only. 
[java]  To migrate Group Rules,
 provide the 'group' parameter only.
[java]  oracle.bpel.services. workflow.util.tools.wfUserConfigData
Migrator.UserConfigDataMigrationE
xception 

This is similar to the previous explanation in this table.

During any rule migration (whether export or import) operation, both user and group parameters cannot have a value.

Provide values for at most one of them.

Table C-20 Troubleshooting Test-to-Production Issues

Symptom Possible Cause Possible Solution

Testing the health of the installed server.

N/A

Before performing a test-to-production migration, it is useful to test the health of the server.

From a browser, you can test some SOAP services. The following list provides a subset of human workflow services and Oracle BPM Worklist:

http://host:port/integration/worklistapp/

http://host:port/integration/services
/TaskQueryService/TaskQueryService

http://host:port/integration/services/IdentityService/configuration

http://host:port/integration/services/IdentityService/identity

http://host:port/integration/services/RuntimeConfigService/RuntimeConfigService

You can randomly test some operations in these services, and verify that the operation yields results. Similarly, you can log in as a user to Oracle BPM Worklist and see if everything is fine.

For more information about test-to-production issues, see Moving Human Workflow Data from a Test to a Production Environment.

Identity Service Issues

Table C-21 and Table C-22 describe symptoms, possible causes, and possible solutions for identity service issues.

Table C-21 Troubleshooting Identity Service Issues

Symptoms Possible Cause Possible Solution

In a clustered environment, you can receive the following error when you attempt to act on an approval task in Oracle BPM Worklist:

Token Decryption Failed

You are using a file-based repository, instead of an LDAP-based repository. This causes the application roles to not be synchronized across the nodes.

For repository configuration information in a clustered environment, you must follow the instructions in the Enterprise Deployment Guide for Oracle SOA Suite.

Only a subset of users in LDAP can log in to Oracle BPM Worklist.

The user base DN is not configured properly.

Mention the user base under which all the groups are seeded. This can be performed in two ways:

Add the base DN under which all the required groups are seeded. For instance, if users are seeded under:

UserDN 1 : cn=users1,dc=us,dc=oracle,dc=com
UserDN 2 : cn=users2,dc=us,dc=oracle,dc=com
UserDN 3 : cn=users3,dc=us,dc=oracle,dc=com

Then mention the group base DN as follows:

dc=us,dc=oracle,dc=com

This is the common DN. If only some user DNs are required (for example, UserDN1 and UserDN2), then the following property must be added to serviceInstance with the name idstore.ldap in the $DOMAIN_HOME/config/fmwconfig/jps-config.xml file:

<serviceInstance name="idstore.ldap"
 provider="idstore.ldap.provider">
 <property name="idstore.config.provider" 
value="oracle.security.jps.wls.internal.idstore.WlsLd
apIdStoreConfigProvider"/>
 <property name="CONNECTION_POOL_CLASS"
value="oracle.security.idm.providers.stdldap.JNDIPool"/>
 <extendedProperty>
   <name>user.search.bases</name>
   <values>
      <value>cn=users1,dc=us,dc=oracle,dc=com</value>
      <value>cn=users2,dc=us,dc=oracle,dc=com</value>
   </values>
 </extendedProperty>
</serviceInstance>

Users and groups seeded only in the first authenticator are visible, but not from the other authenticators.

By default, users and groups from the first authenticator are authorized.

Starting with Release 11.1.1.4, you can authorize users and groups from multiple authenticators. For more information, see Configuring Multiple Authentication Providers.

Table C-22 Troubleshooting Identity Service Issues

Symptom Possible Cause Possible Solution

The following exception appears when myrealm (the default realm in the Oracle WebLogic Server Administration Console configuration) is passed as a parameter to the identity context to the Identity Service APIs.

Exception seen :
 Service"
Unknown macro: {0}
"in configuration" 
Unknown macro: {1}
" could not be
initialized. Error
in initializing
service
"Authentication"
in configuration
"myrealm".

The human workflow identity service uses the identity context that is set in the WorkflowIdentityConfig file (by default, it is jazn.com) and not from the Oracle WebLogic Server configuration. Therefore, in the customer code, if jazn.com is passed as the identity context, the authenticate API should work fine.

To change the realm name, the WorkflowIdentityConfig file can be edited in the System MBean Browser of Oracle Enterprise Manager Fusion Middleware Control.

  1. In the navigator, right-click soa-infra.

  2. Select SOA Infrastructure > Administration > System Mbean Browser.

  3. Select Application Defined Mbeans > oracle.as.soainfra.config > Server > WorkflowIdentityConfig > human-workflow > WorkflowIdentityConfig.ConfigurationType.

  4. Select the configuration and rename it by invoking the operation setRealmName. This change requires a server restart.

After configuring LDAP with Oracle WebLogic Server, the users are visible in the Oracle WebLogic Server Administration Console, but the following error is thrown:

No Role found
matching the
criteria

The group's base DN is not configured properly. Either the group that is being looked up is not present in LDAP or it may be seeded outside the group base DN that is mentioned while configuring LDAP.

Mention the group base under which all the groups are seeded. This can be performed in two ways.

Add the base DN under which all the required groups are seeded. For instance, if groups are seeded under:

GroupDN 1 : cn=groups1,dc=us,dc=oracle,dc=com
GroupDN 2 : cn=groups2,dc=us,dc=oracle,dc=com
GroupDN 3 : cn=groups3,dc=us,dc=oracle,dc=com

Then mention the group base DN as follows:

dc=us,dc=oracle,dc=com

This is the common DN. If only some group DNs are required (for example, GroupDN1 and GroupDN2), then the following property must be added to serviceInstance with the name idstore.ldap in the $DOMAIN_HOME/config/fmwconfig/jps-config.xml file.

<serviceInstance name="idstore.ldap"
 provider="idstore.ldap.provider">
   <property name="idstore.config.provider"
value="oracle.security.jps.wls.internal.idstore.WlsL
dapIdStoreConfigProvider"/>
 <property name="CONNECTION_POOL_CLASS"
value="oracle.security.idm.providers.stdldap.JNDIPool"/>
  extendedProperty>
  <name>group.search.bases</name>
  <values>
    <value>cn=groups1,dc=us,dc=oracle,dc=com</value>
    <value>cn=groups2,dc=us,dc=oracle,dc=com</value>
  </values>
 </extendedProperty>
</serviceInstance>

Business Events and Event Delivery Network Troubleshooting

This section describes how to troubleshoot business event and event delivery network (EDN) issues.

Increasing the JMS Adapter Connection Pool Size

EDN uses the connection factories shown in Table C-23 for the JMS adapter to communicate with the underlying JMS topic for event publishing and subscription.

Table C-23 Connection Factories for the JMS Adapter

JMS Type JMS Connection Factories for the JMS Type

Oracle WebLogic Server JMS

  • eis/wls/EDNxaDurableTopic

  • eis/wls/EDNxaTopic

  • eis/wls/EDNLocalTxDurableTopic

  • eis/wls/EDNLocalTxTopic

AQ JMS

  • eis/aqjms/EDNxaDurableTopic

  • eis/aqjms/EDNxaTopic

  • eis/aqjms/EDNLocalTxDurableTopic

  • eis/aqjms/EDNLocalTxTopic

In environments in which there are a large number of concurrent event publishers, a large number of event subscribers, or both, the default connection pool size (200) for the JMS Adapter can be exceeded. This can cause ResourceLimitException errors similar to that shown below:

The JCA Binding Component was unable to establish an outbound JCA CCI connection
 due to the following issue: 
  javax.resource.spi.ApplicationServerInternalException:
    Unable to get a connection for pool = "eis/wls/EDNxaDurableTopic",
  weblogic.common.resourcepool.ResourceLimitException: 
    Configured maximum limit of (0) on number of threads allowed to wait for a
 resource reached for pool eis/wls/EDNxaDurableTopic
Please make sure that the JCA connection factory and any dependent connection
 factories have been configured with a sufficient limit for max connections.
Please also make sure that the physical connection to the backend EIS is available
 and the backend itself is accepting connections. 

When this occurs, you must increase the connection pool Max Capacity value of the corresponding JMS connection factory for the JMS adapter in Oracle WebLogic Server Administration Console. Table C-24 identifies the use of each JMS connection factory for the JMS adapter.

Table C-24 Uses of Each JMS Connection Factory for the JMS Adapter

JMS Connection Factory Used for Outbound Event Publishing? Used for Inbound Event Subscription?

eis/wls/EDNxaDurableTopic

Yes, for event publishing with a global transaction.

Yes, for durable subscriptions with One And Only One (OAOO) consistency.

eis/wls/EDNxaTopic

No.

Yes, for nondurable subscriptions with OAOO consistency.

eis/wls/EDNLocalTxDurableTopic

Yes, for event publishing with a local transaction.

Yes, for durable subscriptions with guaranteed consistency.

eis/wls/EDNLocalTxTopic

No.

Yes, for nondurable subscriptions with guaranteed consistency.

eis/aqjms/EDNxaDurableTopic

Yes, for event publishing with global transaction.

Yes, for durable subscriptions with OAOO consistency.

eis/aqjms/EDNxaTopic

No.

Yes, for nondurable subscriptions with OAOO consistency.

eis/aqjms/EDNLocalTxDurableTopic

Yes, for event publishing with local transaction.

Yes, for durable subscriptions with guaranteed consistency.

eis/aqjms/EDNLocalTxTopic

No.

Yes, for nondurable subscriptions with guaranteed consistency.

When calculating connections used, you must combine the number of events publishing with the event subscriptions if the corresponding JMS connection factory is used for both event publishing and event subscription.

For example, if there are 200 concurrent BPEL process event publishers that publish events in a global transaction, and the default SOAInternalProcessing_maxThreads is set to 150, then these publishers use all 150 threads, which takes 150 connections from eis/wls/EDNxaDurableTopic. There are also a total of 12 SOA event subscribers with ThreadsPerSubscriber set to 5. These subscribers use 12 x 5 = 60 threads, which takes 60 connections from eis/wls/EDNxaDurableTopic.

Because 150 + 60 = 210 > 200 (the default setting), you receive a ResourceLimitException error.

In this case, you may need to increase the size of the Max Capacity field of the connection pool of eis/wls/EDNxaDurableTopic to a larger value (for example, 300).

To increase the connection pool size:

  1. Log in to Oracle WebLogic Server Administration Console.
  2. In the Domain Structure window, click Environment > Deployments > JmsAdapter > Configuration > Outbound Connection Pools.
  3. In the Outbound Connection Pool Configuration Table, expand oracle.tip.adapter.jms.IJmsConnectionFactory.
  4. Click the appropriate connection factory, as described in Table C-24.
  5. Click Connection Pool.
  6. In the Max Capacity field, increase the value (for example, to 300), and click Save.

Tuning Recommendations For Publishing Many BPEL Process Events with Oracle WebLogic Server JMS

When publishing many BPEL process events (for example, one million) with Oracle WebLogic Server JMS, you can receive numerous recoverable errors (for example, 20% of the one million events) if the EDN is not properly tuned. For example, assume you have the following use case scenario:

  • 200 threads

  • 5000 iterations

After running for some time (for example, 40 minutes), you can receive the error shown below:

<Jan 8, 2014 10:08:33 AM PST> <Error> <oracle.soa.adapter.jms.outbound>
<BEA-000000> <JMSMessageProducer_produce: [default destination =
jms/fabric/EDNTopic]:weblogic.messaging.kernel.QuotaException: Quota blocking
time exceeded and no quota available>
<Jan 8, 2014 10:08:33 AM PST> <Error> <oracle.soa.adapter.jms.outbound>
<BEA-000000> <JMSMessageProducer_produce: [default destination =
jms/fabric/EDNTopic]:weblogic.messaging.kernel.QuotaException: Quota blocking
time exceeded and no quota available> 

In addition, subscriptions are slow (there are many database, Coherence cache, and transaction-related operations for the EDN threads) and the accumulated messages slow down publishing.

If this occurs, tune the following areas:

  • The EDN quota is automatically defined on the default EDN topic on Oracle WebLogic Server JMS. This EDN quota has a maximum message limit of 50,000. You may need to define a quota with a different maximum message limit based on your use case scenario. For more information, see Tuning WebLogic JMS in Tuning Performance of Oracle WebLogic Server.

  • Tune the send timeout value on the EDN connection factories based on your use case scenario. The following EDN JMS connection factories have a default value of 10,000 (ms) as the send timeout value:

    • EDNConnectionFactory

    • xaEDNConnectionFactory

    For more information, see Blocking Senders During Quota Conditions in I.

  • When there are heavy loads, messages accumulate in the JMS topic. The EDN client receives a QuoteException when the number of messages exceeds the quota limit. If the EDN client is a composite with a BPEL process service component, the BPEL process persists the messages into the Error Hospital. There may be a big delay in delivery to the subscribers. The messages in the Error Hospital can be recovered manually from the Error Hospital or automatically from the Recovery tab of the BPEL process service engine. For more information, see Recovering From Faults in the Error Hospital and Performing BPEL Process Service Engine Message Recovery.

Tuning EDN Event Bus and Delivery

You can customize the following EDN properties in the System MBean Browser:

  1. Right-click soa-infra.
  2. Select SOA Infrastructure > Administration > System MBean Browser.
  3. Enter EDNConfig in the Search field and click the Search icon or expand Application Defined MBeans > oracle.as.soainfra.config > Server: soa_server > EDNConfig > edn.
  4. Double-click an attribute.
    Properties Type Details

    JmsType

    WLJMS

    The default JMS type for a topic. By default, EDN events published in a given SOA domain are (temporarily) stored in a default JMS topic for consumption by event subscribers. Valid values are:

    • WLJMS (Oracle WebLogic Server JMS)

    • AQJMS (Oracle Advanced Queuing JMS)

    Note:

    Manually tune the following to avoid ResourceLimitExceptions and performance issue:
    • Max Capacity for JMSAdapter Connection Pool from 200 to 300

    • SOAIncoming_maxThreads from 60 to 300

    • SOA_EDN_WM's min-thread-constraint from current 1 to 100

    NumberOfRetrys

    integer value

    The number of delivery retries that EDN automatically attempts in case of a retriable exception during event delivery. This applies to subscribers of either the guaranteed or one and only one consistency level. Once the retry reaches the NumberOfRetrys value, the event is moved to the Error Hospital and subject to manual recovery.

    If the EDN delivers an event and fails with a nonretriable exception, there is no retry, and the event is moved to the Error Hospital immediately.

    If GlobalTxMaxRetry = 0, the retry is disabled.

    Paused

    boolean value

    Temporarily stops delivering events.

    Values are true or false. If true, the JMS consumer threads for each Oracle SOA Suite subscriber are removed. As a result, published events are lost for nondurable subscribers and retained in a JMS server for durable subscribers.

    QueueWaitTime

    30,000

    The amount of time in milliseconds to wait between retries. This value must be at least 1000. The default value is 30,000 (30 seconds).

    ThreadsPerSubscriber

    1

    The number of inbound poller threads that each SOA subscriber should use for processing events.When an event-subscribing composite is deployed, EDN initializes inbound JMS consumer threads whose number is based on this value. You can customize the thread number by configuring the local thread number as described in Updating the Local Inbound Poller Thread Number Value at the Service Component Level. Once locally customized, the thread number is not subject to changes in this global ThreadsPerSubscriber value.

    Note:

  5. Enter a value, and click Apply.

Events Are Consumed by Multiple Revisions of the Same Composites

By design, different composites can subscribe to the same event. However, it is an error if multiple revisions of the same composite subscribe to the same event. This occurs when you have multiple active versions of the composite that may be a result of a patch failure.

Perform the following steps to determine whether there are multiple, active revisions of the same composite and to retire the composite revision that should not be active:

  1. Go to Oracle Enterprise Manager Fusion Middleware Control.
  2. Click SOA > soa-infra.

    In the Deployed Composites section of the Dashboard tab, you see a list of deployed composite names, revisions, and modes (for example, active).

  3. Identify composites with the same name and with an active mode, but with different revisions.
  4. Click the composite revision that should not be active.
  5. Click the Retire button.

Business Event Is Picked Up Twice (Or More) By SOA Server

Business events may be raised from J2EE applications and picked up more than once. Expected behavior is that they must be picked up only once.

If a patch fails, it may be possible for two versions of a given composite to be active (that is, the older version has not been retired). When multiple versions of a composite are active, they all become subscribers and the event is picked up more than once. This should not happen under normal scenarios. Reapply the patch and ensure that the deployed composite only has one active version.

Some Messages Are Lost Between EDN and Composites or Composites Across Clusters

Messages primarily may get lost for the following reasons:

  1. The EDN message was not delivered.

  2. The target asynchronous service did not respond.

  3. The message was roll backed (though it was not lost; you still see it in Oracle Enterprise Manager Fusion Middleware Control).

This issue typically requires you to identify where the messages are supposed to be and to diagnose the path.

  • Check if the Oracle Web Services Manager Policy Manager security configurations allow the client to invoke the SOA composite.

  • Verify that the transaction is not transient (nonpersistent).

    Oracle BPEL Process Manager uses the dehydration store database to maintain long-running, asynchronous processes and their current state information in a database while they wait for asynchronous callbacks. Storing the process in a database preserves the process and prevents any loss of state or reliability if a system shuts down or a network problem occurs. There are two types of processes in Oracle BPEL Process Manager. These processes impact the dehydration store database in different ways.

    • Transient processes: This process type does not incur any intermediate dehydration points during process execution. If there are unhandled faults or there is system downtime during process execution, the instances of a transient process do not leave a trace in the system. Instances of transient processes cannot be saved in-flight (whether they complete normally or abnormally). Transient processes are typically short-lived, request-response style processes. The synchronous process you design in Oracle JDeveloper is an example of a transient process.

    • Durable processes: This process type incurs one or more dehydration points in the database during execution because of the following activities:

      • Receive activity

      • OnMessage branch of a pick activity

      • OnAlarm branch of a pick activity

      • Wait activity

      Instances of durable processes can be saved in-flight (whether they complete normally or abnormally). These processes are typically long-living and initiated through a one-way invocation. Because of out-of-memory and system downtime issues, durable processes cannot be memory-optimized.

  • If a business flow instance is not visible in Oracle Enterprise Manager Fusion Middleware Control and the SOA Infrastructure is running, check that the Audit Level is not set to Off on the SOA Infrastructure Common Properties page. This can be checked in Oracle Enterprise Manager Fusion Middleware Control:

    1. Right-click soa-infra (SOA_cluster_name).

    2. Select SOA Administration > Common Properties.

  • If the business flow instance is available, check the oneWayDeliveryPolicy BPEL property value. You can check the value in Oracle Enterprise Manager Fusion Middleware Control:

    1. In the navigation pane, expand soa-infra (SOA_cluster_name).

    2. Expand the partition, and select the composite.

      The Dashboard page for the composite is displayed.

    3. In the upper right corner, click the Show XML Definition icon.

      The contents of composite.xml for that composite are displayed.

    If this is set to async.cache, you may lose messages. Set it to async.persist for reliable messages. This is typically specified in the BPEL process service component section of the composite.xml file. Therefore, this can be set for custom composites. If the value is not set in composite.xml, the value for oneWayDeliveryPolicy in the System MBean Browser in Oracle Enterprise Manager Fusion Middleware Control is used. The following values are possible:

    • async.persist: Messages are persisted in the database hash map.

    • async.cache: Messages are stored in memory.

    • sync: Direct invocation occurs on the same thread.

    For more information about these settings, see the "Deployment Descriptor Properties" appendix and the "Transaction and Fault Propagation Semantics in BPEL Processes" chapter of Developing SOA Applications with Oracle SOA Suite.

    For information about setting the oneWayDeliveryPolicy property in the System MBean Browser, see Configuring BPEL Process Service Engine Properties.

    It is also possible that the transaction has invoked a target asynchronous service that has not responded back. In this case, the business flow instance shows the call to the target asynchronous service.

  • Check if the instance has rolled back and the message is in recovery.

    1. Log in to Oracle Enterprise Manager Fusion Middleware Control.

    2. Right-click soa-infra (SOA_cluster_name), and select Service Engines > BPEL.

    3. Click the Recovery tab.

    This may occur if any external references receiving the message are not reachable (for example, an external web service, enterprise applications such as Siebel, and so on).

Performance Troubleshooting

This section describes how to troubleshoot performance issues.

Resolving Issue Where BPEL Compiler Does Not Terminate or Report Exceptions

If a large SOA composite application containing many BPEL process service components is compiled, the BPEL compiler may not terminate and report any exceptions. If this occurs, increase the JVM heap size in the ant-sca-compile.xml file as follows:

  • Change <jvmarg value="-Xmx1024m"/>

  • To <jvmarg value="-Xmx4g"/>

The code sample below provides details:

   <scac input="${scac.input}" outXml="${scac.output}"
       error="${scac.error}" appHome="${scac.application.home}"
       failonerror="true" displayLevel="${scac.displayLevel}"
       reportErrors="${scac.reportSchemaValidationErrors}">
       <jvmarg
value="-Djavax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom=oracle.
 xml.xpath.JXPathFactory"/>
       <jvmarg value="-Xms128m"/>
       <jvmarg value="-Xmx4g"/>    
       <jvmarg value="-XX:PermSize=32m"/>
       <jvmarg value="-XX:MaxPermSize=256m"/>
       <jvmarg value="-Doracle.xdkjava.compatibility.version=11.1.1"/>
       <jvmarg value="-Doracle.soa.compatibility.version=11.1.1"/>
       <!-- jvmarg
 value="-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005"/ -->
     </scac>
   </target>  

Resolving Message Failure Caused by Too Many Open Files

You can receive the following error at runtime or compilation time, depending on the number of JAR files being used, the use of file descriptors by JDK 6/JRE, or both.

Message send failed: Too many open files 

To resolve this error, increase the number of file descriptors to at least 4096.

  1. Use the limit command (for the C shell) or the ulimit command (for the Bash shell) to identify the value for descriptors. A value of 1024 is typically too low, especially for JDK 6.
    % limit
    
    cputime      unlimited
    filesize     unlimited
    datasize     unlimited
    stacksize    10240 kbytes
    coredumpsize unlimited
    memoryuse    unlimited
    vmemoryuse   unlimited
    descriptors  1024
    memorylocked 500000 kbytes
    maxproc      46720
    
  2. Log in as the root user on your operating system.
  3. Edit the /etc/security/limits.conf file to increase the value for descriptors.

    For this example, the limits.conf file appears as follows after increasing the limit for all users to 4096:

    #<domain>      <type>  <item>         <value>
    #
    
    #*               soft    core            0
    #*               hard    rss             10000
    #@student        hard    nproc           20
    #@faculty        soft    nproc           20
    #@faculty        hard    nproc           50
    #ftp             hard    nproc           0
    #@student        -       maxlogins       4
    
    # End of file
    @svrgroup    soft    memlock         500000
    @svrgroup    hard    memlock         500000
    *           soft    nofile          4096
    *           hard    nofile          4096
    
  4. Close your terminal and reopen for the change to take effect. A system restart is not required.

Resolving MaxMessageSizeExceededException Errors Caused By Large Payloads

MaxMessageSizeExceededException error occurs when you provide a large payload (for example, 200vu*500iter which is 100,000 requests) to your deployed SOA composite application. MaxMessageSize configuration must be configured based on the load. Click View XML Document in the audit trail to view the payload, and check for MaxMessageSizeExceededException errors. To resolve the error either increase the MaxMessageSize to 80000000, or decrease the session-flush-threashold to 50.

Note:

This values depend on the load. The default value of MaxMessageSize is 10000000. Change the values according to you load.

Increasing MaxMessageSize

To increase the MaxMessageSize:

  1. Login to admin console.

  2. Navigate to servers, protocols, general, max message size.

  3. Changed the value to 80000000. (both the admin and the soa).

  4. Navigate to servers, configuration, server start, arguments, and set-Dweblogic.MaxMessageSize = 80000000.

You can also change the -Dweblogic.MaxMessageSize value by adding weblogic.MaxMessageSize property. To change the value by adding the property:
  1. Open the following file:

    • On UNIX operating systems, open $MIDDLEWARE_HOME/user_projects/domains/domain_name/bin/setDomainEnv.sh.

    • On Window operating systems, open MIDDLEWARE_HOME\user_projects\domains\domain_name\bin\setDomainEnv.bat.

  2. Add the weblogic.MaxMessageSize property with the following value:

    EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
    -Dweblogic.MaxMessageSize=80000000"
    export EXTRA_JAVA_PROPERTIES 
    
  3. Restart the server.

Decreasing session-flush-threashold

Navigate to console cluster, soa_cluster, Configuration, Replication, Advanced, and change the Session Flush Threshold value to 50 (smaller value).

Extending Tablespaces to Avoid Problems at Runtime

If the database tablespace is not extended, runtime processing can be impacted. Messages are not processed or persisted, and exception errors similar to that shown in Example 1 and Example 2 below can appear in the log files. This is because Oracle BPEL Process Manager relies on the database to store instance data. If the database is not available, runtime processing is impacted.

Example 1

INFO: MediatorServiceEngine returning after processing the request for 
operation = processResponse 

[EL Warning]: 2009.01.14 11:46:16.783--UnitOfWork(32372128)--Exception 
[EclipseLink-4002] (Eclipse Persistence Services - 1.1 (Build 
SNAPSHOT-20081007)): org.eclipse.persistence.exceptions.DatabaseException 
Internal Exception: java.sql.BatchUpdateException: ORA-01691: unable to 
extend lob segment SH_SOAINFRA.SYS_LOB0000145067C00007$$ by 1024 in 
tablespace SH_SOAINFRA 

Error Code: 1691 
Query: InsertObjectQuery(com.collaxa.cube.persistence.dto.AuditTrail@199b33d) 
[EL Warning]: 2009.01.14 11:46:16.782--UnitOfWork(32372128)--Exception 
[EclipseLink-4002] (Eclipse Persistence Services - 1.1 (Build 
SNAPSHOT-20081007)): org.eclipse.persistence.exceptions.DatabaseException 
Internal Exception: java.sql.BatchUpdateException: ORA-01691: unable to 
extend lob segment SH_SOAINFRA.SYS_LOB0000145067C00007$$ by 1024 in 
tablespace SH_SOAINFRA 
. . .
. . .

Example 2

<bpelFault><faultType>0</faultType><subLanguageExecutionFault
xmlns="http://schemas.oracle.com/bpel/extension"><part
name="summary"><summary>XPath expression failed to execute.
An error occurs while processing the XPath expression; the expression is
ora:readBinaryFromFile('/scratch/vbasaval/dropbox/LFile/PO_5Gig.zip').
The XPath expression failed to execute; the reason was:
java.sql.SQLException: ORA-01652: unable to extend temp segment by 128 in
 tablespace DEV12_IAS_TEMP

Ensure that you set a tablespace to automatically extend itself by a specified amount when it reaches its size limit. If you do not enable autoextend, ensure that you respond when alerted that the tablespace is reaching its critical or warning threshold size. You can respond to size alerts by manually increasing the tablespace size.

To manually add data files in SQL*Plus:

SQL> alter tablespace SOA12_SOAINFRA add datafile
'/scratch/db12101/Oracle/oradata/SOA12_soainfra2.dbf' size 1000M autoextend on
next 30M maxsize unlimited;

ALTER DATABASE TEMPFILE '/scratch/db12101/Oracle/oradata/SOA12_iastemp.dbf' RESIZE
 100M;
ALTER DATABASE TEMPFILE '/scratch/db12101/Oracle/oradata/SOA12_iastemp.dbf'
 AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED;
ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_mds.dbf' RESIZE
 100M;
ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_mds.dbf' AUTOEXTEND
 ON NEXT 50M MAXSIZE UNLIMITED;
ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_soainfra.dbf'
 RESIZE 2048M;
ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_soainfra.dbf'
 AUTOEXTEND ON NEXT 50M MAXSIZE UNLIMITED;
ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_svctbl.dbf' RESIZE
 100M;
ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_svctbl.dbf'
 AUTOEXTEND ON NEXT 50M MAXSIZE UNLIMITED;
 
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA1.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA2.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA3.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA4.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA5.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA6.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA7.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA8.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA9.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAA.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAB.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAC.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAD.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAE.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAF.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAG.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAH.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAI.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAJ.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;
ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE
 '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAK.dbf' SIZE 100M AUTOEXTEND ON
 NEXT 30M MAXSIZE UNLIMITED;

Resolving Database Growth Issues Caused by a High Volume of Transactions

If Oracle SOA Suite transactions fail and the logs indicate that the database is running out of space, the dehydration store or MDS store may be running out of space due to a high volume of transactions. In the latter example, you see errors such as the following:

java.sql.SQLException: ORA-01653: unable to extend table SH_MDS.CUBE_INSTANCE by
16 in tablespace FUSION_TS_TOOLS 

This indicates that the tablespace is full and the database cannot extend it.

Perform the following tasks.

  1. Purge the dehydration store tables periodically, taking into account the appropriate record retention policies and ensuring that the applications have no dependencies on runtime data.

    The purge should be followed by commands to coalesce the space. For the purging strategy to work, it is important to understand how long to retain the data in the database. Factors that drive the retention policy include the following:

    • Legal requirements

    • Line of business requirements

    • Overall company policy on retention of data

    The longer the retention policy, the greater the volume of data that must be stored and, correspondingly, the higher the disk capacity requirements.

    For details on creating a purging strategy, see Developing a Database Growth Management Strategy and Managing Database Growth .

  2. Ensure that the database hardware has sufficient resources to handle the demands of Oracle database partitioning before configuring your tables for partitioning.

    For the dehydration store, database partitioning using range partitioning and hash partitioning is an optimal solution. Partitioning by definition means storing data in multiple tables to reduce bigger data sets into smaller, more manageable data sets. Partitioning strategies play a large role in easing maintenance overheads (dropping and pruning the partition) and improving performance. Partitioning should at least be done for tables having high activity. This plays a large role in balancing disk I/O and preventing hot disks. One important requirement that you must meet before configuring your tables for partitioning is to ensure that the database hardware has sufficient resources to handle the demands of Oracle database partitioning. If preproduction testing has indicated that the installation is large, Oracle expects that you have sized your environment (CPU, memory, and disk space) correctly to take advantage of the partitioning features.

  3. Tune database parameters for memory, tablespace, and partitions to get maximum performance. For more information, see Section "Tuning Database Parameters" of Tuning Performance.
  4. For other tablespaces running out of space, use the following query to check for free tablespace:
    SELECT TOTAL.TABLESPACE_NAME "TABLESPACE NAME", FREE_SPACE, (TOTAL_SPACE-FREE_
    SPACE) USED_SPACE, TOTAL_SPACE, ROUND((FREE_SPACE*100/TOTAL_SPACE),2) "FREE %"
    FROM (SELECT TABLESPACE_NAME, SUM(BYTES/1024/1024) FREE_SPACE FROM SYS.DBA_
    FREE_SPACE GROUP BY TABLESPACE_NAME) FREE,
    (SELECT TABLESPACE_NAME, SUM(BYTES/1024/1024) TOTAL_SPACE FROM SYS.DBA_DATA_
    FILES GROUP BY TABLESPACE_NAME) TOTAL
    WHERE FREE.TABLESPACE_NAME = TOTAL.TABLESPACE_NAME AND TOTAL.TABLESPACE_NAME =
     '<TABLESPACENAME>'
    ORDER BY 5;
    
  5. To increase tablespace settings, use the administrator account. For example:
    ALTER TABLESPACE TABLESPACE_NAME ADD DATAFILE 'DATAFILE_NAME' SIZE 500M
    AUTOEXTEND ON;
    

Observing Slow Application Performance Such as Longer Time to Serve Pages or Finish Transactions

You can observe slow application performance and/or memory trashing. For example, it may take longer to load and serve pages or to complete composite transactions. Response time may seem slower compared to normal behavior.

There are various reasons for slow performance. It may be due to a large number of servers running on the same host, or there may be a large number of records/sessions/locking in the database. Thread contention can also be a reason for poor performance.

Note:

All configuration changes in Oracle Enterprise Manager Fusion Middleware Control may be unavailable due to this problem.

To diagnosis this issue:

  1. Check the CPU utilization to see if it is saturated due to a heavy load or too many processes in relation to CPU capacity.

    If CPU utilization is at 100% during normal load hours (the target should be 70-80%), you have no capacity to handle a peak load and the hardware resources are insufficient. Add scale-out servers to handle the additional load.

  2. Check applications using Oracle Enterprise Manager Fusion Middleware Control to report on performance. Check the performance of services and the invoke/response times in the BPEL process audit trail.

  3. To maximize performance, it is recommended that you not set the logging level higher than the default INFO level.

    For debugging purposes, you must set the logging level to the FINEST level. However, once issues are resolved, reset the logging level to the default level for best performance. It is also recommended that you set Audit Level to Production on the SOA Infrastructure Common Properties page. This can be set in Oracle Enterprise Manager Fusion Middleware Control as follows:

    1. Right-click soa-infra (SOA_cluster_name).

    2. Select SOA Administration > Common Properties.

    3. Set Audit Level to Production.

  4. Purge periodically based on retention requirements to maintain any service level agreements (SLAs).

    To identify the tables where data growth may lead to performance issues, see Section "Identifying Tables Impacted By Instance Data Growth" of Tuning Performance.

    For information about using the purge scripts, see Managing Database Growth .

  5. Check the memory/IO/paging/swapping/CPU usage load statistics using Top or Glance or another monitoring tool.

  6. Optimize the JVM to avoid full garbage collection or out-of-memory errors.

    Frequent garbage collection can be either due to higher memory usage or memory leaks.

    1. Ensure that the sum of the maximum heap size of all the JVMs running on your system does not exceed the amount of available physical RAM to avoid operating system level paging activity.

    2. Use the JRockit mission control memory profiling tools to get thread dumps and memory snapshots, which helps Oracle Support Services debug any code issues. For more information about thread dumps, see Diagnosing Problems with SOA Composite Applications.

  7. Optimize threads to avoid contention. Get a thread dump to investigate and submit it to Oracle Support Services.

  8. Check the JVM and thread dumps for methods invoking the database to identify if database performance is a bottleneck.

  9. Run database AWR snapshots to identify causes of database performance issues.

  10. Ensure that database statistics are updated at regular intervals and other tunable parameters for memory, tablespace, and partitions are used effectively to obtain maximum performance.

    Here are some common tuning recommendations. For more information, see the "Tuning Database Parameters" section of Tuning Performance.

    • Put indexes and tables in as physically separate disk areas, if possible.

    • Never put rollback segments with data or index segments.

    • Separate highly active tables and indexes into their own tablespaces.

    • Partition high activity tables and indexes to help balance disk I/O and prevent hot disks.

    • Have processes in place to generate database table statistics at regular intervals.

  11. Tune database tables to control the high watermark (HWM) contention of large objects. Tune database advanced queues (AQ) to control HWM contention and ensure consistent performance of producing and consuming messages from AQ.

    The EDN AQ names to be aware of are EDN_EVENT_QUEUE_TABLE and EDN_OAOO_DELIVERY_TABLE.

  12. Tune the BPEL process properties to reduce overhead (for example, disable the ValidateXML and StatsLastN (statistics gathering batch size) properties on the BPEL Service Engine Properties page in Oracle Enterprise Manager Fusion Middleware Control), if they are not needed.

  13. In case of integration with packaged applications (for example, Siebel), check if the issue lies with the legacy applications.

For more information about performance tuning the various components, see the "Top Performance Areas" chapter and the "SOA Suite Components" part in Tuning Performance.

For information about setting properties on the BPEL Service Engine Properties page, see Configuring BPEL Process Service Components and Engines.

Observing Incoming Message Rates Exceeding Outgoing Message Rates

When the incoming rate of messages exceeds the processing rate of messages (that is, the outgoing rate), you can receive the warning message shown below:

[2012-04-09T16:49:40.342-04:00] [Soa_server1] [WARNING] []
 [oracle.soa.bpel.engine.dispatch] [tid:
 weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@e3d0e3d
] [userId: <anonymous>] [ecid: 0000JSnNKVTBh4Z5ln1FiZ1Fdnmd001gWL,0] [APP:
soa-infra] BPEL service engine is overloaded, the inflow is higher than the
outflow, there are "100000" invoke messages pending in the dispatcher in-memory
cache, the configured threshold is "102",the new incoming invoke messages will not
be dispatched or processed immediately, they are going to be persisted in the
dehydration store and can be recovered later using recovery.

To resolve this issue, either increase the processing rate or lower (that is, throttle down) the incoming rate. The relatively low processing rate may be due to any or all of the following issues:

  • Not enough invoke threads

  • Not enough JVM capacity

  • The external partners that the BPEL process is calling cannot scale, which causes the messages to back up in the BPEL process.

To identify the exact causes and resolve them, perform the following steps:

  1. Check the oneWayDeliveryPolicy property value in the composite.xml file. Is it set to async.persist, async.cache, or sync? For information about this property, see Deployment Descriptor Properties in I.
  2. Ensure that the outgoing invocations from the BPEL process to external services are not backed up due to low scalability of the external services. If the bottleneck is in an external service, any tuning you perform in Oracle SOA Suite (that is, Step 3) does not help. You must first resolve the bottleneck.
  3. Monitor your JVM garbage collection (GC). If the GC frequency appears higher than normal and the JVM appears to be near capacity, increase the JVM heap size.

Server Troubleshooting

This section describes how to troubleshoot server issues.

For more information about server startup issues, see Stopping and Starting the Managed Server and SOA Infrastructure.

Best Practices for Starting and Stopping a Managed Server

As a best practice, it is always recommended that you start and stop a managed server through one, but not both, of the following methods. Do not mix these methods, such as starting the managed server from the command line and stopping it from Oracle Enterprise Manager Fusion Middleware Control, or vice versa.

  • Oracle Enterprise Manager Fusion Middleware Control

    With this method, the node manager must be up and running. The node manager tracks all managed server startups and shutdowns performed from Oracle Enterprise Manager Fusion Middleware Control. With this method, the server state is not an issue.

  • Command line

    With this method, the node manager does not track the server state. Therefore, if you start the server from the command line and shut it down from Oracle Enterprise Manager Fusion Middleware Control, the Oracle WebLogic Administration Server accesses the node manager to determine its status, which returns a state of unknown.

Perform the following steps to stop and start the server from Oracle Enterprise Manager Fusion Middleware Control.

  1. Expand the WebLogic domain.
  2. Select the managed server (for example, named soa_server1).
  3. Select Control > Shut Down.
  4. Select Control > Start Up.

For information on starting and stopping managed servers from the command line, see Installing and Configuring Oracle SOA Suite and Business Process Management.

Diagnosing SOA Server Startup Problems

If the SOA server (soa_infra) does not start, there can be various reasons for server startup issues. A discovery-based approach to finding the root cause is required. Check the server and diagnostic logs as a first attempt to diagnose the issue. In addition, check the following.

To diagnose server startup issues:

  1. Check if the database is not available or there are not enough connections available. Some failures with Oracle SOA Suite runtime can result from database outage/connectivity issues. Perform the following steps:

    1. Log in to Oracle WebLogic Server Administration Console.

    2. In the Domain Structure, view the status by selecting Services > Data Sources > SOADatasource > Monitoring > Testing to test the data source. You can also check this from a SQL prompt.

  2. Check the list of ports used for port conflicts.

  3. Check if the MDS repository cannot load shared documents (incorrect MDS configuration or the database that holds the MDS schema is not reachable).

  4. Check the Oracle Coherence configuration if using an Oracle SOA Suite cluster (use of unicast versus multicast).

    This issue only applies after provisioning is complete. Clustering of SOA servers may fail if there are port conflicts (used by Oracle Coherence). For example, one SOA server may be picking up the deployment, but the other server is not. In some cases, this may only present the following error:

    [soa_server1] [ERROR] [] [Coherence] [tid: Logger@352821903 3.6.0.4] 
    [ecid: 46f620208907e045:63f295ec:12dd091ec2e:-8000-0000000000000003,1:27187] 
    [APP: soa-infra] 2011-01-28 23:06:19.463/414.816 Oracle Coherence GE 3.6.0.4 
    <Error> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
     (self-tuning)', member=n/a): 
    Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout
     during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster[[...
    

    In this example, Oracle Coherence timeouts are prominently available. However, sometimes it presents itself with an unrelated error, such as the following:

    Error creating bean with name 'SensorManager' defined in ServletContext
    resource
    

    The root cause of this is still primarily related to Oracle Coherence configuration.

  5. Check if both the administration server and managed server ports are open and accessible.

  6. Check if managed server startup failed because the administration server is not reachable.

  7. Check network issues (for example, IP routing filtering/rules that may be causing issues).

  8. Check Oracle WebLogic Server LDAP security corruption.

    The managed server may report that policies for the application System MBeans Browser already exist or do not exist.

Specifying the Proxy Server

To use system properties to specify the proxy server, write your client application in the standard way, and then specify Java system properties when you execute the client application. The code sample below provides details:

setenv PROXY_SETTINGS "-DproxySet=true
-Dhttp.proxyHost=www-myproxy.us.mycompany.com -Dhttp.proxyPort=80
 -Dhttp.nonProxyHosts=localhost|*.us.mycompany.com
  |0:0:0:0:0:0:0:1|fe80:0:0:0:250:56ff:fe31"

Note:

When you specify values for proxy properties such as http.proxyHost and http.proxyPort, also specify the http.nonProxyHosts property.

Flow Diagram Does Not Display The First Time on Some Lower End Hosts

The flow diagram for an instance ID of a deployed SOA composite application in Oracle Enterprise Manager Fusion Middleware Control may not display the first time on some lower end hosts. Instead, you receive a failed to load resource message.

As a workaround, close the flow trace page and click the instance ID to return to the flow trace page.

Accessing Oracle Enterprise Manager Fusion Middleware Control on Dual Stack Hosts that Support IPv4 and IPv6

If you run Oracle SOA Suite on a dual stack host that supports both IPv4 and IPv6, you must update the etc/hosts file as shown in Table C-25 for IPv4 clients to access IPv6 URLs in Oracle Enterprise Manager Fusion Middleware Control.

Table C-25 IPv4 and IPv6 Settings in etc/hosts File

On The... Edit the etc/hosts File as Follows....

On the IPv4 client:

xx.xxx.xxx.xxx myhost10-ipv6 

where xx.xxx.xxx.xxx is the IP address of IPv6.

On the IPv6 client

2001:0db8:db0a::0:1  myhost10-ipv6 myhost10-ipv6.us.example.com

Note: Replace 2001:0db8:db0a::0:1 with a value appropriate to your host environment.

Browser Troubleshooting

This section describes how to troubleshoot browser issues.

Limitation on Using the Safari Browser to View WSDL File Content

If you are using the Safari browser, note the following limitation and workaround for viewing WSDL file contents in Oracle Enterprise Manager Fusion Middleware Control. Note also that Mozilla Firefox works correctly and does not require this workaround.

  1. Go to the home page for a SOA composite application.
  2. Click the Show WSDL and endpoint URI link at the top of the page.
  3. Click the WSDL link that is displayed.

    This opens a blank page that does not display the contents of the selected WSDL.

    As a workaround, perform the following additional steps.

  4. In the upper right corner of this page, click the Display a menu for the current page icon.
  5. Select View Source from the menu that is displayed.

    This displays the contents of the selected WSDL in another page.

Additional Troubleshooting Documentation

Table C-26 describes documentation to see for additional troubleshooting issues.

Table C-26 Additional Troubleshooting Documentation

For Information About... See...

SOA Infrastructure startup and shutdown issues

Stopping and Starting the Managed Server and SOA Infrastructure

Using WebLogic Diagnostic Framework (WLDF) and Diagnostics Framework (DFW) to diagnosis Oracle SOA Suite problems

Diagnosing Problems with SOA Composite Applications

Managing large documents and metadata and for managing environments with large numbers of instances

Managing Large Documents and Large Numbers of Instances in Developing SOA Applications with Oracle SOA Suite

Deployment and compilation errors

Testing and Troubleshooting in Developing SOA Applications with Oracle SOA Suite

Oracle Web Services Manager Policy Manager errors

Diagnosing Problems with Oracle Web Services Manager in Securing Web Services and Managing Policies with Oracle Web Services Manager

Oracle Business Rules

Oracle Business Rules Troubleshooting in Designing Business Rules with Oracle Business Process Management

Unable to Get Pool Connection

In scenarios where there are a large number of concurrent event publishers and a large number of event subscribers, (for example, PSR stress tests), ResourceLimitException with following messages or similar can occur.

The JCA Binding Component was unable to establish an outbound JCA CCI connection due to the following issue: 
  javax.resource.spi.ApplicationServerInternalException:
    Unable to get a connection for pool = "eis/wls/EDNDurableTopic",
  weblogic.common.resourcepool.ResourceLimitException: 
    Configured maximum limit of (0) on number of threads allowed to wait for a resource reached for pool eis/wls/EDNDurableTopic
Please make sure that the JCA connection factory and any dependent connection factories have been configured with a sufficient limit for max connections.
Please also make sure that the physical connection to the backend EIS is available and the backend itself is accepting connections. 

Causes

EDN 12c uses following JMS adapter conneciton factories to communicate with underlying JMS topic for both event publish and/or event subscription.

JMS Type JMS Connection Factories for JMS Adapter

WLJMS

  • eis/wls/EDNxaDurableTopic

  • eis/wls/EDNxaTopic

  • eis/wls/EDNLocalTxDurableTopic

  • eis/wls/EDNLocalTxTopic

AQJMS

  • eis/aqjms/EDNxaDurableTopic

  • eis/aqjms/EDNxaTopic

  • eis/aqjms/EDNLocalTxDurableTopic

  • eis/aqjms/EDNLocalTxTopic

When there are many publishers and subscribers, the default Connection Pool Size of JmsAdapter, which is 200, may be all used up.

Solutions

Increase the "Max Capacity" of Connection Pool of corresponding JMS connection factory for JMS adapter. The following table indicates the uses of each of the JMS connection factories for JMS adapter:

JMS Connection Factory Used for Outbound Event Publish? Used for Inbound Event Subscription?

eis/wls/EDNxaDurableTopic

No

Yes, for durable subscription with 'OAOO' consistency

eis/wls/EDNxaTopic

Yes, for event publish with global transaction

Yes, for non-durable subscription with 'OAOO' consistency

eis/wls/EDNLocalTxDurableTopic

No

Yes, for durable subscription with 'Guaranteed' consistency

eis/wls/EDNLocalTxTopic

Yes, for event publish with local transaction

Yes, for non-durable subscription with 'Guaranteed' consistency

eis/aqjms/EDNxaDurableTopic

No

Yes, for durable subscription with 'OAOO' consistency

eis/aqjms/EDNxaTopic

Yes, for event publish with global transaction

Yes, for non-durable subscription with 'OAOO' consistency

eis/aqjms/EDNLocalTxDurableTopic

No

Yes, for durable subscription with 'Guaranteed' consistency

eis/aqjms/EDNLocalTxTopic

Yes, for event publish with local transaction

Yes, for non-durable subscription with 'Guaranteed' consistency

When calculating JMS connections used, you need to add the number of event publishing to that of event subscription if the corresponding JMS connection factory is used for both event publishing and event subscription.

For example, if there are 200 concurrent BPEL event publishers that publish events in global transaction, and the default SOAInternalProcessing_maxThreads is 150 (SOAMaxThreadsConfig = 300 and InternalProcessingPercentage = 50%), then these publishers would use all 150 threads, which takes up 150 JMS connections from "eis/wls/EDNDurableTopic".

Suppose there are also a total of 8 SOA event subscribers with "durable" option and subscription consistency being "Guaranteed". The ThreadsPerSubscriber is also set to 10. Then these subscribers could use up to 8 * 10 = 80 consumer threads. Since the actual consumer threads is limited by the EDN WorkManager's max-threads-constraint, which is SOAIncoming_maxThreads which is 60 by default, the consumer threads would take away 60 connections from "eis/wls/EDNDurableTopic". Since 150 + 60 = 210 > 200 (200 is the default adapter connection pool size of "eis/wls/EDNDurableTopic"), user would get ResourceLimitException. In this case, user needs to increase the "Max Capacity" of Connection Pool of "eis/wls/EDNDurableTopic" to a larger value, for example, 300.

Note:

When a resource is accessed by multiple requests in a cluster environment, resource busy message logs like org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLException: ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired may be logged. These logs are warning messages and require no action if they do not affect the functionality.