B Troubleshooting Oracle SOA Suite and Oracle BPM Suite

This appendix describes how to troubleshoot issues you can encounter when using Oracle SOA Suite and Oracle BPM Suite, including logging level setup, parallel purge and table partitioning issues, connection and transaction timeout issues, runtime diagnostic issues, human workflow issues, Event Delivery Network (EDN) issues, performance issues, server startup best practices, and browser issues. References to additional troubleshooting information are also provided.

This appendix includes the following sections:

B.1 Setting Logging Levels for Troubleshooting

To simplify troubleshooting, it is recommended that you set logging levels to the TRACE:32 FINEST level in Oracle Enterprise Manager Fusion Middleware Control. This section describes loggers to which to set to this level.

To set logging levels for troubleshooting:

  1. See Section 3.4, "Configuring Log Files" for instructions on accessing the Log Configuration page.

  2. From the Oracle Diagnostic Logging Level (Java Level) list, set the following parent loggers to the TRACE:32 FINEST level:

    • oracle.soa

    • oracle.fabric

    • oracle.integration

    • oracle.wsm (Setting this logger to the ERROR level may also be sufficient because this setting logs the required error messages.)

  3. If you want finer-grained control over logging, expand the parent loggers and set any of the following loggers:

    Component Logger
    Human workflow/approval management extensions (AMX)/rules
    • oracle.soa.services.common
    • oracle.soa.services.identity

    • oracle.soa.services.notification

    • oracle.soa.services.rules

    • oracle.soa.services.rules.obrtrace

    • oracle.soa.services.workflow

    • oracle.soa.services.workflow.common

    • oracle.soa.services.workflow.evidence

    • oracle.soa.services.workflow.metadata

    • oracle.soa.services.workflow.persistency

    • oracle.soa.services.workflow.query

    • oracle.soa.services.workflow.report

    • oracle.soa.services.workflow.runtimeconfig

    • oracle.soa.services.workflow.soa

    • oracle.soa.services.workflow.task

    • oracle.soa.services.workflow.task.dispatch

    • oracle.soa.services.workflow.task.routing

    • oracle.soa.services.workflow.user

    • oracle.soa.services.workflow.verification

    • oracle.soa.services.workflow.performance - Workflow Performance API

    • oracle.soa.services.workflow.worklist - Oracle BPM Worklist

    SOA Infrastructure
    • oracle.fabric.common.wsdl - WSDL/schema management
    • oracle.integration.platform.blocks.deploy - Deployment

    • oracle.integration.platform.blocks.soap - WS binding - Calling web services

    • oracle.integration.platform.blocks.local - Local binding

    • oracle.integration.platform.kernel - Startup issues

    • oracle.integration.plaform.blocks.mesh - Message routing

    • oracle.integration.platform.common - Metadata/MDS

    • oracle.integration.platform.instance - Instance

    • oracle.integration.platform.instance.activity - Instance

    • oracle.integration.platform.instance.store - Instance

    EDN
    • oracle.integration.platform.blocks.event
    • oracle.integration.platform.blocks.event.saq

    • oracle.integration.platform.blocks.event.jms

    Deployment oracle.integration
    Oracle Mediator
    • oracle.soa.mediator.common - Logs the processing events related to Oracle Mediator audit message persistence.
    • oracle.soa.mediator.common.cache - Metadata cache (RuntimeMetadataCache) and runtime related cache (RuntimeCache).

    • oracle.soa.mediator.common.error - Logs information related to error enqueuing and handling.

    • oracle.soa.mediator.common.error.recovery - Logs only the error recovery processing. All fault policy handlers are part of this logger.

    • oracle.soa.mediator.common.listener - Processing related to common infrastructure for Oracle Mediator parallel routing rules and resequencer.

    • oracle.soa.mediator.common.message - Logs information related to creation and modification of the Oracle Mediator message. (Oracle Mediator wraps a normalized message into the Oracle Mediator message.)

    • oracle.soa.mediator.common.persistence - Logs information related to the persistence of Oracle Mediator deferred messages to the database.

    • oracle.soa.mediator.dispatch - Logs everything related to message routing inside Oracle Mediator. This includes sequential, parallel, and dynamic routing.

    • oracle.soa.mediator.dispatch.db - Logs information related to the deferred message and container ID infrastructure.

    • oracle.soa.mediator.dispatch.resequencer.toplink - Logs information related to database interactions of the resequencer. This includes execution of all three resequencer strategies.

    • oracle.soa.mediator.monitor - Logs all events related to instance tracking in Oracle Mediator.

    • oracle.soa.mediator.resequencer - Logs everything related to Oracle Mediator resequencer processing such as locker-worker infrastructure, group, and sequencer ID expression evaluation.

    • oracle.soa.mediator.resequencer.besteffort - For best effort strategy of the resequencer; in particular, the locking stage processing.

    • oracle.soa.mediator.resequencer.fifo - For the FIFO strategy of the resequencer.

    • oracle.soa.mediator.resequencer.standard - For standard strategy of the resequencer.

    • oracle.soa.mediator.service - Logs events related to the processing of various message exchange patterns (one way, two way, synchronous, asynchronous, and so on) in Oracle Mediator.

    • oracle.soa.mediator.service.common.functions - Oracle Mediator XPath extension function-related logs.

    • oracle.soa.mediator.service.filter - Evaluation of filter criteria for routing rules.

    • oracle.soa.mediator.service.transformation - Logs transformation-related events such as the payload before and after the transformation.

    • oracle.soa.mediator.serviceEngine - Logs lifecycle events of the Oracle Mediator service engine. Also logs the entry and exit of messages for Oracle Mediator.

    Oracle BPEL Process Manager
    • oracle.soa.bpel
    • oracle.soa.bpel.console

    • oracle.soa.bpel.engine

    • oracle.soa.bpel.engine.activation

    • oracle.soa.bpel.engine.agents

    • oracle.soa.bpel.engine.bpel

    • oracle.soa.bpel.engine.compiler

    • oracle.soa.bpel.engine.data

    • oracle.soa.bpel.engine.delivery

    • oracle.soa.bpel.engine.deployment

    • oracle.soa.bpel.engine.dispatch

    • oracle.soa.bpel.engine.sensor

    • oracle.soa.bpel.engine.translation

    • oracle.soa.bpel.engine.ws

    • oracle.soa.bpel.engine.xml

    • oracle.soa.bpel.entity

    • oracle.soa.bpel.jpa

    • oracle.soa.bpel.system

    Oracle BPM Suite oracle.bpm.bpmn.engine
    Oracle B2B
    • oracle.soa.b2b.apptransport - Runtime logger
    • oracle.soa.b2b.engine - Runtime logger

    • oracle.soa.b2b.transport - Runtime logger

    • oracle.soa.b2b.ui - User interface logger

    • oracle.soa.b2b.repository - Repository access detailed logger

    Oracle adapters oracle.soa.adapter
    Oracle Web Services Manager (OWSM) oracle.wsm - Defaulted to the ERROR level; logs all WSM-0xxxx errors. TRACE:32 results in significant details.

  4. From the Oracle Diagnostic Logging Level (Java Level) list, change the logger level to one of the following settings:

    • TRACE:1 (FINE)

    • TRACE:16 (FINER)

    • TRACE:32 (FINEST) - Most verbose level (recommended for troubleshooting)

    The change takes effect within several minutes.

For information about monitoring Oracle Business Activity Monitoring (BAM) logs, see Section 25.6, "Monitoring Oracle BAM Logs." For information about monitoring Oracle User Messaging Service (UMS) logs, see Section 28.2, "Viewing Log Files."

B.1.1 Log Files and Thread Dumps from All Managed Servers

Table B-1 describes the log files to view and thread dumps to obtain.

Table B-1 Log Files and Thread Dumps

Output Description

Server diagnostic log

View the following file:

$DOMAIN_HOME/servers/server_name/logs/server_name-diagnostic.log

For example, soa_server1-diagnostic.log, if server_name is soa_server1.

This is where the log output is available. By default, only the last 100 MB of the diagnostic logs are retained.

Server log

server_name.log (for example, soa_server1.log, if server_name is soa_server1)

Server console output

<stdout> is also useful, especially for deployment and patching issues.

Server thread dump

Enter the following at the operating system command prompt:

kill -3 managed_server_process_ID

You can also use Oracle WebLogic Server Administration Console.

  1. In the navigation tree of Oracle WebLogic Server Administration Console, select Environment > Servers.

  2. In the table, select the server.

  3. Select the Monitoring tab.

  4. Select the Threads tab.

  5. Click Dump Thread Stacks.

The output is in the console logs.

OWSM message log

The following log captures all SOAP messages on the wire.

$DOMAIN_HOME/servers/server_name/logs/owsm/msglogging/diagnostic.log

This log is not enabled by default. To enable this log:

  1. Go to Fusion Middleware Control > Weblogic Domain > Web Services > Policies.

  2. Select the security level for which to enable logging.

  3. Edit the policy to enable the log assertion.


B.2 Parallel Purging and Table Partitioning Issues

This section describes how to troubleshoot parallel purge and table partitioning issues. Audit level recommendations are also provided.

For more information, see Chapter 10, "Managing Database Growth."

B.2.1 Executing the Parallel Purge Script

This section addresses situations in which the Oracle SOA Suite database has not been managed and space must be reclaimed. This section also describes how to identify the appropriate parameter settings for normal, daily execution.

When trying to reclaim space, the purge scripts must be executed frequently to delete substantially more SOA composite applications than are being created. This has implications for host and database resources that must be monitored to avoid lengthy wait queues.

There must be a clear goal on the number of SOA composite applications to delete. Otherwise, the purge script can be configured too aggressively. The following aspects are described:

B.2.1.1 Specifying the Degree of Parallel Value

When specifying the appropriate degree of parallel (DOP) value, hardware resources must be considered. The DOP identifies the number of jobs scheduled by the database and the number of processes concurrently executed on the host.

The parallel purge jobs are I/O intensive, which means that they are not entirely CPU-bound. However, the following formula is a starting point after which testing and monitoring are required to identify any increases:

DOP <= PARALLEL_THREADS_PER_CPU x CPU_COUNT

While purge jobs are run concurrently, they are actually executed with a 30-second incremental delay to avoid resource contention. The delay is not meant to avoid row contention because the purge jobs must not contend for the same SOA composite applications. This delay has implications for the elapsed time of the purge jobs. As an example, a DOP of 10 means that the tenth purge job sleeps for 270 seconds before starting (that is, (DOP - 1) * 30 sec).

If a high DOP number is specified, then the max_runtime parameter must be set appropriately.

B.2.1.2 Parsing SOA Composite Applications to Delete

The temptation to delete as many SOA composite applications as possible in each execution of the purge script is understandable when trying to reclaim space. However, the script must parse the data for the appropriate SOA composite application, which can monopolize the elapsed time of the purge.

The strategy must be to reduce the data by running the purge script often with modest date and max_count parameter settings. This reduction requires the parallel purge to be scheduled frequently, perhaps twice an hour.

The set of SOA composite applications to parse is identified by the following parameters:

  • min_creation_date

  • max_creation_date

  • retention_date

  • max_count

A modest max_count parameter setting is the most effective method for reducing parse elapsed times. Set the max_count parameter low, perhaps to 50K, to begin until an optimal value can be identified. As the tables start to reduce in size, the max_count parameter and date ranges can be increased.

Note:

This method of reduction is not always effective. On occasions, you may need to introduce parallel query to parse larger amounts of data.

B.2.1.3 Using Parallel Query Slaves

The parsing for candidate SOA composite applications to delete is primarily performed by several INSERT ... SELECT statements. The larger the set of SOA composite applications to parse, the more the database cost-based optimizer favors a full table scan. When large portions of a table must be traversed, a full table scan can be faster than index access because it performs multiblock I/O calls.

A parallel query can significantly improve full table scan performance by coordinating and distributing the work over multiple slave processes. Additionally, the PARALLEL hint on the INSERT portion executes in APPEND mode. This further improves performance through direct path inserts.

As with the scheduled purge jobs, the parallel query slaves are I/O intensive and not entirely CPU bound. However, the following formula is a good starting point when selecting a DOP for the PARALLEL hint:

DOP <= PARALLEL_THREADS_PER_CPU x CPU_COUNT

Note:

There is a DOP setting to identify the number of parallel purge jobs and a DOP setting to identify the number of parallel query slaves. These DOP settings both impact the amount of processes executed on the host, but not simultaneously. The INSERT ... SELECT statement executes first to parse the data followed by the scheduling of the parallel purge jobs.

For the INSERT... SELECT statement, the PARALLEL hint can be specified after the INSERT keyword, after the SELECT keyword, or both. Therefore, parallelism of the INSERT and SELECT operations are independent of each other. The appropriate DOP for the entire statement is identified by the following precedence rule:

  • PARALLEL hint on INSERT portion of the statement.

  • Set at session.

  • DOP on INSERT table.

  • Maximum DOP set on any table in the statement.

To execute parallel DML, the following statement must be executed within the script:

ALTER SESSION FORCE PARALLEL DML STATEMENT

For example, this statement has a DOP of 2 for the INSERT and 4 for the SELECT. However, given the precedence rules, the DOP for the entire statement is 2:

INSERT /*+ PARALLEL(TBL_INS,2) */ INTO TBL_INS
SELECT /*+ PARALLEL(TBL_SEL,4) */ * FROM TBL_SEL;

Note:

A parallel query can also be applied to the DELETE and SUBSELECT commands in the purge script, if required.

B.2.1.4 Debugging and Tracing Purging Operations

This section describes debugging and tracing issues.

B.2.1.4.1 Debugging Purging Operations

The purge scripts are instrumented with a DEBUG flag that provides detailed, timestamped information on each operation. This information can be evaluated to identify which tables are difficult to delete and may be better managed outside of the purge scripts through table partitioning.To set the DEBUG flag:

  1. Create an operating system directory in which to write the debug logs.

    mkdir -p /,,/debuglogs
    
  2. Connect to the database with the SYSDBA account to create the database directory.

    SQL> CREATE OR REPLACE DIRECTORY SOA_PURGE_DIR AS '/../DEBUGLOGS';
    
  3. Grant privileges to the Oracle SOA Suite schema owner:

    SQL> GRANT READ, WRITE ON DIRECTORY SOA_PURGE_DIR TO DEV_SOAINFRA;
    
  4. Connect as the Oracle SOA Suite schema owner and set the DEBUG flag.

    cd MW_HOME/SOA_ORACLE_HOME/rcu/integration/soainfra/sql/soa_purge/
    SQL> @DEBUG_ON.SQL;
    
  5. Execute the purge script:

    SQL> SET SEVEROUT ON
    SQL> EXECUTE_PURGE
    

    Note:

    As an alternative, use the SQL*Plus SPOOL command to capture debug information.

    The Automatic Workload Repository (AWR) reports for the period of the purge can also help to identify the SQL with the longest elapse and execution times.

B.2.1.4.2 Tracing Purging Operations

If SQL tracing is required to diagnose the performance of the parallel purge scripts, then the following is required:

  1. Edit procedures soa.delete_instances_in_parallel and soa.delete_insts_in_parallel_job to add the SQL trace (10046):

    SQL> EXECUTE IMMEDIATE 'ALTER SESSION SET EVENTS ''10046 TRACE NAME CONTEXT
    FOREVER,LEVEL 12''';
    

    Note:

    All quotes are single quotes.

    This creates a SQL trace for the purge session and the scheduled purge jobs (J000) in the database user dump destination.

    SQL> SHOW PARAMETER DUMP
    
  2. Identify the trace for the main database session by searching the trace files for string dbms_scheduler.create_job or delete_instances_in_parallel.

    grep -i dbms_scheduler.create_job *
    

    For example:

    TESTORA2_ora_3893.trc
    

    The number of scheduled jobs depends on the DOP. For example, a DOP of 4:

    • TESTORA2_j000_9585.trc

    • TESTORA2_j001_9587.trc

    • TESTORA2_j002_9604.trc

    • TESTORA2_j003_9606.trc

    If parallel query slaves have been customized into the purge script, then there are trace files (P000) for each. The parallel query trace files are less important because the query plans in the other trace file account for their execution. For example:

    PSOAORA2_p000_4284.trc
    
  3. Execute TKPROF on the trace files and sort by the elapsed time to identify the most costly SQL:

    TKPROF TESTORA2_ORA_3893.TRC ../OUT/ TESTORA2_ORA_3893.OUT WAITS=YES
    SYS=YES AGGREGATE=YES SORT=EXEELA
    

    For more information about TKPROF, see Oracle Database Performance Tuning Guide.

B.2.2 Oracle SOA Suite Table Partitioning

Oracle table partitioning addresses many concerns about the maintenance of large tables by decomposing them into smaller, more manageable segments called partitions. The SOA schema has been instrumented with partition keys to take advantage of the Oracle database range partitioning.

For more information about partitioning, see Section 10.4, "Partitioning Component Tables."

B.2.2.1 Referential Integrity and Equipartitioning

For performance reasons, the Oracle SOA Suite service components (excluding Oracle B2B) have no foreign key constraints to police referential integrity. Without these constraints, the relationship between master and detail tables must be protected to avoid dangling references in the detail tables.

Equipartitioning is a database partitioning feature that partitions master and detailed tables based on their foreign key constraint. This feature creates table partitions that group related master and detail rows in the same date range. This grouping ensures that no dangling references are created when partitions are dropped.

To mimic this feature, Oracle SOA Suite is instrumented to push the creation date of the SOA composite application instances down into the component master tables (Oracle BPEL Process Manager, Oracle Mediator, and human workflow). Because the SOA Infrastructure is the top level component on which all other components depend, this method groups all dependent rows by date range.

For example, the timestamp in the CREATED_TIME column of the COMPOSITE_INSTANCE table is pushed down into the following tables:

  • Oracle BPEL Process Manager master table CUBE_INSTANCE column CPST_INST_CREATED_TIME, which is then pushed down into dependent tables such as CUBE_SCOPE.CI_PARTITON_DATE.

  • Oracle Mediator master table MEDIATOR_INSTANCE column COMPOSITE_CREATION_DATE, which is then pushed down into dependent tables such as MEDIATOR_CASE_INSTANCE.MI_PARTITION_DATE.

Notes:

  • Dependent tables must be partitioned with their master and, in all cases, the top level SOA Infrastructure COMPOSITE_INSTANCE table must be partitioned.

  • To complete equipartitioning, the table partitions must all share the same name and date range.

In summary, equipartitioning means that the associated dependent table rows are in a partition with the same partition key range as their master table rows. Therefore, the state of each detail row in the equipartition can be inferred from its associated master table row.

B.2.2.2 SOA Composite Application Range-Hash Partitions

Range partitions can have their rows hashed into a subpartition to implement SOA composite application range-hash partitioning. Hash subpartitions can benefit in distributing I/O, but it is not currently recommended when partitioning Oracle SOA Suite tables.

Hashing of keys is a method that works well with equality predicates (=, IN), and there are several tables with keys that are good candidates such as the Oracle BPEL Process Manager CIKEY table. However, range-hash partitions alone do not convey the uniqueness of the hash key. For the cost-based optimizer to identify uniqueness, the query must search all partitions of a table (a full table scan). Partition pruning helps by narrowing the range of partitions to search, but Oracle SOA Suite does not use this feature. (See Section B.2.2.5, "Partition Pruning.") Therefore, to avoid a full table scan when the cost-based optimizer identifies uniqueness, indexes are searched that bypass the hash subpartition.

B.2.2.3 Interval Partitioning

Interval partitioning is an extension of range partitioning that instructs the database to automatically create partitions as required.

The verification scripts currently require the partition name to be the same across all tables in a group. The interval partitioning system generates partition names that differ for each partition. Therefore, this is not supported.

B.2.2.4 Global Hash Indexes

The use of global hash indexes is independent of Oracle SOA Suite table partitioning.

Oracle SOA Suite is constantly inserting rows, and many of the index keys are monotonically increasing. Indexes of a BTREE structure insert these keys and target only a few database blocks that can become very hot across an Oracle Real Application Clusters (Oracle RAC) setup. This is typically seen in the AWR reports as excessive buffer busy waits. To distribute the index keys randomly across many database blocks, use global hash indexes.

B.2.2.5 Partition Pruning

When the range partition keys are used as predicates, the optimizer can prune the number of partitions to search. Oracle SOA Suite and especially Oracle Enterprise Manager Fusion Middleware Control do not presently use partition keys in SQL queries and therefore do not take advantage of this performance feature.

With the requirements of equipartitioning, the partition keys are populated with the SOA composite application creation date that is not the creation date of the components. The components (Oracle BPEL Process Manager, Oracle Mediator, and human workflow) have their own creation dates that are used in the console queries:

Example for Oracle BPEL Process Manager:

  • Partition key: Table CUBE_INSTANCE column CPST_INST_CREATED_TIME

  • Creation date: Table CUBE_INSTANCE column CREATION_DATE

B.2.2.6 Purging Partitions

Circumstances may arise that require the partitioned tables to be purged, especially if the row migration scripts are used. This defeats the purpose of partitioning the table for maintenance, which can remove data in bulk through the ALTER TABLE ... DROP PARTITION command.

Attempting to purge the partitioned tables most likely causes the parsing performance problem. If partitions must be purged, then also shrink the partitions. For more information, see Section 9.5, "Understanding Space Management."

B.2.3 Reducing Audit Levels

This section provides tuning information about how to reduce the audit level for SOA composite applications and the data written to the Oracle SOA Suite schema.

B.2.3.1 Setting the Audit Level for Production Environments

Set the Audit Level to Production for all production environments on the SOA Infrastructure Common Properties page in Oracle Enterprise Manager Fusion Middleware Control. This should be the standard setting for all Oracle SOA Suite domains in production. Each component can be set independently of the SOA composite application level. However, Production should be the minimal setting. Changing the Audit Level from Development to Production can result in performance improvements (without tuning the individual component audit settings).

If you can disable auditing (provided the business environment allows for it), there are substantial performance improvements. Changing Production to Off can result in improvements (without tuning the individual component audit settings). However, if you cannot disable auditing, Oracle recommends that you consider the following steps to reduce audit levels.

For synchronous BPEL components in which the audit trail is not required, use the following settings:

  • inMemoryOptimization = true

  • completionPersistPolicy = faulted

With these settings, the synchronous process only appears in the audit trail if the instance faults.

For information about setting inMemoryOptimization and completionPersistPolicy, see Chapter "Deployment Descriptor Properties" of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

For more information about setting the Audit Level, see Section 3.1, "Configuring SOA Infrastructure Properties."

Note:

Use these settings for synchronous BPEL processes and not for any type of asynchronous BPEL processes.

B.2.3.2 Disabling Instance State Tracking

You can reduce the audit level by disabling instance state tracking. Deselect the Capture Composite Instance State checkbox on the SOA Infrastructure Common Properties page. This provides the following benefits:

  • A performance improvement because the state is not captured in the database.

  • Less data in the database for purging.

The implication is that although you may still see SOA composite application instances in Oracle Enterprise Manager Fusion Middleware Control, the state of instances is displayed as a question mark (?). Therefore, you cannot identify from Oracle Enterprise Manager Fusion Middleware Control whether the SOA composite application has completed. The state column does not reflect the states of the individual components (Oracle Business Process Management, Oracle BPEL Process Manager, human workflow user task, Oracle Mediator, and so on) within the SOA composite application. The state of individual components can be tracked by going to the respective component pages (for example, going to soa-infra > Service Engines > BPEL in Oracle Enterprise Manager Fusion Middleware Control).

Each component also has its own audit level that can be set because the initial setting is Inherit, which inherits the global audit setting. These settings can be set in Oracle Enterprise Manager Fusion Middleware Control by selecting soa-infra > SOA Administration > BPMN Properties, BPEL Properties, or Mediator Properties.

You can further tune auditing by setting the SOA composite application audit level. It is set to Inherit by default and inherits the respective component auditing level. For example, setting the following properties as follows:

  • Set the Audit Level on the SOA Infrastructure Common properties page to Production.

  • Deselect the Capture Composite Instance State checkbox.

  • Set the Audit Level on the BPEL Service Engine Properties page to Minimal.

  • Set the Audit Level for the SOA composite application to Inherit.

  • Set inMemoryOptimization and completionPersistPolicy to true and faulted, respectively (for synchronous processes only).

These settings reduce the SOA composite application data traffic and improve purging times.

For more information about setting the Audit Level and deselecting the Capture Composite Instance State checkbox on the SOA Infrastructure Common Properties page, see Section 3.1, "Configuring SOA Infrastructure Properties."

For information about setting inMemoryOptimization and completionPersistPolicy, see Chapter "Deployment Descriptor Properties" of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

B.3 Connection and Transaction Timeout Troubleshooting

This section describes how to troubleshoot connection and transaction timeout issues.

B.3.1 Resolving Connection Timeouts

You can receive a connection timeout error under circumstances such as the following:

  • You run a SOA composite application with a large payload that takes more than 30 seconds to process.

  • You are invoking a stress test using a large payload from the Test Web Service page of Oracle Enterprise Manager Fusion Middleware Control.

  • You are passing a large number of message files (one million) into a composite with a file adapter service.

  • You are retrieving instance and fault count metrics in Oracle Enterprise Manager Fusion Middleware Control.

To avoid receiving timeout errors, increase the transaction timeout property as follows:

  1. Log into Oracle WebLogic Administration Console.

  2. Click JTA.

  3. Change the value of Timeout Seconds (the default is 30).

  4. Click Save.

  5. Restart Oracle WebLogic Server.

B.3.2 Resolving Email Notification Timeouts

You can receive a connection timeout errors when a large number (for example, 200,000) of notification emails are sent.

To avoid these timeout errors:

Restart Oracle WebLogic Server after making these changes.

B.3.3 Increasing Database Connection Values

You can receive the error message shown in Example B-1 because of slow connections to the database.

Example B-1 Slow Database Connections Error

Exception [TOPLINK-4002] (Oracle TopLink - 11g Release 1 (11.1.1.1.0) (Build 
090304)): oracle.toplink.exceptions.DatabaseException 
Internal Exception: java.sql.SQLException: Internal error: Cannot obtain 
XAConnection weblogic.common.resourcepool.ResourceDeadException: Pool 
SOADataSource has been disabled because of hanging connection tests, cannot 
allocate resources to applications.

If this occurs, perform the following steps:

  1. the DOMAIN_HOME\bin\setSOADomainEnv.cmd file on Windows.
  2. Open the DOMAIN_HOME\bin\setSOADomainEnv.cmd file.

  3. Uncomment the lines shown in bold.

    # 8331492: Value of weblogic.resourcepool.max_test_wait_secs is 10 
    # seconds. It can be increased by uncommenting line below if your database 
    # connections are slow. See SOA documentation for more details. 
     EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
    -Dweblogic.resourcepool.max_test_wait_secs=30" 
    export EXTRA_JAVA_PROPERTIES 
    
  4. Save your changes and restart the managed Oracle WebLogic Server.

B.3.4 Updating the EJB Transaction Timeout Value in the Deployment Archive After SOA Infrastructure Failure

Updating the transaction timeout value for the FacadeFinderBean property in Oracle WebLogic Server Administration Console under Deployments > expanded SOA Infrastructure Application > FacadeFinderBean > Configuration tab can result in the error shown in Example B-2 after restarting the SOA Infrastructure:

Example B-2 EJB Transaction Timeout Error

java.lang.IllegalArgumentException: Cannot convert value of type [$Proxy223
implementing
 oracle.bpel.services.workflow.verification.IVerificationService,org.springframe
work.aop.SpringProxy,org.springframework.aop.framework.Advised]
to required type
[oracle.bpel.services.workflow.verification.IVerificationService] for
property 'verificationService': no matching editors or conversion strategy found
Message icon - Warning Errors were encountered while performing this operation.

The SOA Infrastructure status is also displayed as failed.

This error is not specific to FacadeFinderBean; it also applies to any EJB that is part of the SOA Infrastructure application.

To resolve this error, you must manually modify the transaction timeout setting in your deployment archive.

To update the transaction timeout setting:

  1. Open the fabric-ejb.jar file in your deployment archive.

  2. Increase the transaction timeout value in the META-INF/weblogic-ejb-jar.xml file to a larger value.

  3. Rejar the file.

  4. Restart the managed server that includes the SOA Infrastructure by following the instructions in Section 3.2, "Stopping and Starting the Managed Server and SOA Infrastructure."

    Note:

    This issue may also occur while updating any EJBs deployed as part of the SOA Infrastructure application. If this issue occurs, you must update the corresponding contained JAR file for those EJBs in a similar fashion.

B.3.5 Long Running, Synchronous Calls To Remote Web Services Error Out or Asynchronous Transactions Return with an Error after a Long Time

You may have long running, synchronous calls to remote web services that end with JTA transaction rolled-back errors. When executing a transaction making an asynchronous call (for example, to the SOA server), the application may return with an error. The server logs show JTA transaction timeouts, which can cause this behavior.

To check the JTA transaction timeout in the Oracle WebLogic Server Administration Console.

  1. Log in to Oracle WebLogic Server Administration Console.

  2. In the Domain Structure, select Services > JTA to check the timeout value.

    If the transaction is always timing out beyond 30 seconds and is a custom composite synchronous client invocation, then you may need to revisit the design approach. It may be best for the external web service to be invoked as an asynchronous transaction.

    Increasing the JTA for supporting long running, synchronous transactions is simply an interim mechanism. For information on changing the JTA transaction timeout setting, see Section B.3.1, "Resolving Connection Timeouts."

  3. If synchronous client invocations take a long time, check for any performance issues with the system and try to resolve them.

  4. Check for appropriate values for the syncMaxWaitTime property and BPEL's EJB transaction timeout settings in relation to the JTA timeout settings and only then increase the value of the JTA timeout, if needed.

    For information on viewing and changing the syncMaxWaitTime property, see Section 13.1, "Configuring BPEL Process Service Engine Properties."

  5. To view and change the BPEL EJB transaction timeout settings, perform the following steps:

    1. Log in to Oracle WebLogic Server Administration Console.

    2. In the Domain Structure, click Deployments.

    3. Expand soa-infra > EJBs.

    4. Update the following EJBs:

      - BPELActivityManagerBean

      - BPELDeliveryBean

      - BPELDispatcherBean

      - BPELEngineBean

      - BPELFinderBean

      - BPELInstanceManagerBean

      - BPELProcessManagerBean

      - BPELSensorValuesBean

      - BPELServerManagerBean

    5. Click Save.

    6. Restart Oracle WebLogic Server.

  6. For asynchronous transactions, check the values for both the BPEL EJB transaction timeout and the JTA transaction timeout and adjust as needed.

B.4 Runtime Diagnostics Troubleshooting

This section describes how to troubleshoot runtime issues.

B.4.1 Oracle SOA Suite Runtime Failure with a "Cannot read WSDL" Error

The following error is displayed if either the endpoint is not available for a reference or a composite is deployed with an incorrect deployment plan file.

oracle.fabric.common.FabricException: Cannot read WSDL

To diagnose this issue:

  1. Ensure that the endpoint for the reference is up and running.

    The WSDL or endpoint is stated in the error. The service can then be looked up from Oracle Enterprise Manager Fusion Middleware Control to check if it is active. Once the service is active and the endpoint is reachable, search for the SOA instance and retry it through the Recovery tab for the BPEL process service engine in Oracle Enterprise Manager Fusion Middleware Control.

  2. For extensions/customizations, ensure that the correct URL is updated in the deployment configuration plan.

    For information about deployment plans, including examples of using sca_extractPlan to extract plans, see the "Customizing Your Application for the Target Environment Before Deployment" section of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

B.4.2 Automatic Recovery of BPEL Instances is Not Recovering A Specific Instance

BPEL processes have an automatic recovery feature that attempts to automatically recover activities that are recoverable such as unresolved invoke/callback messages, activities not completed over a provided threshold time, and so on. However, the automatic recovery feature only tries to recover a few instances and only retries a fixed number of times. If some instances are not being automatically recovered, they are likely not being picked up because of the configuration of the automatic recovery parameters.

To resolve this issue:

  • Set the maximum number of messages to automatically recover.

    By default, the automatic recovery feature of Oracle BPEL Process Manager processes 50 messages to submit for each recovery attempt. This is controlled by the maxMessageRaiseSize property.

    1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

    2. Select SOA Administration > BPEL Properties > More BPEL Configuration Properties > RecoveryConfig.

    3. Expand both RecurringScheduleConfig > maxMessageRaiseSize and StartupScheduleConfig > maxMessageRaiseSize.

      The default value is 50 for each. A negative value causes all messages selected from the database to be submitted for recovery. A value of 0 causes no messages to be selected from the database (effectively disabling recovery). To recover more than 50 messages, set the property value to that value. Use this property to limit the impact of recovery on the server.

  • Set the maximum number of automatic recovery attempts on a given message.

    You can also configure the number of automatic recovery attempts to submit in the same recoverable instance. The value you provide specifies the maximum number of times that invoke and callback messages are recovered. If the value is 0 (the default value), it recovers all messages. Once the number of recovery attempts on a message exceeds the specified value, a message is marked as nonrecoverable.

    To configure automatic recovery attempts for invoke and callback messages in Oracle Enterprise Manager Fusion Middleware Control:

    1. In the navigation pane, right-click soa-infra (SOA_cluster_name).

    2. Select SOA Administration > BPEL Properties > More BPEL Configuration Properties.

    3. Select MaxRecoverAttempt, and enter a value in the Value field.

    4. Click Apply.

      It may not be desirable in all cases to use automatic recovery. If services are not idempotent, then corruption can occur. Moreover, the automatic recovery restores the composite to the last save point that can be immediately after an asynchronous invoke, wait, and so on. Therefore, it is important to understand the process behavior and what it does next before performing mass recoveries. Attempt mass automatic recovery only after the root cause of the composite failures has been fixed (for example, a service that was unavailable is now available, a database running out of space was fixed, and so on). Automatic recovery can also trigger an unexpected load during failure scenarios. This causes more threads to block on a remote server that can induce hangs in the SOA server in a cascading fashion.

      For more information, see Section 13.2, "Configuring Automatic Recovery for Oracle BPEL Process Manager."

B.4.3 Some Composites Are Retried Multiple Times on Failure

When a BPEL process flow errors out, it is retried with all its invocations. This is undesirable in some cases.

The property GlobalTxMaxRetry (default value is 3) specifies how many retries are performed if an error is identified as a retriable one. For example, after several web service invocations, if dehydration fails due to a data source error, then this is identified as a retriable error and all activities from the prior dehydration state are retried. If the activities being retried are not idempotent (that is, their state can change with each retry and is not guaranteed to give the same behavior), then multiple retries can be problematic.

To rectify this situation, customize the composite by specifically marking the nonidempotent activities with idempotentset set to false in the partner link settings section of the composite.xml file to prevent retries.

<property name="bpel.partnerLink.partner_link_name.idempotent">false</property>

You cannot set the idempotent property in Oracle Enterprise Manager Fusion Middleware Control.

You can also set GlobalTxMaxRetry to 0 in the Systems MBean Browser.

To set GlobalTxMaxRetry:

  1. Right-click soa-infra (SOA_cluster_name).

  2. Select SOA Administration > Common Properties.

  3. Click More SOA Infra Advanced Configuration Properties.

  4. Click GlobalTxMaxRetry.

  5. In the Value field, enter an appropriate value.

  6. Click Apply.

For more information about the idempotent property, see the "BPEL Properties Set Inside a Composite" section of Oracle Fusion Middleware Performance and Tuning Guide and the "Managing Idempotence at the Partner Link Operation Level" section of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

B.4.4 Application Transaction Does Not Complete and the Underlying Composite Is Stuck in a Running State

An application transaction is not completing. For example, a purchase order status may remain processing. Checking the composite instance shows that the composite is stuck in the running state. In this case, the component is probably not running. Instead, it has likely faulted and may need recovery.

Faults may occur for various reasons:

  • A BPEL activity faulted with an error (for example, a business error, security authorization error, or some other error).

  • A BPEL activity invoked an external web service that was unavailable.

  • A BPEL activity has already been terminated by the administrator using Oracle Enterprise Manager Fusion Middleware Control.

  • A BPEL activity invoked an asynchronous ADF service and the message is stuck in the AQ/JMS queue.

  • A BPEL activity invoked an asynchronous ADF service, but because Oracle SOA Suite was unavailable, the callback message did not arrive.

  • A BPEL activity invoked a synchronous ADF service, which is taking a long time (or is hanging).

  • A network error occurred.

To diagnose this issue:

  1. Log in to Oracle Enterprise Manager Fusion Middleware Control.

  2. In the navigator pane, go to domain_name > SOA.

  3. Click soa-infra (SOA_cluster_name).

  4. Click the Instances tab.

  5. Search for the composite instance, and click the instance ID.

    The Flow Trace page appears.

    If the instance is not visible (and the Audit Level is not set to Off on the SOA Infrastructure Common Properties page), this implies that the message is stuck outside of Oracle SOA Suite.

    If the message has reached Oracle Mediator, but not instantiated the BPEL flow, the BPEL instance may have been rolled back from the start due to an error.

    If the BPEL flow exists, the Faults section of the Flow Trace page typically shows the faulted service that can trace the root cause.

  6. In the Trace section, click the BPEL process.

  7. Expand the BPEL audit trail to see the exact point at which the service faulted.

    This information is also available in the Faults tab of the BPEL flow trace. It also indicates whether the fault can be recovered.

  8. Click the View Raw XML link.

    The same information is also available through this link, where you can see the error. For example:

    ...
    <message>Faulted while invoking operation "modifyUserRoles" on provider
     "UserService". </message><details>
    ...
    <tns:message>JBO-27023: Failed to validate all rows in a
     transaction.</tns:message>
    <tns:severity>SEVERITY_ERROR</tns:severity>
    ...
    <message>The transaction was rolled back. The work performed for bpel instance
     "451042" was rolled back to the previous dehydration point, but the audit
     trail has been saved.
    You can recover the instance from the recovery console by resubmitting the
     callback message or activity for execution.</message
    

    Since the instance was rolled back to its previous dehydration point, the status remains as Running.

  9. In the Audit Trail and Faults tabs, make a note of the following:

    • Composite name (for example, POComposite)

    • Component (for example, UpdateGuid BPEL process)

    • BPEL instance ID (for example, bpel:451042)

    This is all used in the recovery of the instance, if it is recoverable. The Audit Trail window may mark the error as a nonrecoverable business fault, but the recoverability of the message can be found in the Recovery tab of the BPEL process service engine.

  10. To attempt to recover the instance, right-click soa-infra (SOA_cluster_name), and select Service Engines > BPEL.

  11. Click the Recovery tab.

  12. From the Type list, select Activity.

  13. Specify the composite and component names captured in step 9, and click Search.

  14. Find the specific BPEL instance ID. You can recover faults marked as Recoverable.

  15. Check the other recovery options in the Type list (for example, Invoke and Callback), if they exist.

    Note:

    You can also search for recoverable messages from the Faults and Rejected Messages tab of the SOA Infrastructure by clicking the message and selecting the appropriate action from the Recovery Actions list.
  16. If the instance is not marked as recoverable, then reinvoking the service is not allowed (most probably because it is not idempotent). In some cases, you may need to provide diagnostic information to Oracle Support Services to resolve issues with nonrecoverable, nonidempotent transactions.

  17. If the BPEL activity has invoked an asynchronous ADF service and the message is stuck in the AQ JMS queue, you can view the server.log and server-diagnostic.log files to see the logging of the message metadata logged by the JRF web services infrastructure.

    In addition, the ADF diagnostic logs are also available to debug, if needed.

    1. Use the ECID field to correlate and track ADF service logging corresponding with the SOA composite application that invoked it.

    2. When viewing the log in Oracle Enterprise Manager Fusion Middleware Control, click the Broaden Target Scope dropdown list and select the farm_name/domain_name (Oracle WebLogic domain) to view messages across the domain.

    3. In the Selected Targets section of the Log Messages page for the Oracle WebLogic Server domain, ensure that the search includes the ECID field with the value noted in step a.

    4. Search and view log records for the execution content ID (ECID) and note any issues.

      For a specific ECID, you find several root instances (top level clients). You must drill down to the appropriate instance to find a specific fault.

    5. Observe if the components completed successfully or completed with an error. See the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.

See the following chapters for the various ways to recover transactions and messages:

B.5 Human Workflow Troubleshooting

This section describes how to troubleshoot human workflow issues.

B.5.1 Task Assignment/Routing/Escalation Issues

Table B-2 describes symptoms, possible causes, and possible solutions for task assignment/routing/escalation issues.

Table B-2 Troubleshooting Task Assignment/Routing/Escalation Issues

Symptom Possible Cause Possible Solution

The task completes without any assignment occurring.

The most common problem is that task assignees are specified using XPath expressions, and the expression does not evaluate to any nodes. Other problems can include incorrect skip conditions for participants.

  1. Correct any issues with the XPath expressions.

  2. Ensure that you get some results for the XPath expression for the given data.

  3. Check the skip conditions specified with the Specify skip rule checkbox for task participants in the Human Task Editor.

The business rules do not return any list builders.

When participants of a task are specified using business rules, it is expected that business rules return at least one list builder. If business rules determine that no participants are needed, the function ignoreParticipant(...) must be used. If modeled correctly and you still see this error, it is likely that none of the rules fired. See the symptom entitled '"A human workflow task chooses the incorrect user if many rules are defined or it errors with the following message:" in this table.

  1. Use the ignoreParticipant(..) function to model your rules.

  2. Ensure the rules are modeled correctly so that at least one rule is fired.

The business rules return list builders of different types.

When participants in a task are specified using business rules, it is expected that business rules return list builders of the same type.

Correct your rules.

A human workflow task chooses the incorrect user if many rules are defined or it errors with the following message:

Ruleset returned lists with
different list builder

At runtime, when a human workflow task tries to fetch the list of users, it may error out with the following error:

Ruleset returned lists with different list builder

This error is displayed in the Task Detail comments field. Alternately, the task may select a user or approver, which may not appear to be the correct or expected one. This is primarily caused by having overlapping rules. When the participants of a task are specified using business rules, it is expected that business rules return list builders of the same type.

Moreover, only one rule from a ruleset must be applicable for a transaction. In case many rules are true, the actions associated with the applicable rule with the highest priority get executed. In case multiple applicable rules have the same priority, then the first rule in the list is picked and its actions executed.

Avoid writing overlapping rules. Constraints from different list builders are different and cannot be mixed. If multiple rules get triggered with a different list builder, this error occurs. In addition, only one set of constraints is honored.

Check that all rules in the ruleset have priorities defined so that multiple rules with the same priority are not applicable for the same transaction.

For more details., see Oracle Fusion Middleware Modeling and Implementation Guide for Oracle Business Process Management.

Parallel assignees have to approve or reject the task even though the parallel completion criteria is met.

In the Add Participant Type dialog for a parallel participant, you can configure the human task during runtime to wait for all parallel participants to complete or to complete when criteria are met.

Make the correct selection for completion in the Add Participant Type dialog.

The task is assigned to the group/role when the expectation is that it goes to every user in the group/role individually.

When a group or a role is used as a task assignee, the task is assigned to the group or role directly. Task runtime does not assign it separately. One of the users in the group/role has to claim the task and work on it. When used with a parallel or serial participant, often times it is expected that this resolution to users is automatic, which it is not.

To assign separately to the members of the group or role, use the XPath functions ids:getUsersInGroup and ids:getUsersInAppRole.

A task errors out when invoking the decision service for the evaluation of routing rules or rule-based participants.

Payload validation is enabled on the SOA Infrastructure instance.

Deselect the Payload Validation checkbox for the instance. For more information, see Section 3.1, "Configuring SOA Infrastructure Properties."


B.5.2 Task Action Issues

Table B-3 describes symptoms, possible causes, and possible solutions for task action issues.

Table B-3 Troubleshooting Task Action Issues

Symptom Possible Cause Possible Solution

A user is not allowed to perform an action on a task.

The most common problem is that the user does not have permissions to perform that action on that task at that point in time.

Find out if the user can be an assignee, owner, or creator of the task, or if they are an administrator. If the user should have been allowed to perform the action, check the server log file for a detailed log message, which includes information such as the task state, task assignees, user who acquired it, permitted actions, roles played by this user for the given task, and so on.

A task is locked and cannot be updated in Oracle BPM Worklist. An ORA-30279 error is displayed (with an underlying ORA SQL exception of ORA-20001 also displayed).

Two users are trying to update the same task concurrently.

Wait for the other user to finish.


B.5.3 Notification Issues

Table B-4 describes symptoms, possible causes, and possible solutions for notification issues.

Table B-4 Troubleshooting Notifications Issues

Symptom Possible Cause Possible Solution

The task email notification is not being sent out.

Notification Mode is set to NONE on the Workflow Notification Properties page in Oracle Enterprise Manager Fusion Middleware Control.

Change this setting to Email or All. For information, see Section 21.1, "Configuring Human Workflow Notification Properties."

The email notification is not being sent out.

Incorrect outgoing server settings are used in the email driver configuration.

Check the Simple Mail Transfer Protocol (SMTP) port/SMTP host/user name/password/email values.

Tip: Validate the values by using them in any email client for connecting to the SMTP server.

Perform the following steps to verify the settings in Oracle Enterprise Manager Fusion Middleware Control:

  1. In the navigator, right-click User Messaging Service.

  2. Select usermessagingdriver-email > Email Driver Properties.

  3. Check the settings.

The notifications are sent, but are not actionable.

The Actionable Address field is not configured.

In Oracle Enterprise Manager Fusion Middleware Control, configure the Actionable Address field with a valid email address. For information, see Section 21.1, "Configuring Human Workflow Notification Properties."

Ensure that the same email address is used when configuring the incoming server setting in the Oracle User Messaging Server email driver. For information, see Section 21.1, "Configuring Human Workflow Notification Properties."

Notifications are sent, but are not actionable.

The human workflow task is not set to send actionable notifications.

In the Human Task Editor (you can double click the .task file in Oracle JDeveloper to start the editor), expand the Notification section, click the Advanced tab, and select the Make notification actionable checkbox.

Actionable notifications are sent, but no action is taken after responding.

The Actionable Address field is incorrect.

Check the IMAP/POP3 server/port values. Ensure the Actionable Address field is used in the email driver configuration.

Tip: Validate the values by using them in any email client for connecting to the IMAP/POP3 server.

Actionable notifications are sent, but no action is taken after responding.

The nondefault email client is configured for receiving notifications.

When the user clicks the approval link, the default mail client page opens, which may send emails to a different email server. Configure the default email client to receive actionable notifications.

Enter the correct value in the Actionable Email Account field of the Workflow Task Service Properties page as the incoming, actionable email account to use. The default account name is Default.

For information, see Section 21.3, "Configuring Human Workflow Task Service Properties."

Actionable notifications are sent but no action is taken after responding.

An email client is configured with the same account used in the email driver.

The mail may be downloaded and marked as read or deleted by the email client before the human workflow notification service can download and process the mail. Remove that account from the email client.

The Oracle BPM Worklist link appears in email notifications.

This is the default behavior. By default, email notifications point to Oracle BPM Worklist.

Perform the following steps:

  1. In the Notification section of the Human Task Editor, click the Advanced tab.

  2. Deselect the Show worklist URL in notifications checkbox.

Performance is slow for group notifications.

The group notification performance depends on the number of members in the group (size of group).

  1. Provide an email ID for the group in LDAP. In this case, human workflow sends one email to the group email ID, instead of individual emails to each group member.

  2. In the Notification section of the Human Task Editor, click the Advanced tab.

  3. If you want to send one email to all group members, select Send one email containing all user addresses from the Group notification configuration list. This action enables all members to see the to list and common content is sent to all members (without considering locale, and so on).

  4. If you want to send individual emails, but reuse content between members, select Send individual emails (the default selection) from the Group notification configuration list and unselect Use separate task forms based on locale. This action enables group members to receive individual mails in their locale. Task forms generated for creating notification content are reused between members in the same locale.

You may receive the following error message:

[2012-07-05T09:13:05.246-04:
00] [soa_server1] [ERROR] []
[oracle.soa.services.workflow
.common] [tid:
[ACTIVE].ExecuteThread: '6'
for queue:
'weblogic.kernel.Default
(self-tuning)'] [userId:
<anonymous>] [ecid:
dc2ff0568380b04f:23824bc7:13
8570a4e67:-8000-000000000000
1716,0] [APP: soa-infra] <.>
[[ORABPEL-0
at
oracle.bpel.services.workflo
w.task.notification.TaskNoti
fications.notifyForTask(Task
Notifications.java:481)
. . .
. . .

Check if you have apostrophes in the body of an email notification message.

<%string('À titre de responsable fonctionnel, une demande d'approbation 
. . .

Escape each apostrophe by adding double apostrophes in your string. For example, if you have 'd in your message, escape it as follows:

<%string('d'd')%>

to:

<%string('d''d')%>

B.5.4 Task View Issues

Table B-5 describes symptoms, possible causes, and possible solutions for task view issues.

Table B-5 Troubleshooting Task View Issues

Symptom Possible Cause Possible Solution

Showing custom (mapped attribute) columns in a view.

Attribute mappings are created for specific task types. The view must be associated with one or more task types to use mapped attributes.

In Oracle BPM Worklist (view/create/edit UI), specify a task type for the view in the Definition tab. The attribute labels used in the mappings for that task type are now available as columns that can be used in the view in the Display tab.

It is possible to associate a view with multiple task types. Multiple task types can be selected from the Task Type browser. If multiple task types are selected, then the attribute labels for all those task types are available for use in the view.

View grantees can view and edit tasks belonging to the view owner.

The view is shared as data. This type of sharing allows grantees to use the view as if they are the view owner, and can see and act on the view owner's task.

In the Definition tab of Oracle BPM Worklist (view/edit UI), ensure that Share View is set to Definition only, which enables grantees to use the view against their own tasks. Setting Share View to Data enables grantees to use the view against the view owner's data.

Creating a new standard view.

Only users with administration privileges can create standard views.

  1. Ensure the logged-in user has administration privileges.

  2. Define the view as usual, using Oracle BPM Worklist (create/edit UI).

  3. Check the add to standard views checkbox.

    The view is created as a standard view.

Internationalizing a standard view name.

The value specified in the name field for standard views can be used as a resource key to look up a display name from the WorkflowLabels resource bundle.

Add a new resource key to the WorkflowLabels resource bundle. The key is the name you used for the view, prefixed by STD_VIEW. (Note the required trailing period.)

For more information about resource bundles, see workflow sample workflow-110-workflowCustomizations, available with the Oracle SOA Suite Samples.

Migrating views and standard views you have created on one instance to another SOA server.

You must use the test-to-production utility.

The test-to-production utility enables you to export user views and standard views as an XML file, and to import the views from the XML file into another instance. For information about this utility, see Section 23.6, "Moving Human Workflow Data from a Test to a Production Environment."


B.5.5 Task Attribute Mapping Issues

Table B-6 describes symptoms, possible causes, and possible solutions for task attribute mapping issues.

Table B-6 Troubleshooting Task Attribute Mapping Issues

Symptom Possible Cause Possible Solution

No payload attributes are available for mapping to a public attribute in Oracle BPM Worklist.

Oracle BPM Worklist only supports creation of mappings to simple payload attributes. Only simple attributes from the task payload are displayed for creating mappings in Oracle BPM Worklist.

  • Add simple attributes to the payload.

  • Create protected mappings at design time. Mappings can be made to be complex payload attributes using XPath expressions for protected attribute mappings at design time in Oracle JDeveloper.

  • Use the RuntimeConfigService API to create public mappings using XPath expressions.

You cannot create mappings for the protected attribute label in Oracle BPM Worklist.

Protected mappings can only be created as part of the task definition at design time. Protected mappings cannot be created or updated at runtime using Oracle BPM Worklist, or the RuntimeConfigService API.

  • Use a public attribute label for the mapping.

  • Create the mapping to the protected attribute label in the task definition at design time.

You cannot see any attribute labels for which to create mappings in Oracle JDeveloper.

Design-time mappings can only be created for protected attribute labels. Ensure that protected attribute labels have been created in the SOA instance to which you are connected.

  1. Log in to Oracle BPM Worklist as an administrator.

  2. Go to Administration > Protected Flexfields.

  3. Check that protected attribute labels exist.

  4. Create any protected attribute labels as required.

Internationalizing the name of an attribute label.

You can use the attribute label name as a resource key to look up a display name from the WorkflowLabels resource bundle.

Add a new resource key to the WorkflowLabels resource bundle. The key is the name you used for the label, prefixed by FLEX_LABEL. (note the trailing period).

For more information on the resource bundle, see the workflow sample workflow-110-workflowCustomizations, available with the Oracle SOA Suite Samples.

Migrating attribute labels and mappings from one server to another.

Use the test-to-production utility.

The test-to-production utility enables you to export public attribute labels, public attribute mappings, and protected attribute labels as an XML file, and to import the labels and mappings from the XML file into another instance.

For more information, see Section 23.6, "Moving Human Workflow Data from a Test to a Production Environment."


B.5.6 Task Report Issues

Table B-7 describes symptoms, possible causes, and possible solutions for task report issues.

Table B-7 Troubleshooting Task Report Issues

Symptom Possible Cause Possible Solution

You receive the following error:

Null Pointer Exception when
running Task Productivity
Report

This is caused by an issue with the handling of dates when the worklist client locale and server default locale are different.

The workaround is to change the locale for the worklist client to be the same as the server, or to run a report without specifying dates.


B.5.7 Task History Issues

Table B-8 describes symptoms, possible causes, and possible solutions for task history issues.

Table B-8 Troubleshooting Task History Issues

Symptom Possible Cause Possible Solution

The Add Participant button is disabled.

A current or past participant is selected in the history table.

This is designed behavior. Adding adhoc participants is not allowed with the current or past participant. The current participant means the task is with that participant at that point in time.

All the added adhoc participants disappeared after a page refresh.

You may not have saved your modifications to the history table.

Ensure that you save your changes. Otherwise, all changes disappear. If you think you have saved your changes and the changes still disappear, file a bug.

Do not see future approvers in the history table.

The Future Approvers checkbox may not be selected.

Select the Future Approvers checkbox in Oracle BPM Worklist (configuration in the task sequence table).

You see the message in the history table about the correlation ID not being passed or any exception related to the correlation ID.

If the task is uninitiated, the correlation ID may not have been passed.

Ensure that you pass the correlation ID to the uninitiated task.

The edit toolbar is disabled or is not displayed.

The user may not have privileges to edit the participants.

  1. In the Assignment tab of the Human Task Editor, click the Task will go from starting to final participant icon in the upper right corner.

    The Configure Assignment dialog is displayed.

  2. Select the Allow participants to edit new participants and Allow initiator to add participants checkboxes.

You receive the following error:

<Warning>
<oracle.adf.controller.intern
al.metadata.MetadataService>
 <BEA-000000><ADFc:
 /META-INF/adfc-config.xml: > 
  <Warning>
<oracle.adf.controller.intern
al.metadata.MetadataService><
ADFC-52024> <ADFc: Duplicate
 managed bean definition for
 'aleCompBindings' detected.> 

Shared library oracle.soa.worklist.webapp is referenced in weblogic.xml and the JAR files adflibWorklistComponents.jar and adflibTasklistTaskflow.jar are packaged in the web application.

These JARs ideally should not be packaged inside the web application. They should only be referenced as a shared library. Do not package these JARs in the web application.

You receive the following error:

<Error> <Deployer>
 <BEA-149265> <Failure
 occurred in the execution of
 deployment request with ID
 '1297964056778' for task
 '3'. Error is:
 'weblogic.management.Deployme
ntException:
[J2EE:160149]Error while
processing library
references. Unresolved
application library
references, defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false].'
weblogic.management.Deploymen
tException:
[J2EE:160149]Error while
processing library
references. Unresolved
application library
references, defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false].

Shared library oracle.soa.workflow.wc is referenced in weblogic.xml, but not available on the server.

Ensure that this shared library is deployed on the server to which you are deploying your application. It may happen that the shared library is deployed, but not targeted, for that server.

You receive the following error:

java.lang.IllegalStateExcepti
on: Attempt to validate an
already invalid RegionSite: 

This is a generic exception that sometimes is displayed in the server logs (for example, AdminServer.log).

See the real exception in the diagnostic logs (for example, AdminServer-diagnostic.log) and provide that exception with the bug you file.

You receive the following error:

[AdminServer] [NOTIFICATION]
 [J2EE JSP-00008]
 [oracle.j2ee.jsp] [tid:
 [ACTIVE].ExecuteThread: '15'
 for queue:
 'weblogic.kernel.Default
 (self-tuning)'] [userId:
 weblogic] [ecid:
17011f2a001d6b0e:7e22d6ce:12e
3444eb1b:-8000-0000000000002f
0a,0] [APP: FederatedApp_
application1] unable to
 dispatch JSP page: The
 following exception
 occurred:.[[
java.lang.RuntimeException:
 Cannot find FacesContext
        at
javax.faces.webapp.UIComponen
tClassicTagBase.getFacesConte
xt(UIComponentClassicTagBase.
java:2122) 

This is a common mistake and is not related to any components you are using. You forget to put faces in the URL. For example:

http://server:port/FederatedApp
/test.jspx

Put faces in the URL as follows:

http://server:port/FederatedApp/fac
es/test.jspx

B.5.8 Task Form/Action Issues

Table B-9 describes symptoms, possible causes, and possible solutions for task form/ action issues.

Table B-9 Troubleshooting Task Form/ Action Issues

Symptom Possible Cause Possible Solution

The task form application does not have an empty JSPX page.

N/A

The task forms are now invoked using an ADF task flow and control is returned to the module that initiated the task form task flow when the task flow completes. Therefore, no empty JSPX is needed.

The task form does not load in Microsoft Internet Explorer.

Microsoft Internet Explorer has a URL length limit.

Your task form URL length is too long.

Deployment fails with a class not found exception.

The shared library entry is missing from weblogic.xml.

If you see the following error:

Caused By:
 java.lang.ClassNotFoundException:
oracle.bpel.services.datacontrol.ty
pes.Number 

during deployment of a task form, then it is likely due to the missing shared library in weblogic.xml. Add the following element in weblogic.xml:

<library-ref>
 <library-name>oracle.soa.worklist.w
 ebapp</library-name>
 <specification-version>11.1.1</spec
 ification-version>
</library-ref> 

Deployment/access of task form fails when the hostname is used.

The DNS entry is missing.

If you are using a server with DHCP, the DNS entry may be missing for the host. Therefore, deployment/access using the IP address may succeed, but deployment/access using a hostname may fail. Update your client computer by manually adding the host/IP address:

  • On Windows operating systems, this is typically is in %windir%\drivers\etc\lmhosts.

  • On Linux/UNIX, this is typically in /etc/hosts.

Task form URL protocol (HTTP or HTTPS).

You are unable to access the task form through HTTPS or HTTP.

  • DefaultToDoTaskForm: The worklist accesses the default to-do task form using the port/protocol returned by the getServerInfo API from the runtime config service. This API uses the frontend host setup for the cluster or managed server. It gives preference to the HTTPS protocol if it is enabled. It is dynamic and can be changed using Oracle Enterprise Manager Fusion Middleware Control for default to-do tasks as the other custom task form.

  • CustomTaskForm: The custom task form gets the HTTP and HTTPS information from the setup. It uses the JRF API to get the information. This gets the frontend host information from the cluster and then the managed server if the cluster frontend host is not set up. In case the managed server frontend host is not set up, it uses a regular port. If the user enables the HTTPS port during deployment, the task form stores this information in the database. After this occurs, if the user disables the HTTPS port using Oracle WebLogic Server Administration Console, then the user must remove the HTTPS port for the task form using Oracle Enterprise Manager Fusion Middleware Control.


Note:

If you receive a java.lang.OutOfMemoryError: PermGen space error when deploying multiple task forms, you may need to increase PermGen memory. For more information, see Section 7.1.1, "PermGen Memory Requirements for Multiple ADF Task Form Deployments."

B.5.9 Task Comments/Attachment Issues

Table B-10 describes symptoms, possible causes, and possible solutions for task comments/attachment issues.

Table B-10 Troubleshooting Task Comments/Attachment Issues

Symptom Possible Cause Possible Solution

The file is not getting uploaded.

The file is too big

By default, ADF has a size limit of 2000 KB for each request. Add the following parameters in web.xml to adjust the file size and temporary storage for uploaded files:

<context-param>
  <!-- Maximum memory per request
(in bytes) -->
<param-name>oracle.adf.view.faces.U
PLOAD_MAX_MEMORY</param-name>
  <!-- Use 500K -->
  <param-value>512000</param-value>
</context-param>
<context-param>
  <!-- Maximum disk space per
 request (in bytes) -->
<param-name>oracle.adf.view.faces.U
PLOAD_MAX_DISK_SPACE</param-name>
  <!-- Use 5,000K -->
 <param-value>5120000</param-value>
</context-param>
<context-param>
  <!-- directory to store temporary
 files -->
<param-name>oracle.adf.view.faces.U
PLOAD_TEMP_DIR</param-name>
  <!-- Use an ADFUploads
 subdirectory of /tmp -->
<param-value>/tmp/ADFUploads/</para
m-value>
</context-param>

The file uploaded in the task details application is not visible in the same task flow.

After uploading a file, the attachment link generated in the task form is invalid. Clicking this link returns an empty stream.

When you upload a file, you see the attachment link in the table. However, this link does not work. You must reload the task details to view the file.

Adding file attachments creates a new task version, but adding a URL attachment does not create a new version.

Inconsistent behavior of the URL attachment and file attachment.

When a file is uploaded, the task is saved because the file is uploaded to persistency storage. This creates a new task version. The URL attachments only update the local task object in the user interface application. Therefore, no task version is created.


B.5.10 Design Time at Runtime Issues

Table B-11 describes symptoms, possible causes, and possible solutions for design time at runtime issues. Two design time at runtime tools are available for use:

  • Oracle SOA Composer

  • Task Configuration tab of Oracle BPM Worklist

Table B-11 Troubleshooting Design Time at Runtime UI Issues

Symptom Possible Cause Possible Solution

Modifications made to a task in a design time at runtime tool do not appear for the task.

The task was instantiated before you actually edited it using a design time at runtime tool.

Design time at runtime updates go into effect only for instances created after the changes, and not for those that were created before the change.

Therefore, if you edit a task using a design time at runtime tool, and then instantiate a new task, the new instance of the task has the changes you made.

Modifications made to a task in a design time at runtime tool do not appear for the task.

The changes made were probably not committed to the MDS repository.

The Save button just saves the changes made in a design time at runtime tool to the sandbox. To see these changes in action, click Commit to send them to the MDS repository.

When a user tries to commit design time at run time changes to Rules, it fails with the error: oracle.mds.sandbox.SandboxNotFoundException: MDS-01805: Sandbox error

More than one user is trying to update the same rules dictionary file. This is not supported. Only one user can update the rules in a rule dictionary file at a given time.

Have one designated user to update all the rules for a given rule dictionary file. Sometimes it might map to more than one task based rules, based on the composite design. If there is only one rule dictionary for a given composite application, have a dedicated user to modify all the rules for that composite application.


B.5.11 Human Workflow API (Including SOAP/EJB) Usage Issues

Table B-12 describes symptoms, possible causes, and possible solutions for human workflow API (including SOAP/EJB) usage issues.

Table B-12 Troubleshooting Human Workflow API Usage Issues

Symptom Possible Cause Possible Solution

Location of the JavaDoc for human workflow APIs.

N/A

See Oracle Fusion Middleware Workflow Services Java API Reference for Oracle BPEL Process Manager, which is available in the documentation library.

Understanding the API usage.

N/A

  1. Refer to the API documentation.

  2. See sample workflow-118-JavaSamples, available with the Oracle SOA Suite Samples.

Using .net to access the APIs.

N/A

It is possible to write a .net client that accesses the SOAP web service APIs. See sample workflow-119-DotNetSample, available with the Oracle SOA Suite Samples.

You receive Class not found errors when attempting to use the Java API clients.

Not all required JAR files are in the client class path.

See the workflow-118-JavaSamples definition of client.classpath in the build.xml ant file, available with the Oracle SOA Suite Samples.The sample includes all required JAR files.

Creating a routing slip for simple patterns to use with a simple approval task or to dynamically route a task during task initiation.

N/A

See oracle.bpel.services.workflow.task.impl.SimpleApprovalTaskUtil.


B.5.12 Oracle JDeveloper Data Control / Form Generation Issues

Table B-13 describes symptoms, possible causes, and possible solutions for Oracle JDeveloper data control/form generation issues.

Table B-13 Troubleshooting Oracle JDeveloper Data Control / Form Generation Issues

Symptom Possible Cause Possible Solution

Empty1.jspx is not generated when creating the task form.

Created an initiator task based on an XSD element and tried to autogenerate the task form.

This file is not required except for the BPM initiator task. If you encounter it, create an Empty1.jspx file in the same directory as referenced by adfc-config.xml and put the following content in it:

<?xml version='1.0'
 encoding='UTF-8'?><jsp:root
xmlns:jsp="http://java.sun.com/JSP/
Page" version="2.1"xmlns:f="http://java.sun.com/jsf/core"
xmlns:h="http://java.sun.com/jsf/html"
xmlns:af="http://xmlns.oracle.com/a
df/faces/rich">  <jsp:directive.page
contentType="text/html;charset=UTF-8"/>
  <f:view>
    <af:document id="d1">
      <af:form id="f1"></af:form>
    </af:document>
  </f:view>
</jsp:root>

B.5.13 Human Workflow Service/ System MBean Browser Issues

Table B-14 describes symptoms, possible causes, and possible solutions for human workflow service/System MBean Browser issues.

Table B-14 Troubleshooting Human Workflow Service/ System MBean Browser Issues

Symptom Possible Cause Possible Solution

Setting commonly used human workflow configuration parameters.

N/A

Use the Workflow Task Service and Workflow Notification pages of Oracle Enterprise Manager Fusion Middleware Control:

  1. In the navigator, right-click soa-infra.

  2. Select SOA Administration > Workflow Config > Mailer tab to access notification properties.

  3. Select SOA Administration > Workflow Config > Task tab to access task service properties.

For more information, see Section 21.1, "Configuring Human Workflow Notification Properties" and Section 21.3, "Configuring Human Workflow Task Service Properties."

Setting human workflow configuration parameters not available in the Oracle Enterprise Manager Fusion Middleware Control properties pages.

N/A

Use the System MBean Browser in Oracle Enterprise Manager Fusion Middleware Control:

  1. In the navigator, right-click soa-infra.

  2. Select SOA Infrastructure >Administration > System MBean Browser.

  3. Select Application Defined MBeans > oracle.as.soainfra.config > server > WorkflowConfig > human-workflow.

  4. Set simple parameters by editing the appropriate field in the Attributes tab, and clicking Apply.

  5. Make more complex parameter changes (for example, adding a new locale) by switching to the Operations tab, selecting the appropriate operation, entering required fields, and clicking Invoke.

The System MBean Browser does not reflect my changes after editing the human workflow configuration MBeans.

The System MBean Browser is showing a previously cached version of beans.

Click the refresh cached tree data button in the System MBean Browser.

Human workflow services are not locating resource bundles or classes located at the workflow customizations class path URL.

The protocol is not specified in the URL, or the URL is missing a trailing forward slash (/).

Ensure that the configured URL is formatted correctly, and specifies a protocol. If the class path points to a directory (rather than a JAR file), it is important that the URL has a trailing forward slash character. For example:

file:///home/wstallar/wfcustomizati
ons/

Manually setting the URL used for displaying task details for a particular task component.

N/A

Use the Administration page in Oracle Enterprise Manager Fusion Middleware Control for the human task service component.

See Section 23.3, "Managing the URI of the Human Task Service Component Task Details Application" for instructions.

You can edit or delete existing task display URL entries, and add new entries. For task display URLs used from Oracle BPM Worklist, the application name must be set to worklist.


B.5.14 AMX Extension Issues

Table B-15 describes symptoms, possible causes, and possible solutions for AMX extension issues.

Table B-15 Troubleshooting AMX Extension Issues

Symptom Possible Cause Possible Solution

The dynamic approval group class is not found.

The class file is not accessible in the Oracle SOA Suite class path.

To make the dynamic approval group class accessible, the class file must be placed in the following directory:

$FMW_HOME/SOA_HOME/soa/modules/ora
cle.soa.ext_11.1.1/classes

This directory is part of the SOA class path.

The Oracle WebLogic Server must be restarted.

During design time at runtime, while defining a rule based on the Approval Group list builder, a message keeps appearing indicating that the group does not exist.

The Approval Group name is not enclosed in quotes (" ").

Enclose the name in quotes (for example, "Sample Approval Group Name").

In a ruleset, many rules defined are applicable for a transaction. It appears that the correct constraints are not getting applied; therefore, the generated approver list is not correct.

Only one rule from a ruleset must be applicable for a transaction.

In case many rules are true, the actions associated with the applicable rule with the highest priority are executed.

In case multiple applicable rules have the same priority, the first rule in the list is picked and its actions are executed.

Check that all rules in the ruleset have priorities defined so that multiple rules with the same priority are not applicable for the same transaction.


B.5.15 Oracle BPM Worklist/Task Region Issues

Table B-16 describes symptoms, possible causes, and possible solutions for Oracle BPM Worklist/task region issues.

Table B-16 Troubleshooting Oracle BPM Worklist/Task Region Issues

Symptom Possible Cause Possible Solution

You receive the following exception message in the logs:

<Warning>
<oracle.adf.controller.internal.
metadata.MetadataService>
<BEA-000000><ADFc:
/META-INF/adfc-config.xml: >
<Warning>
<oracle.adf.controller.internal.
metadata.MetadataService><ADFC-5
2024> <ADFc: Duplicate managed
bean definition for
'aleCompBindings' detected.> 

The shared library oracle.soa.worklist.webapp is referenced in the weblogic.xml file and also the JAR files adflibWorklistComponents.jar and adflibTasklistTaskflow.jar are packaged in the web application.

These JARs should not be packaged inside the web application. They should only be referenced as a shared library. Do not package these JARs in the web application.

You receive the following exception message in the logs:

Duplicate default server in
client configuration.
Configuration needs to have only
one default server in client
configuration.
Specify one default server in
client configuration. 

Two default servers are specified in the client configuration file or in the JAXB object passed to the task flow.

Mark only one server as the default in the client configuration file or in the JAXB object passed.

You receive the following exception message in the logs:

The default server is not
specified 

The default server is not specified in the client configuration file or in the JAXB object passed to the task flow.

Ensure that the default server is marked in the client configuration file or the JAXB object.

You receive the following exception message in the logs:

Invalid display column.
The display column COLUMN NAME
is not a valid Task column.
Specify a valid column name.

The column name passed to the task flow parameter displayColumnsList is not correct.

Ensure that you pass the correct column name to the task flow parameter.

You receive the following exception message in the logs:

java.lang.IllegalStateException:
Attempt to validate an already
invalid RegionSite: 

This is a generic exception that sometimes appears in server logs (for example, AdminServer.log).

See the real exception in the diagnostic logs (for example, AdminServer-diagnostic.log) and provide that exception with the bug you can file.

You receive the following exception message in the logs:

Caused by:
oracle.adf.controller.Controller
Exception: ADFC-02001: The ADF
Controller cannot find
'/WEB-INF/taskList-task-flow-def
inition.xml'

The Oracle BPM Worklist JARs are not provided in the class path, either by referring to the shared library oracle.soa.worklist.webapp or by packaging those in the web application.

Ensure either the JARs are referred through the shared library or packaged inside the application.

Filters for the task list are removed when the task list is refreshed.

Because an inbox is not a persisted view, filters set on it are removed when rendering the page again or refreshing the task list.

Instead of setting filters on the task list, create a user view with the required set of filters and pass the viewId of that view as the value of parameter ViewFilter. This makes that view the default view of the task list.

You have set the taskTypesFilterList parameter, but instead of seeing assigned tasks, you are seeing all tasks.

If you specified the taskTypesFilterList parameter, then you forgot to specify the attributesFilterList parameter.

You have to use both parameters with the AND operator. For example:

<parameter
id="taskTypesFilterList"
value="http://xmlns.oracle.com/H
elpDeskRequestSOAApp/HelpDeskReq
uestComposite/HelpDeskRequestHum
anTask,[
http://xmlns.oracle.com/Vacation
RequestApp/VacationRequest/Vacat
ionRequestTask]"/>

<parameter id=
"attributesFilterOperator"
value="and"/>
 <parameter id= "attributesFilterList"
 value="state=ASSIGNED"/>

You receive the following exception message in the logs:

[AdminServer] [NOTIFICATION]
 [J2EE JSP-00008]
 [oracle.j2ee.jsp] [tid:
 [ACTIVE].ExecuteThread: '15'
 for queue:
 'weblogic.kernel.Default
 (self-tuning)'] [userId:
 weblogic] [ecid:
17011f2a001d6b0e:7e22d6ce:12e344
4eb1b:-8000-0000000000002f0a,0]
 [APP: FederatedApp_
application1] unable to dispatch
 JSP page: The following
 exception occurred:.[[
java.lang.RuntimeException:
 Cannot find FacesContext 
at
javax.faces.webapp.UIComponentCl
assicTagBase.getFacesContext(UIC
omponentClassicTagBase.java:2122)

This is a common mistake that is generic in nature and is not related to any components you are using. You forgot to put faces in the URL. For example:

http://server:port/FederatedApp/test.jspx

Put faces in the URL as follows:

http://server:port/FederatedApp/
faces/test.jspx

You receive the following exception message in the logs:

[AdminServer] [TRACE] [] []
[tid: [ACTIVE].ExecuteThread:
'5' for queue:
'weblogic.kernel.Default
(self-tuning)'] [userId:
weblogic] [ecid:
17011f2a001d6b0e:7e22d6ce:12e344
4eb1b:-8000-0000000000001d39,0]
[SRC_CLASS:
oracle.bpel.services.workflow.cl
ient.config.ClientConfigurationU
til] [APP: FederatedApp_
application1] [SRC_METHOD:
getClientConfiguration]
WorkflowServiceClientContext:
Cannot find client
configuration file: wf_client_
config.xml 

There are three possible causes for this issue:

  1. The client configuration file is not provided in the class path.

  2. The JAXB object is not passed to the task flow.

  3. If one of the above is provided, the port number for the remote/SOAP client is incorrect.

Ensure either the client configuration file wf_client_config.xml is provided in the class path or the JAXB object is passed to the task flow. If it is already done, ensure that the port number refers to the SOA server.

You receive the following exception message in the logs:

<Error> <Deployer> <BEA-149265>
<Failure occurred in the
execution of deployment request
with ID '1297964056778' for
task '3'. Error is:
'weblogic.management.DeploymentE
xception: [J2EE:160149]Error
while processing library
references. Unresolved
application library references,
defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false].'
weblogic.management.DeploymentEx
ception: [J2EE:160149]Error
while processing library
references. Unresolved
application library references,
defined in
weblogic-application.xml:
[Extension-Name:
oracle.soa.workflow.wc,
exact-match: false]. 

Shared library oracle.soa.workflow.wc is referenced in weblogic.xml, but is not available on the server.

Ensure that this shared library is deployed on the server on which you are deploying your application. The shared library may be deployed, but not targeted, for that server.

You cannot see the mapped attributes mapped columns.

Note: Starting with Release 11g R1 (11.1.1.4), Oracle BPM Worklist flex fields are now known as mapped attributes.

The right set of parameters is not being passed to the task list task flow.

The correct set of parameters to be passed is as follows:

<parameter id= "displayColumnsList"
value="assignees,creator,
assignedDate,state,
textAttribute1,textAttribute2"/>

textAttribute1,textAttribute2 is the correct way to provide a value for the column name. Providing 'label name'(Name of the mapping) associated with these values does not work.

You must specifically pass the fully qualified value to parameter taskTypesFilterList. Otherwise, the column creation does not work.

For example:

<parameter
 id="taskTypesFilterList"
value="http://xmlns.oracle.com/Hel
pDeskRequestSOAApp/HelpDeskRequest
Composite/HelpDeskRequestHumanTask
"/>

B.5.16 Test-to-Production Issues

Table B-17 through Table B-20 describe symptoms, possible causes, and possible solutions for test-to-production issues.

Table B-17 Troubleshooting Test-to-Production Issues

Symptom Possible Cause Possible Solution

Finding the default realm name for a SOA server.

N/A

The defaultRealmName can be obtained from the identity configuration service. You can get this from the SOAP service test page.

  1. From a browser, go to the following URL:

    http:host:port/integration/servic
    es/IdentityService/configuration
    
  2. Select getDefaultRealmName from the Operation drop-down menu.

  3. Click the Invoke button.

This retrieves the default realm name. Here is a sample answer from an invocation:

<env:Envelope
xmlns:env="http://schemas.xmlsoap.org/soap
/envelope/">
 <env:Header/>
 <env:Body>
  <realmName
xmlns="http://xmlns.oracle.com/bpel/servi
ces/IdentityService">jazn.com</realmName>
 </env:Body>
</env:Envelope>

Table B-18 Troubleshooting Test-to-Production Issues

Symptom Possible Cause Possible Solution

The following user authentication error (in this example, the user is FMW_USERID and the identity context is jazn.com) is displayed:

[java] Error in workflow service Web
service operation invocation. The
error is ORA-30501:Error in
authenticating user.
[java] Error in authenticating and
creating a workflow context for
user jazn.com/FMW_USERID. [java]
Verify that the user credentials
and identity service configurations
are correct.

This occurs if the given user is not seeded and available in the LDAP provider.

To identify if the user is seeded properly, try to log in to Oracle BPM Worklist from a browser as this user. If the user can log in to Oracle BPM Worklist, that means the user is seeded.

If Oracle Internet Directory or another LDAP provider is used, ensure the configuration of the LDAP provider is completed correctly. Otherwise, you cannot get past this error.

While importing task payload mapped attribute mappings (previously known as flex fields) into the target SOA server, you may encounter the following error in the console logs:

[java] Caused by:
java.sql.SQLIntegrityConstrain
tViolationException: ORA-02291:
integrity constraint (UAT_
SOAINFRA.SYS_C0018364) violated -
parent key not found* * 

The importing of task payload mapped attribute mappings into the target SOA server is a two-step process.

Even before the import of task payload mapped attribute mappings into the target SOA server operation is attempted, there is a prerequisite step that must be performed. This is the import of attribute labels into the target SOA server operation.

To be successful, perform the following operations (in the correct order) with the human workflow test-to-production migration tool:

  1. Import attribute labels into the target SOA server.

  2. Import payload mappings into the target SOA server.

See Section 23.6, "Moving Human Workflow Data from a Test to a Production Environment" for more details.


Table B-19 Troubleshooting Test-to-Production Issues

Symptom Possible Cause Possible Solution

Assume you encounter the following error during rule migration:

[java]  Error encountered during
migration. [java] Exception in thread
"main"[java] 
UserConfigDataMigrationException:[java]
faultString:Invalid parameters for RULE.
[java] Invalid user and group: both
parameters can not have null
values.[java] To migrate User Rules,
provide the 'user' parameter only.
[java]  To migrate Group Rules, provide
the 'group' parameter only.[java]
oracle.bpel.services.workflow.util.
tools.wfUserConfigDataMigrator.
UserConfigDataMigrationE xception [java]
at oracle.bpel.services.workflow.
util.tools.wfUserConfigDataMigrator.
implhwfMigrator.parseParametersNode

There are two properties in the migration.properties file that are of interest for this error.

  • user

  • group

During any rule migration (whether export or import) operation, at most one of them (user or group) should have a value. That is, both user and group cannot have null or empty values.

Set values for at most one of them. To perform user rule migration, set the user parameter alone. To perform group rule migration, set the group parameter alone.

Assume you encounter the following error during rule migration:

[java]  Error encountered during
migration.[java] Exception in
thread "main"[java] 
UserConfigDataMigrationExcepti
on: [java]  faultString:
Invalid parameters for RULE.
[java]  Invalid user and group: both
 parameters can not have values.
[java]  To migrate User Rules,
 provide the 'user'parameter only. 
[java]  To migrate Group Rules,
 provide the 'group' parameter only.
[java]  oracle.bpel.services. workflow.util.tools.wfUserConfigData
Migrator.UserConfigDataMigrationE
xception 

This is similar to the previous explanation in this table.

During any rule migration (whether export or import) operation, both user and group parameters cannot have a value.

Provide values for at most one of them.


Table B-20 Troubleshooting Test-to-Production Issues

Symptom Possible Cause Possible Solution

Testing the health of the installed server.

N/A

Before performing a test-to-production migration, it is useful to test the health of the server.

From a browser, you can test some SOAP services. The following list provides a subset of human workflow services and Oracle BPM Worklist:

http://host:port/integration/worklistapp/

http://host:port/integration/services
/TaskQueryService/TaskQueryService

http://host:port/integration/services/IdentityService/configuration

http://host:port/integration/services/IdentityService/identity

http://host:port/integration/services/RuntimeConfigService/RuntimeConfigService

You can randomly test some operations in these services, and verify that the operation yields results. Similarly, you can log in as a user to Oracle BPM Worklist and see if everything is fine.


For more information about test-to-production issues, see Section 23.6, "Moving Human Workflow Data from a Test to a Production Environment."

B.5.17 Identity Service Issues

Table B-21 and Table B-22 describe symptoms, possible causes, and possible solutions for identity service issues.

Table B-21 Troubleshooting Identity Service Issues

Symptoms Possible Cause Possible Solution

In a clustered environment, you can receive the following error when you attempt to act on an approval task in Oracle BPM Worklist:

Token Decryption Failed

You are using a file-based repository, instead of an LDAP-based repository. This causes the application roles to not be synchronized across the nodes.

For repository configuration information in a clustered environment, you must follow the instructions in Chapter "Integrating an Enterprise Deployment with Oracle Identity Management" of Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

Only a subset of users in LDAP can log in to Oracle BPM Worklist.

The user base DN is not configured properly.

Mention the user base under which all the groups are seeded. This can be performed in two ways:

Add the base DN under which all the required groups are seeded. For instance, if users are seeded under:

UserDN 1 : cn=users1,dc=us,dc=oracle,dc=com
UserDN 2 : cn=users2,dc=us,dc=oracle,dc=com
UserDN 3 : cn=users3,dc=us,dc=oracle,dc=com

Then mention the group base DN as follows:

dc=us,dc=oracle,dc=com

This is the common DN. If only some user DNs are required (for example, UserDN1 and UserDN2), then the following property must be added to serviceInstance with the name idstore.ldap in the $DOMAIN_HOME/config/fmwconfig/jps-config.xml file:

<serviceInstance name="idstore.ldap"
 provider="idstore.ldap.provider">
 <property name="idstore.config.provider" 
value="oracle.security.jps.wls.internal.idstore.WlsLd
apIdStoreConfigProvider"/>
 <property name="CONNECTION_POOL_CLASS"
value="oracle.security.idm.providers.stdldap.JNDIPool"/>
 <extendedProperty>
   <name>user.search.bases</name>
   <values>
      <value>cn=users1,dc=us,dc=oracle,dc=com</value>
      <value>cn=users2,dc=us,dc=oracle,dc=com</value>
   </values>
 </extendedProperty>
</serviceInstance>

Users and groups seeded only in the first authenticator are visible, but not from the other authenticators.

By default, users and groups from the first authenticator are authorized.

Starting with 11.1.1.4, you can authorize users and groups from multiple authenticators. Add the following property to the idstore instance in the $DOMAIN_HOME/config/fmwconfig/jps-config.xml file.

<serviceInstance name="idstore.ldap"
  provider="idstore.ldap.provider">
   ............................
   <property name="virtualize" value="true"/>
   ..............................
</serviceInstance> 

Table B-22 Troubleshooting Identity Service Issues

Symptom Possible Cause Possible Solution

The following exception appears when myrealm (the default realm in the Oracle WebLogic Server Administration Console configuration) is passed as a parameter to the identity context to the Identity Service APIs.

Exception seen :
 Service"
Unknown macro: {0}
"in configuration" 
Unknown macro: {1}
" could not be
initialized. Error
in initializing
service
"Authentication"
in configuration
"myrealm".

The human workflow identity service uses the identity context that is set in the WorkflowIdentityConfig file (by default, it is jazn.com) and not from the Oracle WebLogic Server configuration. Therefore, in the customer code, if jazn.com is passed as the identity context, the authenticate API should work fine.

To change the realm name, the WorkflowIdentityConfig file can be edited in the System MBean Browser of Oracle Enterprise Manager Fusion Middleware Control.

  1. In the navigator, right-click soa-infra.

  2. Select SOA Infrastructure > Administration > System Mbean Browser.

  3. Select Application Defined Mbeans > oracle.as.soainfra.config > Server > WorkflowIdentityConfig > human-workflow > WorkflowIdentityConfig.ConfigurationType.

  4. Select the configuration and rename it by invoking the operation setRealmName. This change requires a server restart.

After configuring LDAP with Oracle WebLogic Server, the users are visible in the Oracle WebLogic Server Administration Console, but the following error is thrown:

No Role found
matching the
criteria

The group's base DN is not configured properly. Either the group that is being looked up is not present in LDAP or it may be seeded outside the group base DN that is mentioned while configuring LDAP.

Mention the group base under which all the groups are seeded. This can be performed in two ways.

Add the base DN under which all the required groups are seeded. For instance, if groups are seeded under:

GroupDN 1 : cn=groups1,dc=us,dc=oracle,dc=com
GroupDN 2 : cn=groups2,dc=us,dc=oracle,dc=com
GroupDN 3 : cn=groups3,dc=us,dc=oracle,dc=com

Then mention the group base DN as follows:

dc=us,dc=oracle,dc=com

This is the common DN. If only some group DNs are required (for example, GroupDN1 and GroupDN2), then the following property must be added to serviceInstance with the name idstore.ldap in the $DOMAIN_HOME/config/fmwconfig/jps-config.xml file.

<serviceInstance name="idstore.ldap"
 provider="idstore.ldap.provider">
   <property name="idstore.config.provider"
value="oracle.security.jps.wls.internal.idstore.WlsL
dapIdStoreConfigProvider"/>
 <property name="CONNECTION_POOL_CLASS"
value="oracle.security.idm.providers.stdldap.JNDIPool"/>
  extendedProperty>
  <name>group.search.bases</name>
  <values>
    <value>cn=groups1,dc=us,dc=oracle,dc=com</value>
    <value>cn=groups2,dc=us,dc=oracle,dc=com</value>
  </values>
 </extendedProperty>
</serviceInstance>

B.6 EDN Troubleshooting

This section describes how to troubleshoot EDN issues.

B.6.1 Tuning EDN Event Bus and Delivery

You can customize the following EDN properties in the System MBean Browser:

  1. Right-click soa-infra.

  2. Select SOA Infrastructure > Administration > System MBean Browser.

  3. Expand Application Defined MBeans > oracle.as.soainfra.config > Server: soa_server-x > EDNConfig > edn.

  4. Double-click an attribute.

    Properties Type Details
    NumberOfThreads int Defaults to 3 for the EDN-DB implementation or 2 for the EDN-JMS implementation. Setting this property to 0 is effectively the same as 2 for EDN-DB or 1 for EDN-JMS.

    For more information about EDN-DB and EDN-JMS, see Section 35.1, "Introduction to the Event Delivery Network."

    Paused boolean Values are true or false. If true, the EDN listener threads are decreased to 0, which effectively stops the delivery of events. This works for EDN-DB and EDN-JMS.
    NumberOfRetrys int The maximum number of EDN deliveries and redeliveries before moving the event to the error hospital. This parameter works only for one-and-only-one delivery, and not for guaranteed delivery. The maximum allowed value is 5. If set to less than 5, then it is effectively the same as 5.

    If NumberOfRetrys is set to the following values:

    • >= 2, then EDN delivers the event once, and may try redeliveries up to (NumberOfRetrys -1) times in case of XA failures.

    • = 1, then EDN delivers the event only once. There is no retry with XA failures.

    • = 0, then this is the same as = 1 (that is, EDN delivers an event only once). There is no retry with XA failures.

    • > 5, then it is effectively the same as NumberOfRetrys = 5.


  5. Enter a value, and click Apply.

B.6.2 Rolled Back One-and-Only-One Event Delivery Messages are Displayed in the Log Files

If the Java debug level is set to TRACE:16 (FINER) or a lower value, you may see log messages such as the following:

Began XA for OAOO
Rolled back XA for OAOO

These are normal messages of one-and-only-one (OAOO) event delivery when there are no events waiting to be delivered. These are not error conditions. You can turn off these messages by setting the Java logging level to TRACE:1 (FINE) or a higher value.

B.6.3 Events Are Consumed by Multiple Revisions of the Same Composites

By design, different composites can subscribe to the same event. However, it is an error if multiple revisions of the same composite subscribe to the same event. This occurs when you have multiple active versions of the composite that may be a result of a patch failure.

Perform the following steps to determine whether there are multiple, active revisions of the same composite and to retire the composite revision that should not be active:

  1. Go to Oracle Enterprise Manager Fusion Middleware Control.

  2. Click SOA > soa-infra.

    In the Deployed Composites section of the Dashboard tab, you see a list of deployed composite names, revisions, and modes (for example, active).

  3. Identify composites with the same name and with an active mode, but with different revisions.

  4. Click the composite revision that should not be active.

  5. Click the Retire button.

B.6.4 Business Event Is Picked Up Twice (Or More) By SOA Server

Business events may be raised from J2EE applications and picked up more than once. Expected behavior is that they must be picked up only once.

If a patch fails, it may be possible for two versions of a given composite to be active (that is, the older version has not been retired). When multiple versions of a composite are active, they all become subscribers and the event is picked up more than once. This should not happen under normal scenarios. Reapply the patch and ensure that the deployed composite only has one active version.

B.6.5 Some Messages Are Lost Between EDN and Composites or Composites Across Clusters

Messages primarily may get lost for the following reasons:

  1. The EDN message was not delivered.

  2. The target asynchronous service did not respond.

  3. The message was roll backed (though it was not lost; you still see it in Oracle Enterprise Manager Fusion Middleware Control).

This issue typically requires you to identify where the messages are supposed to be and to diagnose the path.

  • Check if the Oracle WSM Policy Manager security configurations allow the client to invoke the SOA composite.

  • Verify that the transaction is not transient (nonpersistent).

    Oracle BPEL Process Manager uses the dehydration store database to maintain long-running, asynchronous processes and their current state information in a database while they wait for asynchronous callbacks. Storing the process in a database preserves the process and prevents any loss of state or reliability if a system shuts down or a network problem occurs. There are two types of processes in Oracle BPEL Process Manager. These processes impact the dehydration store database in different ways.

    • Transient processes: This process type does not incur any intermediate dehydration points during process execution. If there are unhandled faults or there is system downtime during process execution, the instances of a transient process do not leave a trace in the system. Instances of transient processes cannot be saved in-flight (whether they complete normally or abnormally). Transient processes are typically short-lived, request-response style processes. The synchronous process you design in Oracle JDeveloper is an example of a transient process.

    • Durable processes: This process type incurs one or more dehydration points in the database during execution because of the following activities:

      • Receive activity

      • OnMessage branch of a pick activity

      • OnAlarm branch of a pick activity

      • Wait activity

      Instances of durable processes can be saved in-flight (whether they complete normally or abnormally). These processes are typically long-living and initiated through a one-way invocation. Because of out-of-memory and system downtime issues, durable processes cannot be memory-optimized.

  • If a composite instance is not visible in Oracle Enterprise Manager Fusion Middleware Control and the SOA Infrastructure is running, check that the Audit Level is not set to Off on the SOA Infrastructure Common Properties page. This can be checked in Oracle Enterprise Manager Fusion Middleware Control:

    1. Right-click soa-infra (SOA_cluster_name).

    2. Select SOA Administration > Common Properties.

  • If the composite instance is available, check the oneWayDeliveryPolicy BPEL property value. You can check the value in Oracle Enterprise Manager Fusion Middleware Control:

    1. In the navigation pane, expand soa-infra (SOA_cluster_name).

    2. Expand the partition, and select the composite.

      The Dashboard page for the composite is displayed.

    3. In the upper right corner, click the Show XML Definition icon.

      The contents of composite.xml for that composite are displayed.

    If this is set to async.cache, you may lose messages. Set it to async.persist for reliable messages. This is typically specified in the BPEL process service component section of the composite.xml file. Therefore, this can be set for custom composites. If the value is not set in composite.xml, the value for oneWayDeliveryPolicy in the System MBean Browser in Oracle Enterprise Manager Fusion Middleware Control is used. The following values are possible:

    • async.persist: Messages are persisted in the database hash map.

    • async.cache: Messages are stored in memory.

    • sync: Direct invocation occurs on the same thread.

    For more information about these settings, see the "Deployment Descriptor Properties" appendix and the "Transaction and Fault Propagation Semantics in BPEL Processes" chapter of the Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

    For information about setting the oneWayDeliveryPolicy property in the System MBean Browser, see Section 13.1, "Configuring BPEL Process Service Engine Properties."

    It is also possible that the transaction has invoked a target asynchronous service that has not responded back. In this case, the composite instance flow shows the call to the target asynchronous service.

  • Check if the instance has rolled back and the message is in recovery.

    1. Log in to Oracle Enterprise Manager Fusion Middleware Control.

    2. Right-click soa-infra (SOA_cluster_name), and select Service Engines > BPEL.

    3. Click the Recovery tab.

    This may occur if any external references receiving the message are not reachable (for example, an external web service, enterprise applications such as Siebel, and so on).

B.6.6 Checking Whether Bad Composites Exist in the SOA Domain that Slow Down Overall EDN Event Delivery

EDN internally uses two advanced queuing (AQ) queues in a serial processing fashion for all business events. One or more bad SOA composite applications that fail to consume certain types of events can slow down EDN delivery of all other event types and cause an event backlog in the AQ queues.

  1. Check if there are repeated errors from bad SOA composite applications in the SOA server log with EDN logging enabled. Even if the errors are in completely separate SOA composite applications that are unrelated to the event types of interest, it can delay the overall delivery of events of interest to the target SOA composite applications.

  2. Check the event backlog count as detailed in Step 4 (events are stuck in advanced queue). Obtain the counts by running the following SQL queries (substitute SOAINFRA-SCHEMA-USER with the schema owner).

    SELECT COUNT(*) FROM SOAINFRA-SCHEMA-USER.EDN_OAOO_DELIVERY_TABLE; 
    SELECT COUNT(*) FROM SOAINFRA-SCHEMA-USER.EDN_EVENT_QUEUE_TABLE;
    

    If bad composites are found in Step 1 and large numbers of backlogged events occur in Step 2, then proceed to 3.

  3. Stop bad composites identified in Step 1. This can be achieved from Oracle Enterprise Manager Fusion Middleware Control or the WLST command line utility.

  4. Temporarily increase the number of EDN threads to a larger number, (for example, from 3 to 5 or 7) to clear up the backlogged events faster.

    To change the number of EDN threads through the configuration of the EDNConfig MBean in Oracle Enterprise Manager Fusion Middleware Control:

    1. Go to the navigation pane.

    2. Right-click soa-infra.

    3. Select SOA Infrastructure > Administration > System MBean Browser.

    4. Expand Application Defined MBeans > oracle.as.soainfra.config > Server: soa_server-x > EDNConfig > edn.

    5. Change the value for the NumberOfThreads attribute, and click Apply.

      If you do not care about backlog event processing, you can purge the backed up events directly in AQ. The purge script is as follows:

      DECLARE
        purge_options dbms_aqadm.aq$_purge_options_t;
      BEGIN
        purge_options.block := FALSE;
        DBMS_AQADM.PURGE_QUEUE_TABLE(
          queue_table     => '&edn_user..edn_event_queue_table',
          purge_condition => NULL,
          purge_options   => purge_options);
        DBMS_AQADM.PURGE_QUEUE_TABLE(
          queue_table     => '&edn_user..edn_oaoo_delivery_table',
          purge_condition => NULL,
          purge_options   => purge_options);
      END;
      /
      commit;
      
  5. Periodically monitor the EDN backlog count (see Step 2) to confirm that it is being reduced. If the clear-up rate is very slow, you can further increase the EDN thread number (see Step 4).

  6. Continue to monitor the EDN backlog count over a period. Also monitor the recent instances and faults in Oracle Enterprise Manager Fusion Middleware Control to see if composite instances are being created or new faults are showing up both across the board (that is, at SOA Infrastructure level) and for the composite of interest.

  7. Once the backlog is cleared up, revert the number of EDN threads to the previous setting or an appropriate number.

  8. Publish a business event of interest and verify that the target composite is triggered with expected instance creations.

B.7 Performance Troubleshooting

This section describes how to troubleshoot performance issues.

B.7.1 Optimizing the Loading of Pages with Instance and Fault Metrics

Since production systems can include numerous composite instances and faults, there is a possibility of timeouts in the Oracle Enterprise Manager Fusion Middleware Control pages as information retrieval becomes relatively slow. To optimize the loading performance of Dashboard pages for the SOA Infrastructure, SOA composite applications, service components, and service engines; Deployed Composites page of the SOA Infrastructure and service engines; and partition home page, you can enable two property settings on the SOA Infrastructure Common Properties page that perform the following:

  • Disable the loading of all metrics information upon page load. The instances and faults metrics can be obtained on demand from the server.

    Note:

    This setting disables fault, instance, and count metrics, and not DMS metrics.
  • Restrict the retrieval of instances, faults, and counts to a specified time period (default selection).

To optimize the loading of pages with instance and fault metrics:

  1. In the navigator, click soa-infra.

  2. Note that values appear in the Running and Total fields in the Recent Composite Instances section and the Instances and Faulted Instances columns of the Deployed Composites section. When these values are large, it can take time to load this page and other pages with similar information.

    Description of soaadmin_fetch1.gif follows
    Description of the illustration soaadmin_fetch1.gif

  3. From the SOA Infrastructure menu, select SOA Administration > Common Properties.

  4. In the Data Display Options section, select the Disable fetching of instance and fault count metrics checkbox.

  5. Click Apply.

  6. Return to the Dashboard page of the SOA Infrastructure.

  7. Note that the values that previously displayed have been replaced with links.

  8. In the Deployed Composites section, click a link for a specific SOA composite application to display details about the number of instances. The values are calculated for the link you select.

    Description of soaadmin_fetch7.gif follows
    Description of the illustration soaadmin_fetch7.gif

  9. Click OK.

  10. Click Number of Instances.

    Description of soaadmin_fetch6.gif follows
    Description of the illustration soaadmin_fetch6.gif

    A message displays the number of running and total instances in the SOA Infrastructure.

  11. Click OK.

  12. Return to the SOA Infrastructure Common Properties page.

  13. In the Data Display Options section, select the Restrict display of instances and faults to the last time_period checkbox if it is not already selected. By default, this checkbox is selected and the time period duration is set to 24 hours (one day).

  14. Specify a time period during which to retrieve recent instances, faults, and count metrics for display (for example, 10 minutes).

  15. Click Apply.

  16. Return to the Dashboard page of the SOA Infrastructure.

  17. Note that the following message is displayed in the upper right section of the page:

    Recent Instances and Faults for the last 10 minutes
    
  18. Click Number of Instances.

    The value that you specified for the time period is displayed in the message title. The number of running and total messages for that time period is also displayed.

    Description of soaadmin_fetch8.gif follows
    Description of the illustration soaadmin_fetch8.gif

  19. Click OK.

  20. In the Deployed Composites section, click a link.

    Description of soaadmin_fetch2.gif follows
    Description of the illustration soaadmin_fetch2.gif

    The value that you specified for the time period is displayed in the message title. The number of running and total messages for the selected SOA composite application for that time period is also displayed.

    Description of soaadmin_fetch3.gif follows
    Description of the illustration soaadmin_fetch3.gif

For more information about setting these properties, see Section 3.1, "Configuring SOA Infrastructure Properties."

Notes:

  • If you click a link to retrieve instance and fault count metrics, and Oracle Enterprise Manager Fusion Middleware Control times out, increase the transaction timeout property. For more information, see Section B.3.1, "Resolving Connection Timeouts."

  • If you click Recalculate, and the recalculation occurs quickly, the progress indicator does not have a chance to render. However, any updates to the data are reflected on-screen.

B.7.2 Resolving Message Failure Caused by Too Many Open Files

You can receive the following error at runtime or compilation time, depending on the number of JAR files being used, the use of file descriptors by JDK 6/JRE, or both.

Message send failed: Too many open files 

To resolve this error, increase the number of file descriptors to at least 4096.

  1. Use the limit command (for the C shell) or the ulimit command (for the Bash shell) to identify the value for descriptors. A value of 1024 is typically too low, especially for JDK 6.

    % limit
    
    cputime      unlimited
    filesize     unlimited
    datasize     unlimited
    stacksize    10240 kbytes
    coredumpsize unlimited
    memoryuse    unlimited
    vmemoryuse   unlimited
    descriptors  1024
    memorylocked 500000 kbytes
    maxproc      46720
    
  2. Log in as the root user on your operating system.

  3. Edit the /etc/security/limits.conf file to increase the value for descriptors.

    For this example, the limits.conf file appears as follows after increasing the limit for all users to 4096:

    #<domain>      <type>  <item>         <value>
    #
    
    #*               soft    core            0
    #*               hard    rss             10000
    #@student        hard    nproc           20
    #@faculty        soft    nproc           20
    #@faculty        hard    nproc           50
    #ftp             hard    nproc           0
    #@student        -       maxlogins       4
    
    # End of file
    @svrgroup    soft    memlock         500000
    @svrgroup    hard    memlock         500000
    *           soft    nofile          4096
    *           hard    nofile          4096
    
  4. Close your terminal and reopen for the change to take effect. A system restart is not required.

B.7.3 Resolving MaxMessageSizeExceededException Errors Caused By Large Payloads

If you provide a large payload (for example, 12 MB) to your deployed SOA composite application, then click View XML Document in the audit trail to view the payload, you can encounter MaxMessageSizeExceededException errors. This error can be resolved by setting the following JVM parameter.

  1. Open the following file:

    • On UNIX operating systems, open $MIDDLEWARE_HOME/user_projects/domains/domain_name/bin/setDomainEnv.sh.

    • On Window operating systems, open MIDDLEWARE_HOME\user_projects\domains\domain_name\bin\setDomainEnv.bat.

  2. Add the weblogic.MaxMessageSize property with the following value:

    EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
    -Dweblogic.MaxMessageSize=20000000"
    export EXTRA_JAVA_PROPERTIES 
    
  3. Restart the server.

B.7.4 Extending Tablespaces to Avoid Problems at Runtime

If the database tablespace is not extended, runtime processing can be impacted. Messages are not processed or persisted, and exception errors similar to that shown in Example B-3 can appear in the log files. This is because Oracle BPEL Process Manager relies on the database to store instance data. If the database is not available, runtime processing is impacted.

Example B-3 Exception Error

INFO: MediatorServiceEngine returning after processing the request for 
operation = processResponse 

[EL Warning]: 2009.01.14 11:46:16.783--UnitOfWork(32372128)--Exception 
[EclipseLink-4002] (Eclipse Persistence Services - 1.1 (Build 
SNAPSHOT-20081007)): org.eclipse.persistence.exceptions.DatabaseException 
Internal Exception: java.sql.BatchUpdateException: ORA-01691: unable to 
extend lob segment SH_SOAINFRA.SYS_LOB0000145067C00007$$ by 1024 in 
tablespace SH_SOAINFRA 

Error Code: 1691 
Query: InsertObjectQuery(com.collaxa.cube.persistence.dto.AuditTrail@199b33d) 
[EL Warning]: 2009.01.14 11:46:16.782--UnitOfWork(32372128)--Exception 
[EclipseLink-4002] (Eclipse Persistence Services - 1.1 (Build 
SNAPSHOT-20081007)): org.eclipse.persistence.exceptions.DatabaseException 
Internal Exception: java.sql.BatchUpdateException: ORA-01691: unable to 
extend lob segment SH_SOAINFRA.SYS_LOB0000145067C00007$$ by 1024 in 
tablespace SH_SOAINFRA 
. . .
. . .

Ensure that you set a tablespace to automatically extend itself by a specified amount when it reaches its size limit. If you do not enable autoextend, ensure that you respond when alerted that the tablespace is reaching its critical or warning threshold size. You can respond to size alerts by manually increasing the tablespace size.

B.7.5 Resolving Database Growth Issues Caused by a High Volume of Transactions

If Oracle SOA Suite transactions fail and logs indicate that the database is running out of space, the dehydration store or MDS store may be running out of space due to a high volume of transactions. In the latter example, you see errors such as the following:

java.sql.SQLException: ORA-01653: unable to extend table SH_MDS.CUBE_INSTANCE by
16 in tablespace FUSION_TS_TOOLS 

This indicates that the tablespace is full and the database cannot extend it.

Perform the following tasks.

  1. Purge the dehydration store tables periodically, taking into account the appropriate record retention policies and ensuring that the applications have no dependencies on runtime data.

    The purge should be followed by commands to coalesce the space. For the purging strategy to work, it is important to understand how long to retain the data in the database. Factors that drive the retention policy include the following:

    • Legal requirements

    • Line of business requirements

    • Overall company policy on retention of data

    The longer the retention policy, the greater the volume of data that must be stored and, correspondingly, the higher the disk capacity requirements.

    For details on creating a purging strategy, see Chapter 9, "Developing a Database Growth Management Strategy" and Chapter 10, "Managing Database Growth."

  2. Ensure that the database hardware has sufficient resources to handle the demands of Oracle database partitioning before configuring your tables for partitioning.

    For the dehydration store, database partitioning using range partitioning and hash partitioning is an optimal solution. Partitioning by definition means storing data in multiple tables to reduce bigger data sets into smaller, more manageable data sets. Partitioning strategies play a large role in easing maintenance overheads (dropping and pruning the partition) and improving performance. Partitioning should at least be done for tables having high activity. This plays a large role in balancing disk I/O and preventing hot disks. One important requirement that you must meet before configuring your tables for partitioning is to ensure that the database hardware has sufficient resources to handle the demands of Oracle database partitioning. If preproduction testing has indicated that the installation is large, Oracle expects that you have sized your environment (CPU, memory, and disk space) correctly to take advantage of the partitioning features.

  3. Tune database parameters for memory, tablespace, and partitions to get maximum performance. For more information, see Section "Tuning Database Parameters" of the Oracle Fusion Middleware Performance and Tuning Guide.

  4. For other tablespaces running out of space, use the following query to check for free tablespace:

    SELECT TOTAL.TABLESPACE_NAME "TABLESPACE NAME", FREE_SPACE, (TOTAL_SPACE-FREE_
    SPACE) USED_SPACE, TOTAL_SPACE, ROUND((FREE_SPACE*100/TOTAL_SPACE),2) "FREE %"
    FROM (SELECT TABLESPACE_NAME, SUM(BYTES/1024/1024) FREE_SPACE FROM SYS.DBA_
    FREE_SPACE GROUP BY TABLESPACE_NAME) FREE,
    (SELECT TABLESPACE_NAME, SUM(BYTES/1024/1024) TOTAL_SPACE FROM SYS.DBA_DATA_
    FILES GROUP BY TABLESPACE_NAME) TOTAL
    WHERE FREE.TABLESPACE_NAME = TOTAL.TABLESPACE_NAME AND TOTAL.TABLESPACE_NAME =
     '<TABLESPACENAME>'
    ORDER BY 5;
    
  5. To increase tablespace settings, use the administrator account. For example:

    ALTER TABLESPACE TABLESPACE_NAME ADD DATAFILE 'DATAFILE_NAME' SIZE 500M
    AUTOEXTEND ON;
    

For more details, see Section B.7.2, "Resolving Message Failure Caused by Too Many Open Files."

B.7.6 Observing Slow Application Performance Such as Longer Time to Serve Pages or Finish Transactions

You can observe slow application performance and/or memory trashing. For example, it may take longer to load and serve pages or to complete composite transactions. Response time may seem slower compared to normal behavior.

There are various reasons for slow performance. It may be due to a large number of servers running on the same host, or there may be a large number of records/sessions/locking in the database. Thread contention can also be a reason for poor performance.

Note:

All configuration changes in Oracle Enterprise Manager Fusion Middleware Control may be unavailable due to this problem.

To diagnosis this issue:

  1. Check the CPU utilization to see if it is saturated due to a heavy load or too many processes in relation to CPU capacity.

    If CPU utilization is at 100% during normal load hours (the target should be 70-80%), you have no capacity to handle a peak load and the hardware resources are insufficient. Add scale-out servers to handle the additional load.

  2. Check applications using Oracle Enterprise Manager Fusion Middleware Control to report on performance. Check the performance of services and the invoke/response times in the BPEL process audit trail.

  3. To maximize performance, it is recommended that you not set the logging level higher than the default INFO level.

    For debugging purposes, you must set the logging level to the FINEST level. However, once issues are resolved, reset the logging level to the default level for best performance. It is also recommended that you set Audit Level to Production on the SOA Infrastructure Common Properties page. This can be set in Oracle Enterprise Manager Fusion Middleware Control as follows:

    1. Right-click soa-infra (SOA_cluster_name).

    2. Select SOA Administration > Common Properties.

    3. Set Audit Level to Production.

  4. Purge periodically based on retention requirements to maintain any service level agreements (SLAs).

    To identify the tables where data growth may lead to performance issues, see Section "Identifying Tables Impacted By Instance Data Growth" of the Oracle Fusion Middleware Performance and Tuning Guide.

    For information about using the purge scripts, see Chapter 10, "Managing Database Growth."

  5. Check the memory/IO/paging/swapping/CPU usage load statistics using Top or Glance or another monitoring tool.

  6. Optimize the JVM to avoid full garbage collection or out-of-memory errors.

    Frequent garbage collection can be either due to higher memory usage or memory leaks.

    1. Ensure that the sum of the maximum heap size of all the JVMs running on your system does not exceed the amount of available physical RAM to avoid operating system level paging activity.

    2. Use the JRockit mission control memory profiling tools to get thread dumps and memory snapshots, which helps Oracle Support Services debug any code issues. For more information about thread dumps, see Chapter 12, "Diagnosing Problems with SOA Composite Applications."

  7. Optimize threads to avoid contention. Get a thread dump to investigate and submit it to Oracle Support Services.

  8. Check the JVM and thread dumps for methods invoking the database to identify if database performance is a bottleneck.

  9. Run database AWR snapshots to identify causes of database performance issues.

  10. Ensure that database statistics are updated at regular intervals and other tunable parameters for memory, tablespace, and partitions are used effectively to obtain maximum performance.

    Here are some common tuning recommendations. For more information, see the "Tuning Database Parameters" section in the Oracle Fusion Middleware Performance and Tuning Guide.

    • Put indexes and tables in as physically separate disk areas, if possible.

    • Never put rollback segments with data or index segments.

    • Separate highly active tables and indexes into their own tablespaces.

    • Partition high activity tables and indexes to help balance disk I/O and prevent hot disks.

    • Have processes in place to generate database table statistics at regular intervals.

  11. Tune database tables to control the high watermark (HWM) contention of large objects. Tune database advanced queues (AQ) to control HWM contention and ensure consistent performance of producing and consuming messages from AQ.

    The EDN AQ names to be aware of are EDN_EVENT_QUEUE_TABLE and EDN_OAOO_DELIVERY_TABLE.

  12. Tune the BPEL process and EDN thread counts to ensure optimal settings (for example, the Dispatcher Invoke Threads and Dispatcher Engine Threads properties on the BPEL Service Engine Properties page in Oracle Enterprise Manager Fusion Middleware Control). If the thread configuration is too high, the servers run out of memory. If they are too low, the messages start backing up.

  13. Tune the BPEL process properties to reduce overhead (for example, disable the ValidateXML and StatsLastN (statistics gathering batch size) properties on the BPEL Service Engine Properties page in Oracle Enterprise Manager Fusion Middleware Control), if they are not needed.

  14. In case of integration with packaged applications (for example, Siebel), check if the issue lies with the legacy applications.

For more information about performance tuning the various components, see the "Top Performance Areas" chapter and the "SOA Suite Components" part in the Oracle Fusion Middleware Performance and Tuning Guide.

For information about setting properties on the BPEL Service Engine Properties page, see Chapter 13, "Configuring BPEL Process Service Components and Engines."

B.7.7 Observing Incoming Message Rates Exceeding Outgoing Message Rates

When the incoming rate of messages exceeds the processing rate of messages (that is, the outgoing rate), you can receive the warning message shown in Example B-4.

Example B-4 Warning Message When Incoming Message Rates Exceed the Processing Message Rate

[2012-04-09T16:49:40.342-04:00] [Soa_server1] [WARNING] []
 [oracle.soa.bpel.engine.dispatch] [tid:
 weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@e3d0e3d
] [userId: <anonymous>] [ecid: 0000JSnNKVTBh4Z5ln1FiZ1Fdnmd001gWL,0] [APP:
soa-infra] BPEL service engine is overloaded, the inflow is higher than the
outflow, there are "100000" invoke messages pending in the dispatcher in-memory
cache, the configured threshold is "102",the new incoming invoke messages will not
be dispatched or processed immediately, they are going to be persisted in the
dehydration store and can be recovered later using recovery.

To resolve this issue, either increase the processing rate or lower (that is, throttle down) the incoming rate. The relatively low processing rate may be due to any or all of the following issues:

  • Not enough invoke threads

  • Not enough JVM capacity

  • The external partners that the BPEL process is calling cannot scale, which causes the messages to back up in the BPEL process.

To identify the exact causes and resolve them, perform the following steps:

  1. Check the oneWayDeliveryPolicy property value in the composite.xml file. Is it set to async.persist, async.cache, or sync? For information about this property, see Appendix "Deployment Descriptor Properties" of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

  2. Ensure that the outgoing invocations from the BPEL process to external services are not backed up due to low scalability of the external services. If the bottleneck is in an external service, any tuning you perform in Oracle SOA Suite (that is, Steps 3 and 4) does not help. You must first resolve the bottleneck.

  3. Monitor your JVM garbage collection (GC). If the GC frequency appears higher than normal and the JVM appears to be near capacity, increase the JVM heap size. If the JVM does not have enough capacity, any tuning you perform in Step 4 makes matters worse.

  4. Increase the values for the dispatcher invoke threads and dispatcher engine threads on the BPEL Service Engine Properties page so that the server has more threads to process the incoming messages. For more information, see Section 13.1, "Configuring BPEL Process Service Engine Properties."

B.8 Server Troubleshooting

This section describes how to troubleshoot server issues.

For more information about server startup issues, see Section 3.2, "Stopping and Starting the Managed Server and SOA Infrastructure."

B.8.1 Best Practices for Starting and Stopping a Managed Server

As a best practice, it is always recommended that you start and stop a managed server through one, but not both, of the following methods. Do not mix these methods, such as starting the managed server from the command line and stopping it from Oracle Enterprise Manager Fusion Middleware Control, or vice versa.

  • Oracle Enterprise Manager Fusion Middleware Control

    With this method, the node manager must be up and running. The node manager tracks all managed server startups and shutdowns performed from Oracle Enterprise Manager Fusion Middleware Control. With this method, the server state is not an issue.

  • Command line

    With this method, the node manager does not track the server state. Therefore, if you start the server from the command line and shut it down from Oracle Enterprise Manager Fusion Middleware Control, the Oracle WebLogic Administration Server accesses the node manager to determine its status, which returns a state of unknown.

Perform the following steps to stop and start the server from Oracle Enterprise Manager Fusion Middleware Control.

  1. Expand the WebLogic domain.

  2. Select the managed server (for example, named soa_server1).

  3. Select Control > Shut Down.

  4. Select Control > Start Up.

For information on starting and stopping managed servers from the command line, see Oracle Fusion Middleware Installation Guide for Oracle SOA Suite and Oracle Business Process Management Suite.

B.8.2 Diagnosing SOA Server Startup Problems

If the SOA server (soa_infra) does not start, there can be various reasons for server startup issues. A discovery-based approach to finding the root cause is required. Check the server and diagnostic logs as a first attempt to diagnose the issue. In addition, check the following.

To diagnose server startup issues:

  1. Check if the database is not available or there are not enough connections available. Some failures with Oracle SOA Suite runtime can result from database outage/connectivity issues. Perform the following steps:

    1. Log in to Oracle WebLogic Server Administration Console.

    2. In the Domain Structure, view the status by selecting Services > Data Sources > SOADatasource > Monitoring > Testing to test the data source. You can also check this from a SQL prompt.

  2. Check the list of ports used for port conflicts.

  3. Check if the MDS repository cannot load shared documents (incorrect MDS configuration or the database that holds the MDS schema is not reachable).

  4. Check the Oracle Coherence configuration if using an Oracle SOA Suite cluster (use of unicast versus multicast).

    This issue only applies after provisioning is complete. Clustering of SOA servers may fail if there are port conflicts (used by Oracle Coherence). For example, one SOA server may be picking up the deployment, but the other server is not. In some cases, this may only present the following error:

    [soa_server1] [ERROR] [] [Coherence] [tid: Logger@352821903 3.6.0.4] 
    [ecid: 46f620208907e045:63f295ec:12dd091ec2e:-8000-0000000000000003,1:27187] 
    [APP: soa-infra] 2011-01-28 23:06:19.463/414.816 Oracle Coherence GE 3.6.0.4 
    <Error> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
     (self-tuning)', member=n/a): 
    Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout
     during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster[[...
    

    In this example, Oracle Coherence timeouts are prominently available. However, sometimes it presents itself with an unrelated error, such as the following:

    Error creating bean with name 'SensorManager' defined in ServletContext
    resource
    

    The root cause of this is still primarily related to Oracle Coherence configuration.

  5. Check if both the administration server and managed server ports are open and accessible.

  6. Check if managed server startup failed because the administration server is not reachable.

  7. Check network issues (for example, IP routing filtering/rules that may be causing issues).

  8. Check Oracle WebLogic Server LDAP security corruption.

    The managed server may report that policies for the application System MBeans Browser already exist or do not exist.

For more information, see the "Setting the Frontend URL for the Administration Console and Setting Redirection Preferences" section and the "Setting the Frontend HTTP Host and Port" section in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

B.8.3 Specifying the Proxy Server

To use system properties to specify the proxy server, write your client application in the standard way, and then specify Java system properties when you execute the client application. Example B-5 provides details.

Example B-5 Proxy Server Specification

setenv PROXY_SETTINGS "-DproxySet=true
-Dhttp.proxyHost=www-myproxy.us.mycompany.com -Dhttp.proxyPort=80
 -Dhttp.nonProxyHosts=localhost|*.us.mycompany.com
  |0:0:0:0:0:0:0:1|fe80:0:0:0:250:56ff:fe31"

Note:

When you specify values for proxy properties such as http.proxyHost and http.proxyPort, also specify the http.nonProxyHosts property.

B.8.4 Flow Diagram Does Not Display The First Time on Some Lower End Hosts

The flow diagram for an instance ID of a deployed SOA composite application in Oracle Enterprise Manager Fusion Middleware Control may not display the first time on some lower end hosts. Instead, you receive a failed to load resource message.

As a workaround, close the flow trace page and click the instance ID to return to the flow trace page.

B.8.5 Accessing Oracle Enterprise Manager Fusion Middleware Control on Dual Stack Hosts that Support IPv4 and IPv6

If you run Oracle SOA Suite on a dual stack host that supports both IPv4 and IPv6, you must update the etc/hosts file as shown in Table B-23 for IPv4 clients to access IPv6 URLs in Oracle Enterprise Manager Fusion Middleware Control.

Table B-23 IPv4 and IPv6 Settings in etc/hosts File

On The... Edit the etc/hosts File as Follows....

On the IPv4 client:

xx.xxx.xxx.xxx myhost10-ipv6 

where xx.xxx.xxx.xxx is the IP address of IPv6.

On the IPv6 client

2001:0db8:db0a::0:1  myhost10-ipv6 myhost10-ipv6.us.example.com

Note: Replace 2001:0db8:db0a::0:1 with a value appropriate to your host environment.


B.9 Browser Troubleshooting

This section describes how to troubleshoot browser issues.

B.9.1 Limitation on Using the Safari Browser to View WSDL File Content

If you are using the Safari browser, note the following limitation and workaround for viewing WSDL file contents in Oracle Enterprise Manager Fusion Middleware Control. Note also that Mozilla Firefox works correctly and does not require this workaround.

  1. Go to the home page for a SOA composite application.

  2. Click the Show WSDL and endpoint URI link at the top of the page.

  3. Click the WSDL link that is displayed.

    This opens a blank page that does not display the contents of the selected WSDL.

    As a workaround, perform the following additional steps.

  4. In the upper right corner of this page, click the Display a menu for the current page icon.

  5. Select View Source from the menu that is displayed.

    This displays the contents of the selected WSDL in another page.

B.10 Additional Troubleshooting Documentation

Table B-24 describes documentation to see for additional troubleshooting issues.

Table B-24 Additional Troubleshooting Documentation

For Information About... See...

SOA Infrastructure startup and shutdown issues

Section 3.2, "Stopping and Starting the Managed Server and SOA Infrastructure."

Using WebLogic Diagnostic Framework (WLDF) and Diagnostics Framework (DFW) to diagnosis Oracle SOA Suite problems

Chapter 12, "Diagnosing Problems with SOA Composite Applications."

Managing large documents and metadata and for managing environments with large numbers of instances

Chapter "Managing Large Documents and Large Numbers of Instances" of Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

Deployment and compilation errors

Section "Testing and Troubleshooting" in Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite.

Oracle WSM Policy Manager errors

Section "Diagnosing Problems" chapter in the Oracle Fusion Middleware Security and Administrator's Guide for Web Services.

Oracle Business Activity Monitoring (BAM)

Appendix "Troubleshooting" of Oracle Fusion Middleware User's Guide for Oracle Business Activity Monitoring.

Oracle Business Rules

Appendix "Oracle Business Rules Troubleshooting" of Oracle Fusion Middleware User's Guide for Oracle Business Rules.

Oracle User Messaging Service

Section 27.7, "Troubleshooting Oracle User Messaging Service."