B Troubleshooting Oracle SOA Suite and Oracle BPM Suite
This appendix includes the following sections:
Setting Logging Levels for Troubleshooting
To simplify troubleshooting, it is recommended that you set logging levels to the TRACE:32 FINEST level in Oracle Enterprise Manager Fusion Middleware Control. This section describes loggers to which to set to this level.
To set logging levels for troubleshooting:
Log Files and Thread Dumps from All Managed Servers
Table B-1 describes the log files to view and thread dumps to obtain.
Table B-1 Log Files and Thread Dumps
Output | Description |
---|---|
Server diagnostic log |
View the following file: $DOMAIN_HOME/servers/server_name/logs/server_name-diagnostic.log For example, This is where the log output is available. By default, only the last 100 MB of the diagnostic logs are retained. |
Server log |
|
Server console output |
|
Server thread dump |
Enter the following at the operating system command prompt: kill -3 managed_server_process_ID You can also use Oracle WebLogic Server Administration Console.
The output is in the console logs. |
OWSM message log |
The following log captures all SOAP messages on the wire. $DOMAIN_HOME/servers/server_name/logs/owsm/msglogging/diagnostic.log This log is not enabled by default. To enable this log:
|
Parallel Purging and Table Partitioning Issues
This section describes how to troubleshoot parallel purge and table partitioning issues. Audit level recommendations are also provided.
You can delete instances and rejected messages with the PL/SQL purge script or from the Auto Purge page in Oracle Enterprise Manager Fusion Middleware Control. For more information, see Managing Database Growth .
Executing the Parallel Purge Script
This section addresses situations in which the Oracle SOA Suite database has not been managed and space must be reclaimed. This section also describes how to identify the appropriate parameter settings for normal, daily execution.
When trying to reclaim space, the purge scripts must be executed frequently to delete substantially more SOA composite applications than are being created. This has implications for host and database resources that must be monitored to avoid lengthy wait queues.
There must be a clear goal on the number of SOA composite applications to delete. Otherwise, the purge script can be configured too aggressively. The following aspects are described:
Specifying the Degree of Parallel Value
When specifying the appropriate degree of parallel (DOP) value, hardware resources must be considered. The DOP identifies the number of jobs scheduled by the database and the number of processes concurrently executed on the host.
The parallel purge jobs are I/O intensive, which means that they are not entirely CPU-bound. However, the following formula is a starting point after which testing and monitoring are required to identify any increases:
DOP <= PARALLEL_THREADS_PER_CPU x CPU_COUNT
While purge jobs are run concurrently, they are actually executed with a 30-second incremental delay to avoid resource contention. The delay is not meant to avoid row contention because the purge jobs must not contend for the same SOA composite applications. This delay has implications for the elapsed time of the purge jobs. As an example, a DOP of 10 means that the tenth purge job sleeps for 270 seconds before starting (that is, (DOP - 1) * 30 sec)
.
If a high DOP number is specified, then the max_runtime
parameter must be set appropriately.
Parsing SOA Composite Applications to Delete
The temptation to delete as many SOA composite applications as possible in each execution of the purge script is understandable when trying to reclaim space. However, the script must parse the data for the appropriate SOA composite application, which can monopolize the elapsed time of the purge.
The strategy must be to reduce the data by running the purge script often with modest date and max_count
parameter settings. This reduction requires the parallel purge to be scheduled frequently, perhaps twice an hour.
The set of SOA composite applications to parse is identified by the following parameters:
-
min_creation_date
-
max_creation_date
-
retention_date
-
max_count
A modest max_count
parameter setting is the most effective method for reducing parse elapsed times. Set the max_count
parameter low, perhaps to 50K, to begin until an optimal value can be identified. As the tables start to reduce in size, the max_count
parameter and date ranges can be increased.
Note:
This method of reduction is not always effective. On occasions, you may need to introduce parallel query to parse larger amounts of data.
Using Parallel Query Slaves
The parsing for candidate SOA composite applications to delete is primarily performed by several INSERT
...
SELECT
statements. The larger the set of SOA composite applications to parse, the more the database cost-based optimizer favors a full table scan. When large portions of a table must be traversed, a full table scan can be faster than index access because it performs multiblock I/O calls.
A parallel query can significantly improve full table scan performance by coordinating and distributing the work over multiple slave processes. Additionally, the PARALLEL
hint on the INSERT
portion executes in APPEND
mode. This further improves performance through direct path inserts.
As with the scheduled purge jobs, the parallel query slaves are I/O intensive and not entirely CPU bound. However, the following formula is a good starting point when selecting a DOP for the PARALLEL
hint:
DOP <= PARALLEL_THREADS_PER_CPU x CPU_COUNT
Note:
There is a DOP setting to identify the number of parallel purge jobs and a DOP setting to identify the number of parallel query slaves. These DOP settings both impact the amount of processes executed on the host, but not simultaneously. The INSERT
...
SELECT
statement executes first to parse the data followed by the scheduling of the parallel purge jobs.
For the INSERT... SELECT
statement, the PARALLEL
hint can be specified after the INSERT
keyword, after the SELECT
keyword, or both. Therefore, parallelism of the INSERT
and SELECT
operations are independent of each other. The appropriate DOP for the entire statement is identified by the following precedence rule:
-
PARALLEL
hint onINSERT
portion of the statement. -
Set at session.
-
DOP on
INSERT
table. -
Maximum DOP set on any table in the statement.
To execute parallel DML, the following statement must be executed within the script:
ALTER SESSION FORCE PARALLEL DML STATEMENT
For example, this statement has a DOP of 2
for the INSERT
and 4
for the SELECT
. However, given the precedence rules, the DOP for the entire statement is 2
:
INSERT /*+ PARALLEL(TBL_INS,2) */ INTO TBL_INS SELECT /*+ PARALLEL(TBL_SEL,4) */ * FROM TBL_SEL;
Note:
A parallel query can also be applied to the DELETE
and SUBSELECT
commands in the purge script, if required.
Debugging and Tracing Purging Operations
This section describes debugging and tracing issues.
Debugging Purging Operations
The purge scripts are instrumented with a DEBUG
flag that provides detailed, timestamped information on each operation. This information can be evaluated to identify which tables are difficult to delete and may be better managed outside of the purge scripts through table partitioning.To set the DEBUG
flag:
Oracle SOA Suite Table Partitioning
Oracle table partitioning addresses many concerns about the maintenance of large tables by decomposing them into smaller, more manageable segments called partitions. The SOA schema has been instrumented with partition keys to take advantage of the Oracle database range partitioning.
For more information about partitioning, see Partitioning Component Tables.
Referential Integrity and Equipartitioning
For performance reasons, the Oracle SOA Suite service components (excluding Oracle B2B) have no foreign key constraints to police referential integrity. Without these constraints, the relationship between master and detail tables must be protected to avoid dangling references in the detail tables.
Equipartitioning is a database partitioning feature that partitions master and detailed tables based on their foreign key constraint. This feature creates table partitions that group related master and detail rows in the same date range. This grouping ensures that no dangling references are created when partitions are dropped.
To mimic this feature, Oracle SOA Suite is instrumented to push the creation date of the business flow instances down into the component master tables (Oracle BPEL Process Manager, Oracle Mediator, and human workflow). Because the SOA Infrastructure is the top level component on which all other components depend, this method groups all dependent rows by date range.
For example, the timestamp in the CREATED_TIME
column of the SCA_FLOW_INSTANCE
table is pushed down into the following tables:
-
Oracle BPEL Process Manager master table
CUBE_INSTANCE
columnCPST_INST_CREATED_TIME
, which is then pushed down into dependent tables such asCUBE_SCOPE.CI_PARTITON_DATE
. -
Oracle Mediator master table
MEDIATOR_INSTANCE
columnCOMPOSITE_CREATION_DATE
, which is then pushed down into dependent tables.
Note:
-
Dependent tables must be partitioned with their master and, in all cases, the top level SOA Infrastructure
SCA_FLOW_INSTANCE
table must be partitioned. -
To complete equipartitioning, the table partitions must all share the same name and date range.
In summary, equipartitioning means that the associated dependent table rows are in a partition with the same partition key range as their master table rows. Therefore, the state of each detail row in the equipartition can be inferred from its associated master table row.
SOA Composite Application Range-Hash Partitions
Range partitions can have their rows hashed into a subpartition to implement SOA composite application range-hash partitioning. Hash subpartitions can benefit in distributing I/O, but it is not currently recommended when partitioning Oracle SOA Suite tables.
Hashing of keys is a method that works well with equality predicates (=
, IN
), and there are several tables with keys that are good candidates such as the Oracle BPEL Process Manager CIKEY
table. However, range-hash partitions alone do not convey the uniqueness of the hash key. For the cost-based optimizer to identify uniqueness, the query must search all partitions of a table (a full table scan). Partition pruning helps by narrowing the range of partitions to search, but Oracle SOA Suite does not use this feature. (See Partition Pruning.) Therefore, to avoid a full table scan when the cost-based optimizer identifies uniqueness, indexes are searched that bypass the hash subpartition.
Interval Partitioning
Interval partitioning is an extension of range partitioning that instructs the database to automatically create partitions as required.
The verification scripts currently require the partition name to be the same across all tables in a group. The interval partitioning system generates partition names that differ for each partition. Therefore, this is not supported.
Global Hash Indexes
The use of global hash indexes is independent of Oracle SOA Suite table partitioning.
Oracle SOA Suite is constantly inserting rows, and many of the index keys are monotonically increasing. Indexes of a BTREE structure insert these keys and target only a few database blocks that can become very hot across an Oracle Real Application Clusters (Oracle RAC) setup. This is typically seen in the AWR reports as excessive buffer busy waits. To distribute the index keys randomly across many database blocks, use global hash indexes.
Partition Pruning
When the range partition keys are used as predicates, the optimizer can prune the number of partitions to search. Oracle SOA Suite and especially Oracle Enterprise Manager Fusion Middleware Control do not presently use partition keys in SQL queries and therefore do not take advantage of this performance feature.
With the requirements of equipartitioning, the partition keys are populated with the SOA composite application creation date that is not the creation date of the components. The components (Oracle BPEL Process Manager, Oracle Mediator, and human workflow) have their own creation dates that are used in the console queries:
Example for Oracle BPEL Process Manager:
-
Partition key: Table
CUBE_INSTANCE
columnCPST_INST_CREATED_TIME
-
Creation date: Table
CUBE_INSTANCE
columnCREATION_DATE
Purging Partitions
Circumstances may arise that require the partitioned tables to be purged, especially if the row migration scripts are used. This defeats the purpose of partitioning the table for maintenance, which can remove data in bulk through the ALTER
TABLE
... DROP
PARTITION
command.
Attempting to purge the partitioned tables most likely causes the parsing performance problem. If partitions must be purged, then also shrink the partitions. For more information, see Understanding Space Management.
Reducing Audit Levels
This section provides tuning information about how to reduce the audit level for SOA composite applications and the data written to the Oracle SOA Suite schema.
Setting the Audit Level for Production Environments
The default Audit Level is set to Production at the SOA Infrastructure level. To limit database growth, and for optimal performance, Oracle recommends that you turn Off the setting at the SOA Infrastructure level, and instead turn on auditing at the composite level for composites that require debugging or monitoring. Your compliance requirements might also determine the granularity of your audit requirements.
The following image depicts setting the audit level for a composite from the composite home page:
Note:
You can change the audit level setting at the SOA Infrastructure level using the SOA Infrastructure Common Properties page in Oracle Enterprise Manager Fusion Middleware Control. See Configuring SOA Infrastructure Properties for information about turning off audit at the SOA Infrastructure level.
See Managing the State of an Application from the SOA Composite Application Home Page for more information about setting the audit levels at the composite level.
If you can disable auditing (provided the business requirements allow for it), there are substantial performance improvements. Changing Production to Off can result in improvements (without tuning the individual component audit settings). However, if you cannot disable auditing, Oracle recommends that you consider the following steps to reduce audit levels.
For synchronous BPEL components for which the audit trail is not required, use the following settings:
-
inMemoryOptimization
=true
-
completionPersistPolicy
=faulted
With these settings, the synchronous process only appears in the audit trail if the instance faults.
For information about setting inMemoryOptimization
and completionPersistPolicy
, see "Deployment Descriptor Properties" in Developing SOA Applications with Oracle SOA Suite.
Note:
Use these settings for synchronous BPEL processes and not for asynchronous BPEL processes.
Connection and Transaction Timeout Troubleshooting
This section describes how to troubleshoot connection and transaction timeout issues.
Resolving Connection Timeouts
You can receive a connection timeout error under circumstances such as the following:
-
You run a SOA composite application with a large payload that takes more than 30 seconds to process.
-
You are invoking a stress test using a large payload from the Test Web Service page of Oracle Enterprise Manager Fusion Middleware Control.
-
You are passing a large number of message files (one million) into a composite with a file adapter service.
To avoid these timeout errors, increase the transaction timeout property as follows:
- Log into Oracle WebLogic Administration Console.
- Click JTA.
- Change the value of Timeout Seconds (the default is
30
). - Click Save.
- Restart Oracle WebLogic Server.
Resolving Email Notification Timeouts
You can receive a connection timeout errors when a large number (for example, 200,000) of notification emails are sent.
To avoid these timeout errors:
Increasing Database Connection Values
You can receive the error message shown below because of slow connections to the database:
090304)): oracle.toplink.exceptions.DatabaseException Internal Exception: java.sql.SQLException: Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceDeadException: Pool SOADataSource has been disabled because of hanging connection tests, cannot allocate resources to applications.
If this occurs, perform the following steps:
Updating the EJB Transaction Timeout Value in the Deployment Archive After SOA Infrastructure Failure
Updating the transaction timeout value for the FacadeFinderBean property in Oracle WebLogic Server Administration Console under Deployments > expanded SOA Infrastructure Application > FacadeFinderBean > Configuration tab can result in the error shown below after restarting the SOA Infrastructure:
java.lang.IllegalArgumentException: Cannot convert value of type [$Proxy223 implementing oracle.bpel.services.workflow.verification.IVerificationService,org.springframe work.aop.SpringProxy,org.springframework.aop.framework.Advised] to required type [oracle.bpel.services.workflow.verification.IVerificationService] for property 'verificationService': no matching editors or conversion strategy found Message icon - Warning Errors were encountered while performing this operation.
The SOA Infrastructure status is also displayed as failed.
This error is not specific to FacadeFinderBean; it also applies to any EJB that is part of the SOA Infrastructure application.
To resolve this error, you must manually modify the transaction timeout setting in your deployment archive.
To update the transaction timeout setting:
Long Running, Synchronous Calls To Remote Web Services Error Out or Asynchronous Transactions Return with an Error after a Long Time
You may have long running, synchronous calls to remote web services that end with JTA transaction rolled-back errors. When executing a transaction making an asynchronous call (for example, to the SOA server), the application may return with an error. The server logs show JTA transaction timeouts, which can cause this behavior.
To check the JTA transaction timeout in the Oracle WebLogic Server Administration Console.
-
Log in to Oracle WebLogic Server Administration Console.
-
In the Domain Structure, select Services > JTA to check the timeout value.
If the transaction is always timing out beyond 30 seconds and is a custom composite synchronous client invocation, then you may need to revisit the design approach. It may be best for the external web service to be invoked as an asynchronous transaction.
Increasing the JTA for supporting long running, synchronous transactions is simply an interim mechanism. For information on changing the JTA transaction timeout setting, see Resolving Connection Timeouts.
-
If synchronous client invocations take a long time, check for any performance issues with the system and try to resolve them.
-
Check for appropriate values for the syncMaxWaitTime property and BPEL's EJB transaction timeout settings in relation to the JTA timeout settings and only then increase the value of the JTA timeout, if needed.
For information on viewing and changing the syncMaxWaitTime property, see Configuring BPEL Process Service Engine Properties.
-
To view and change the BPEL EJB transaction timeout settings, perform the following steps:
-
Log in to Oracle WebLogic Server Administration Console.
-
In the Domain Structure, click Deployments.
-
Expand soa-infra > EJBs.
-
Update the following EJBs:
- BPELActivityManagerBean
- BPELDeliveryBean
- BPELDispatcherBean
- BPELEngineBean
- BPELFinderBean
- BPELInstanceManagerBean
- BPELProcessManagerBean
- BPELSensorValuesBean
- BPELServerManagerBean
-
Click Save.
-
Restart Oracle WebLogic Server.
-
-
For asynchronous transactions, check the values for both the BPEL EJB transaction timeout and the JTA transaction timeout and adjust, as needed.
Increasing the HTTP POST Timeout Value to Resolve Broken Pipe Errors
If you receive the broken pipe error as shown below, increase the HTTP POST timeout value in Oracle WebLogic Server Administration Console.
<BEA-000000> <got FabricInvocationException, Cikey=73410002, FlowId=10001, Current Activity Key=73460002-BpInv0-BpSeq0.3-3, Current Activity Label=Invoke_1, ComponentDN=default/TestClientWSMEDWSBPELFAProj!2.0*soa_ b2e412f3-1167-4a9a-bbe0-73795f0743d4/BPELProcess1 java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
To increase the HTTP POST timeout value:
- Log in to Oracle WebLogic Server Administration Console.
- In the Domain Structure, expand Environment > Servers.
- In the Name column, click the SOA server.
- At the top of the page, click the Protocols tab.
- In the Post Timeout field, increase the value to a maximum of
120
.
Resolving Exception Errors When Processing Large Documents
If you receive the error shown below when processing large attachments, increase EJB property timeout values.
java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: Transaction rolled back: Transaction timed out after 501 seconds
To increase EJB property timeout values:
Runtime Diagnostics Troubleshooting
This section describes how to troubleshoot runtime issues.
Unavailability of Work Manager Threads for Incoming Processing
The exception message shown below occurs because there are a fixed number of work manager threads allocated for incoming processing, and Oracle SOA Suite has reached the thread limit. Oracle SOA Suite gracefully backs out of processing to prevent a resource bottleneck that may eventually lead to a stuck thread scenario. If you receive this error, you must resubmit your request.
<Mar 11, 2014 2:10:16 PM PDT> <Error> <HTTP> <BEA-101017> <[ServletContext@207971169[app:soa-infra module:/soa-infra path:null spec-version:3.0]] Root cause of ServletException. java.lang.Exception: Unable to acquire the workmanager thread resource in order to process requests directed at compsoite default/POProcessing!1.0*soa_7a9c6320-c26e-4df2-b29f-7b80782d3693 at oracle.integration.platform.blocks.deploy.CompositeLazyLoader.load(CompositeLa zyLoader.java:112) at . . . . . .
Oracle SOA Suite Runtime Failure with a "Cannot read WSDL" Error
The following error is displayed if either the endpoint is not available for a reference or a composite is deployed with an incorrect deployment plan file.
oracle.fabric.common.FabricException: Cannot read WSDL
To diagnose this issue:
Automatic Recovery of BPEL Instances is Not Recovering a Specific Instance
BPEL processes have an automatic recovery feature that attempts to automatically recover activities that are recoverable such as unresolved invoke and callback messages, activities not completed over a provided threshold time, and so on. However, the automatic recovery feature only tries to recover a few instances and only retries a fixed number of times. If some instances are not being automatically recovered, they are likely not being picked up because of the configuration of the automatic recovery parameters.
To resolve this issue:
-
Set the maximum number of messages to automatically recover.
By default, the automatic recovery feature of Oracle BPEL Process Manager processes 50 messages to submit for each recovery attempt. This is controlled by the maxMessageRaiseSize property.
-
In the navigation pane, right-click soa-infra (SOA_cluster_name).
-
Select SOA Administration > BPEL Properties > More BPEL Configuration Properties > RecoveryConfig.
-
Expand both RecurringScheduleConfig > maxMessageRaiseSize and StartupScheduleConfig > maxMessageRaiseSize.
The default value is
50
for each. A negative value causes all messages selected from the database to be submitted for recovery. A value of0
causes no messages to be selected from the database (effectively disabling recovery). To recover more than 50 messages, set the property value to that value. Use this property to limit the impact of recovery on the server.
-
-
Set the maximum number of automatic recovery attempts on a given message.
You can also configure the number of automatic recovery attempts to submit in the same recoverable instance. The value you provide specifies the maximum number of times that invoke and callback messages are recovered. If the value is
0
(the default value), it recovers all messages. Once the number of recovery attempts on a message exceeds the specified value, a message is marked as nonrecoverable.To configure automatic recovery attempts for invoke and callback messages in Oracle Enterprise Manager Fusion Middleware Control:
-
In the navigation pane, right-click soa-infra (SOA_cluster_name).
-
Select SOA Administration > BPEL Properties > More BPEL Configuration Properties.
-
Select MaxRecoverAttempt, and enter a value in the Value field.
-
Click Apply.
It may not be desirable in all cases to use automatic recovery. If services are not idempotent, then corruption can occur. Moreover, the automatic recovery restores the composite to the last save point that can be immediately after an asynchronous invoke, wait, and so on. Therefore, it is important to understand the process behavior and what it does next before performing mass recoveries. Attempt mass automatic recovery only after the root cause of the composite failures is fixed (for example, a service that was unavailable is now available, a database running out of space was fixed, and so on). Automatic recovery can also trigger an unexpected load during failure scenarios. This causes more threads to block on a remote server that can induce hangs in the SOA server in a cascading fashion.
For more information, see Configuring Automatic Recovery for Oracle BPEL Process Manager.
-
Some Composites Are Retried Multiple Times on Failure
When a BPEL process flow errors out, it is retried with all its invocations. This is undesirable in some cases.
The property GlobalTxMaxRetry (default value is 3
) specifies how many retries are performed if an error is identified as retriable. For example, after several web service invocations, if dehydration fails due to a data source error, then this is identified as a retriable error and all activities from the prior dehydration state are retried. If the activities being retried are not idempotent (that is, their state can change with each retry and is not guaranteed to give the same behavior), then multiple retries can be problematic.
To rectify this situation, customize the composite by specifically marking the nonidempotent activities with idempotent
set to false
in the partner link settings section of the composite.xml
file to prevent retries.
<property name="bpel.partnerLink.partner_link_name.idempotent">false</property>
You cannot set the idempotent
property in Oracle Enterprise Manager Fusion Middleware Control.
You can also set GlobalTxMaxRetry to 0
in the Systems MBean Browser.
To set GlobalTxMaxRetry:
- Right-click soa-infra (SOA_cluster_name).
- Select SOA Administration > Common Properties.
- Click More SOA Infra Advanced Configuration Properties.
- Click GlobalTxMaxRetry.
- In the Value field, enter an appropriate value.
- Click Apply.
For more information about the idempotent
property, see "Managing Idempotence at the Partner Link Operation Level" section of Developing SOA Applications with Oracle SOA Suite.
Application Transaction Does Not Complete and the Underlying Composite is Stuck in a Running State
Assume an application transaction is not completing. For example, a purchase order status may remain processing. Checking the business flow instance shows that the composite is stuck in the running state. In this case, the component is probably not running. Instead, it has likely faulted and may need recovery.
Faults may occur for various reasons:
-
A BPEL activity faulted with an error (for example, a business error, security authorization error, or some other error).
-
A BPEL activity invoked an external web service that was unavailable.
-
A BPEL activity has already been terminated by the administrator using Oracle Enterprise Manager Fusion Middleware Control.
-
A BPEL activity invoked an asynchronous ADF service and the message is stuck in the AQ JMS queue.
-
A BPEL activity invoked an asynchronous ADF service, but because Oracle SOA Suite was unavailable, the callback message did not arrive.
-
A BPEL activity invoked a synchronous ADF service, which is taking a long time (or is hanging).
-
A network error occurred.
To diagnose this issue:
-
Log in to Oracle Enterprise Manager Fusion Middleware Control.
-
In the navigator pane, go to domain_name > SOA.
-
Click soa-infra (SOA_cluster_name).
-
Click the Flow Instances tab.
-
Search for the business flow instance, and click the flow ID.
The Flow Trace page appears.
If the instance is not visible (and the Audit Level is not set to Off on the SOA Infrastructure Common Properties page), this implies that the message is stuck outside of Oracle SOA Suite.
If the message has reached Oracle Mediator, but not instantiated the BPEL flow, the BPEL instance may have been rolled back from the start due to an error.
If the BPEL flow exists, the Faults section of the Flow Trace page typically shows the faulted service that can trace the root cause.
-
In the Trace section, click the BPEL process.
-
Expand the BPEL audit trail to see the exact point at which the service faulted.
This information is also available in the Faults tab of the BPEL flow trace. It also indicates whether the fault can be recovered.
-
Click the View Raw XML link.
The same information is also available through this link, where you can see the error. For example:
... <message>Faulted while invoking operation "modifyUserRoles" on provider "UserService". </message><details> ... <tns:message>JBO-27023: Failed to validate all rows in a transaction.</tns:message> <tns:severity>SEVERITY_ERROR</tns:severity> ... <message>The transaction was rolled back. The work performed for bpel instance "451042" was rolled back to the previous dehydration point, but the audit trail has been saved. You can recover the instance from the recovery console by resubmitting the callback message or activity for execution.</message
Since the instance was rolled back to its previous dehydration point, the status remains as Running.
-
In the Audit Trail tab, make a note of the following:
-
Composite name (for example, POComposite)
-
Component (for example, UpdateGuid BPEL process)
-
BPEL instance ID (for example, bpel:451042)
This is all used in the recovery of the instance, if it is recoverable. The Audit Trail may mark the error as a nonrecoverable business fault, but the recoverability of the message can be found in the Recovery tab of the BPEL process service engine.
-
-
To attempt to recover the instance, right-click soa-infra (SOA_cluster_name), and select Service Engines > BPEL.
-
Click the Recovery tab.
-
From the Type list, select Activity.
-
Specify the composite and component names captured in step 9, and click Search.
-
Find the specific BPEL instance ID. You can recover faults marked as Recoverable.
-
Check the other recovery options in the Type list (for example, Invoke and Callback), if they exist.
Note:
You can also search for recoverable messages from the Flow Instances page of the SOA Infrastructure.
-
If the instance is not marked as recoverable, then reinvoking the service is not allowed (most probably because it is not idempotent). In some cases, you may need to provide diagnostic information to Oracle Support Services to resolve issues with nonrecoverable, nonidempotent transactions.
-
If the BPEL activity has invoked an asynchronous ADF service and the message is stuck in the AQ JMS queue, you can view the
server
.log
andserver
-diagnostic.log
files to see the logging of the message metadata logged by the JRF web services infrastructure.In addition, the ADF diagnostic logs are also available to debug, if needed.
-
Use the ECID field to correlate and track ADF service logging corresponding with the SOA composite application that invoked it.
-
When viewing the log in Oracle Enterprise Manager Fusion Middleware Control, click the Broaden Target Scope dropdown list and select the farm_name/domain_name (Oracle WebLogic domain) to view messages across the domain.
-
In the Selected Targets section of the Log Messages page for the Oracle WebLogic Server domain, ensure that the search includes the ECID field with the value noted in step 17.a.
-
Search and view log records for the execution content ID (ECID) and note any issues.
For a specific ECID, you find several root instances (top level clients). You must drill down to the appropriate instance to find a specific fault.
-
Observe if the components completed successfully or completed with an error. See the "Viewing and Searching Log Files" section in the Administering Oracle Fusion Middleware.
-
See the following chapters for the various ways to recover transactions and messages:
-
Managing SOA Composite Application Business Flow Instances to recover from the SOA composite application page in Oracle Enterprise Manager Fusion Middleware Control.
-
Managing BPEL Process Service Components and Engines to recover from the BPEL process service component and BPEL process service engine message recovery pages.
Errors During Analytics Measurement Event Processing
When errors occur during processing of BPMN, Human Workflow, CASE analytics measurement events for any reason like target BAM is down or the database is not available and so on, (i) analytics message processing related errors are seen in the logs and (ii) BPM_MEASUREMENT_ACTION_EXCEPS table also has one or more records of these errors. You can republish these events by using any of the two methods.
Using Measurement Action Service client
Use republishErroredEvents operation to republish errored measurement events of the past 24 hours. This operation is available as part of AnalyticsConfig mbean To republish the error events, perform the following activities. Enter password when prompted..
java [CLASSPATH] [PROPERTIES] oracle.bpm.metrics.action.MeasurementActionRecoveryClient
CLASSPATH:
-classpath $MW_HOME/soa/soa/modules/oracle.bpm.runtime_<version number of instance>/oracle.bpm.analytics.metrics.interface.jar:
$MW_HOME/soa/wlserver/server/lib/weblogic.jar
Enter the following required properties.
-Dhost=<host> (soa server host name)
-Dport=<port> (soa server port)
-Duser=<admin user> (soa server admin user name)
-DcompositeName=<soa composite name> (soa server host name)
Enter the following optional properties depending on your need.
-Drealm = <hostrealm Name> (default is jazn.com)
-Dprotocol=<protocol> (jndi protocol. Default is t3)
-DinitialContextFactory = <jndi initial context factory> (Default is weblogic.jndi.WLInitialContextFactory)
-DmeasurementAction=<action name> (measurement action name. Default is ProcessMetrics (BAM 12c))
-DrangeStartDate=mm/dd/yyyy (Range start date . Default is 24 hours before rangeEndDate )
-DrangeEndDate=mm/dd/yyyy (Range end date. Default is current time)
-Ddebug=true/false (if true, prints the exception stacktraces in case of any error. default is false)
Using Oracle Enterprise Manager AnalyticsConfig MBean - republishErroredEvents operation
republishErroredEvents operation is also available as part of AnalyticsConfig mbean (mbean object name = oracle.as.soainfra.config:name=analytics,type=AnalyticsConfig,Application=soa-infra)
. Oracle Enterprise Manager administrator can access and republsih errored messages of the last 24 hours.
-
Log into Oracle Enterprise Manager
-
Expand oracle.as.soainfra.config, Server:AdminServer, AnalyticsConfig and select analytics.
-
Open Operations tab in the right pane.
-
Click on republishErroredEvents.
-
Provide the necessary details as described above.
-
Click Invoke.
Human Workflow Troubleshooting
This section describes how to troubleshoot human workflow issues.
Unavailability of Human Workflow Service Engine
During SOA server startup, you can receive the exception shown below. This exception occurs because the SOA server is still initializing. Wait for SOA server initialization to complete.
ORABPEL-30132 exception.code:30132 exception.type: ERROR exception.severity: 2 exception.name: Workflow Service Engine not ready. SOA server is still starting. exception.description: Could not obtain instance of Workflow Service Engine as the SOA server is still initializing. exception.fix: Wait for SOA server to finish starting up, and retry.
For information about monitoring SOA server initialization status in Oracle Enterprise Manager Fusion Middleware Control, see Viewing the Overall Runtime Health of the SOA Infrastructure.
Task Assignment/Routing/Escalation Issues
Table B-2 describes symptoms, possible causes, and possible solutions for task assignment/routing/escalation issues.
Table B-2 Troubleshooting Task Assignment/Routing/Escalation Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
The task completes without any assignment occurring. |
The most common problem is that task assignees are specified using XPath expressions, and the expression does not evaluate to any nodes. Other problems can include incorrect skip conditions for participants. |
|
The business rules do not return any list builders. |
When participants of a task are specified using business rules, it is expected that business rules return at least one list builder. If business rules determine that no participants are needed, the function |
|
The business rules return list builders of different types. |
When participants in a task are specified using business rules, it is expected that business rules return list builders of the same type. |
Correct your rules. |
A human workflow task chooses the incorrect user if many rules are defined or it errors with the following message: Ruleset returned lists with different list builder |
At runtime, when a human workflow task tries to fetch the list of users, it may error out with the following error: Ruleset returned lists with different list builder This error is displayed in the Task Detail comments field. Alternately, the task may select a user or approver, which may not appear to be the correct or expected one. This is primarily caused by having overlapping rules. When the participants of a task are specified using business rules, it is expected that business rules return list builders of the same type. Moreover, only one rule from a ruleset must be applicable for a transaction. In case many rules are true, the actions associated with the applicable rule with the highest priority get executed. In case multiple applicable rules have the same priority, then the first rule in the list is picked and its actions executed. |
Avoid writing overlapping rules. Constraints from different list builders are different and cannot be mixed. If multiple rules get triggered with a different list builder, this error occurs. In addition, only one set of constraints is honored. Check that all rules in the ruleset have priorities defined so that multiple rules with the same priority are not applicable for the same transaction. For more details., see Developing Business Processes with Oracle Business Process Management Studio. |
Parallel assignees have to approve or reject the task even though the parallel completion criteria is met. |
In the Add Participant Type dialog for a parallel participant, you can configure the human task during runtime to wait for all parallel participants to complete or to complete when criteria are met. |
Make the correct selection for completion in the Add Participant Type dialog. |
The task is assigned to the group/role when the expectation is that it goes to every user in the group/role individually. |
When a group or a role is used as a task assignee, the task is assigned to the group or role directly. Task runtime does not assign it separately. One of the users in the group/role has to claim the task and work on it. When used with a parallel or serial participant, often times it is expected that this resolution to users is automatic, which it is not. |
To assign separately to the members of the group or role, use the XPath functions |
A task errors out when invoking the decision service for the evaluation of routing rules or rule-based participants. |
Payload validation is enabled on the SOA Infrastructure instance. |
Deselect the Payload Validation checkbox for the instance. For more information, see Configuring SOA Infrastructure Properties. |
Task Action Issues
Table B-3 describes symptoms, possible causes, and possible solutions for task action issues.
Table B-3 Troubleshooting Task Action Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
A user is not allowed to perform an action on a task. |
The most common problem is that the user does not have permissions to perform that action on that task at that point in time. |
Find out if the user can be an assignee, owner, or creator of the task, or if they are an administrator. If the user should have been allowed to perform the action, check the server log file for a detailed log message, which includes information such as the task state, task assignees, user who acquired it, permitted actions, roles played by this user for the given task, and so on. |
A task is locked and cannot be updated in Oracle BPM Worklist. An |
Two users are trying to update the same task concurrently. |
Wait for the other user to finish. |
Notification Issues
Table B-4 describes symptoms, possible causes, and possible solutions for notification issues.
Table B-4 Troubleshooting Notifications Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
You may receive the following error message: [2012-07-05T09:13:05.246-04: 00] [soa_server1] [ERROR] [] [oracle.soa.services.workflow .common] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <anonymous>] [ecid: dc2ff0568380b04f:23824bc7:13 8570a4e67:-8000-000000000000 1716,0] [APP: soa-infra] <.> [[ORABPEL-0 at oracle.bpel.services.workflo w.task.notification.TaskNoti fications.notifyForTask(Task Notifications.java:481) . . . . . . |
Check if you have apostrophes in the body of an email notification message. <%string('À titre de responsable fonctionnel, une demande d'approbation . . . |
Escape each apostrophe by adding double apostrophes in your string. For example, if you have <%string('d'd')%> to: <%string('d''d')%> |
The email notification is not being sent out. |
Incorrect outgoing (SMTP) server settings are used in the email driver configuration. |
Check the Simple Mail Transfer Protocol (SMTP) port/SMTP host/user name/password/email values. Tip: Validate the values by using them in any email client for connecting to the SMTP server. Perform the following steps to verify the settings in Oracle Enterprise Manager Fusion Middleware Control:
|
The task email notification is not being sent out. |
Notification Mode is set to NONE on the Workflow Notification Properties page in Oracle Enterprise Manager Fusion Middleware Control. |
Change this setting to Email or All. For information, see Configuring Human Workflow Notification Properties. |
Notifications are not being sent because of the following error message: No matching drivers found for sender address = <address> |
The User Messaging Service driver for the appropriate channel is configured with a specific list of sender addresses, and the message sent by the application has set a nonmatching sender address. Note: The User Messaging Service server matches the outbound message's sender address, if set, against the available drivers' sender addresses to find a matching driver to use for delivering the message. If a driver has set one or more sender addresses, then the User Messaging Service server only sends messages with the matching sender address to it. |
Perform the following tasks:
|
The notifications are sent, but are not actionable. |
The Actionable Address field is not configured on the Workflow Notification Properties page in Oracle Enterprise Manager Fusion Middleware Control. |
In Oracle Enterprise Manager Fusion Middleware Control, configure the Actionable Address field with a valid email address. For information, see Configuring Human Workflow Notification Properties. Ensure that the same email address is used when configuring the incoming server setting in the User Messaging Service email driver. For information, see Configuring Human Workflow Notification Properties. |
Notifications are sent, but are not actionable. |
The human workflow task is not set to send actionable notifications. |
In the Human Task Editor (you can double click the |
Actionable notifications are sent, but no action is taken after responding. |
The Actionable Address field is incorrect. |
Check the IMAP/POP3 server/port values. Ensure the Actionable Address field is used in the email driver configuration. Tip: Validate the values by using them in any email client for connecting to the IMAP/POP3 server. |
Actionable notifications are sent, but no action is taken after responding. |
The nondefault email client is configured for receiving notifications. |
When the user clicks the approval link, the default mail client page opens, which may send emails to a different email server. Configure the default email client to receive actionable notifications. Enter the correct value in the Actionable Email Account field of the Workflow Task Service Properties page as the incoming, actionable email account to use. The default account name is Default. For information, see Configuring Human Workflow Task Service Properties. |
Human workflow actionable notifications are sent, but no action is taken after responding. |
Incoming mail server settings for the email driver are incorrect. |
Check the following email driver settings in Oracle Enterprise Manager Fusion Middleware Control:
|
Actionable notifications are sent but no action is taken after responding. |
An email client is configured with the same account used in the email driver. |
The mail may be downloaded and marked as read or deleted by the email client before the human workflow notification service can download and process the mail. Remove that account from the email client. |
User notifications or human workflow notifications are sent through the correct delivery type (email, SMS, and so on), but to the wrong address. |
A self-provisioned messaging channel was created by the user in User Messaging Preferences for use in a BPEL process user notification or human workflow notification. Note: The User Messaging Preferences user interface enables you to create your own messaging channel for various use cases. However, do not use these channels for BPEL process user notification and human workflow. |
Do not use a self-provisioned messaging channel for BPEL process user notification and human workflow use cases (that is, do not set as default channel and do not use in a messaging filter for such use cases). BPEL process user notification and human workflow use User Messaging Preferences only for the delivery type preference. The the actual address is retrieved from the user profile in the identity management system. Note: Addresses from the user profile in the identity management system are available through User Messaging Preferences using predefined channel names, such as business email, business mobile, business phone, and instant messaging. Use these predefined messaging channels instead for bpel process user notification and human workflow use cases. |
The Oracle BPM Worklist link appears in email notifications. |
This is the default behavior. By default, email notifications point to Oracle BPM Worklist. |
Perform the following steps:
|
Performance is slow for group notifications. |
The group notification performance depends on the number of members in the group (size of group). |
|
Task View Issues
Table B-5 describes symptoms, possible causes, and possible solutions for task view issues.
Table B-5 Troubleshooting Task View Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
Showing custom (mapped attribute) columns in a view. |
Attribute mappings are created for specific task types. The view must be associated with one or more task types to use mapped attributes. |
In Oracle BPM Worklist (view/create/edit UI), specify a task type for the view in the Definition tab. The attribute labels used in the mappings for that task type are now available as columns that can be used in the view in the Display tab. It is possible to associate a view with multiple task types. Multiple task types can be selected from the Task Type browser. If multiple task types are selected, then the attribute labels for all those task types are available for use in the view. |
View grantees can view and edit tasks belonging to the view owner. |
The view is shared as data. This type of sharing allows grantees to use the view as if they are the view owner, and can see and act on the view owner's task. |
In the Definition tab of Oracle BPM Worklist (view/edit UI), ensure that Share View is set to Definition only, which enables grantees to use the view against their own tasks. Setting Share View to Data enables grantees to use the view against the view owner's data. |
Creating a new standard view. |
Only users with administration privileges can create standard views. |
|
Internationalizing a standard view name. |
The value specified in the name field for standard views can be used as a resource key to look up a display name from the |
Add a new resource key to the |
Migrating views and standard views you have created on one instance to another SOA server. |
You must use the test-to-production utility. |
The test-to-production utility enables you to export user views and standard views as an XML file, and to import the views from the XML file into another instance. For information about this utility, see Moving Human Workflow Data from a Test to a Production Environment. |
Task Attribute Mapping Issues
Table B-6 describes symptoms, possible causes, and possible solutions for task attribute mapping issues.
Table B-6 Troubleshooting Task Attribute Mapping Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
No payload attributes are available for mapping to a public attribute in Oracle BPM Worklist. |
Oracle BPM Worklist only supports creation of mappings to simple payload attributes. Only simple attributes from the task payload are displayed for creating mappings in Oracle BPM Worklist. |
|
You cannot create mappings for the protected attribute label in Oracle BPM Worklist. |
Protected mappings can only be created as part of the task definition at design time. Protected mappings cannot be created or updated at runtime using Oracle BPM Worklist, or the |
|
You cannot see any attribute labels for which to create mappings in Oracle JDeveloper. |
Design-time mappings can only be created for protected attribute labels. Ensure that protected attribute labels have been created in the SOA instance to which you are connected. |
|
Internationalizing the name of an attribute label. |
You can use the attribute label name as a resource key to look up a display name from the |
Add a new resource key to the |
Migrating attribute labels and mappings from one server to another. |
Use the test-to-production utility. |
The test-to-production utility enables you to export public attribute labels, public attribute mappings, and protected attribute labels as an XML file, and to import the labels and mappings from the XML file into another instance. For more information, see Moving Human Workflow Data from a Test to a Production Environment. |
Task Report Issues
Table B-7 describes symptoms, possible causes, and possible solutions for task report issues.
Table B-7 Troubleshooting Task Report Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
You receive the following error: Null Pointer Exception when running Task Productivity Report |
This is caused by an issue with the handling of dates when the worklist client locale and server default locale are different. |
The workaround is to change the locale for the worklist client to be the same as the server, or to run a report without specifying dates. |
Task History Issues
Table B-8 describes symptoms, possible causes, and possible solutions for task history issues.
Table B-8 Troubleshooting Task History Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
The Add Participant button is disabled. |
A current or past participant is selected in the history table. |
This is designed behavior. Adding adhoc participants is not allowed with the current or past participant. The current participant means the task is with that participant at that point in time. |
All the added adhoc participants disappeared after a page refresh. |
You may not have saved your modifications to the history table. |
Ensure that you save your changes. Otherwise, all changes disappear. If you think you have saved your changes and the changes still disappear, file a bug. |
Do not see future approvers in the history table. |
The Future Approvers checkbox may not be selected. |
Select the Future Approvers checkbox in Oracle BPM Worklist (configuration in the task sequence table). |
You see the message in the history table about the correlation ID not being passed or any exception related to the correlation ID. |
If the task is uninitiated, the correlation ID may not have been passed. |
Ensure that you pass the correlation ID to the uninitiated task. |
The edit toolbar is disabled or is not displayed. |
The user may not have privileges to edit the participants. |
|
You receive the following error: <Warning> <oracle.adf.controller.intern al.metadata.MetadataService> <BEA-000000><ADFc: /META-INF/adfc-config.xml: > <Warning> <oracle.adf.controller.intern al.metadata.MetadataService>< ADFC-52024> <ADFc: Duplicate managed bean definition for 'aleCompBindings' detected.> |
Shared library |
These JARs ideally should not be packaged inside the web application. They should only be referenced as a shared library. Do not package these JARs in the web application. |
You receive the following error: <Error> <Deployer> <BEA-149265> <Failure occurred in the execution of deployment request with ID '1297964056778' for task '3'. Error is: 'weblogic.management.Deployme ntException: [J2EE:160149]Error while processing library references. Unresolved application library references, defined in weblogic-application.xml: [Extension-Name: oracle.soa.workflow.wc, exact-match: false].' weblogic.management.Deploymen tException: [J2EE:160149]Error while processing library references. Unresolved application library references, defined in weblogic-application.xml: [Extension-Name: oracle.soa.workflow.wc, exact-match: false]. |
Shared library |
Ensure that this shared library is deployed on the server to which you are deploying your application. It may happen that the shared library is deployed, but not targeted, for that server. |
You receive the following error: java.lang.IllegalStateExcepti on: Attempt to validate an already invalid RegionSite: |
This is a generic exception that sometimes is displayed in the server logs (for example, |
See the real exception in the diagnostic logs (for example, |
You receive the following error: [AdminServer] [NOTIFICATION] [J2EE JSP-00008] [oracle.j2ee.jsp] [tid: [ACTIVE].ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 17011f2a001d6b0e:7e22d6ce:12e 3444eb1b:-8000-0000000000002f 0a,0] [APP: FederatedApp_ application1] unable to dispatch JSP page: The following exception occurred:.[[ java.lang.RuntimeException: Cannot find FacesContext at javax.faces.webapp.UIComponen tClassicTagBase.getFacesConte xt(UIComponentClassicTagBase. java:2122) |
This is a common mistake and is not related to any components you are using. You forget to put http://server:port/FederatedApp /test.jspx |
Put http://server:port/FederatedApp/fac es/test.jspx |
Task Form/Action Issues
Table B-9 describes symptoms, possible causes, and possible solutions for task form/ action issues.
Table B-9 Troubleshooting Task Form/ Action Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
The task form application does not have an empty JSPX page. |
N/A |
The task forms are now invoked using an ADF task flow and control is returned to the module that initiated the task form task flow when the task flow completes. Therefore, no empty JSPX is needed. |
The task form does not load in Microsoft Internet Explorer. |
Microsoft Internet Explorer has a URL length limit. |
Your task form URL length is too long. |
Deployment fails with a |
The shared library entry is missing from |
If you see the following error: Caused By: java.lang.ClassNotFoundException: oracle.bpel.services.datacontrol.ty pes.Number during deployment of a task form, then it is likely due to the missing shared library in <library-ref> <library-name>oracle.soa.worklist.w ebapp</library-name> <specification-version>11.1.1</spec ification-version> </library-ref> |
Deployment/access of task form fails when the hostname is used. |
The DNS entry is missing. |
If you are using a server with DHCP, the DNS entry may be missing for the host. Therefore, deployment/access using the IP address may succeed, but deployment/access using a hostname may fail. Update your client computer by manually adding the host/IP address:
|
Task form URL protocol (HTTP or HTTPS). |
You are unable to access the task form through HTTPS or HTTP. |
|
Note:
If you receive a java.lang.OutOfMemoryError:
PermGen
space
error when deploying multiple task forms, you may need to increase PermGen
memory. For more information, see PermGen Memory Requirements for Multiple ADF Task Form Deployments.
Task Comments/Attachment Issues
Table B-10 describes symptoms, possible causes, and possible solutions for task comments/attachment issues.
Table B-10 Troubleshooting Task Comments/Attachment Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
The file is not getting uploaded. |
The file is too big |
By default, ADF has a size limit of 2000 KB for each request. Add the following parameters in <context-param> <!-- Maximum memory per request (in bytes) --> <param-name>oracle.adf.view.faces.U PLOAD_MAX_MEMORY</param-name> <!-- Use 500K --> <param-value>512000</param-value> </context-param> <context-param> <!-- Maximum disk space per request (in bytes) --> <param-name>oracle.adf.view.faces.U PLOAD_MAX_DISK_SPACE</param-name> <!-- Use 5,000K --> <param-value>5120000</param-value> </context-param> <context-param> <!-- directory to store temporary files --> <param-name>oracle.adf.view.faces.U PLOAD_TEMP_DIR</param-name> <!-- Use an ADFUploads subdirectory of /tmp --> <param-value>/tmp/ADFUploads/</para m-value> </context-param> |
The file uploaded in the task details application is not visible in the same task flow. |
After uploading a file, the attachment link generated in the task form is invalid. Clicking this link returns an empty stream. |
When you upload a file, you see the attachment link in the table. However, this link does not work. You must reload the task details to view the file. |
Adding file attachments creates a new task version, but adding a URL attachment does not create a new version. |
Inconsistent behavior of the URL attachment and file attachment. |
When a file is uploaded, the task is saved because the file is uploaded to persistency storage. This creates a new task version. The URL attachments only update the local task object in the user interface application. Therefore, no task version is created. |
Design Time at Runtime Issues
Table B-11 describes symptoms, possible causes, and possible solutions for design time at runtime issues. Two design time at runtime tools are available for use:
-
Oracle SOA Composer
-
Task Configuration tab of Oracle BPM Worklist
Table B-11 Troubleshooting Design Time at Runtime UI Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
Modifications made to a task in a design time at runtime tool do not appear for the task. |
The task was instantiated before you actually edited it using a design time at runtime tool. |
Design time at runtime updates go into effect only for instances created after the changes, and not for those that were created before the change. Therefore, if you edit a task using a design time at runtime tool, and then instantiate a new task, the new instance of the task has the changes you made. |
Modifications made to a task in a design time at runtime tool do not appear for the task. |
The changes made were probably not committed to the MDS repository. |
The Save button just saves the changes made in a design time at runtime tool to the sandbox. To see these changes in action, click Commit to send them to the MDS repository. |
Human Workflow API (Including SOAP/EJB) Usage Issues
Table B-12 describes symptoms, possible causes, and possible solutions for human workflow API (including SOAP/EJB) usage issues.
Table B-12 Troubleshooting Human Workflow API Usage Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
Location of the JavaDoc for human workflow APIs. |
N/A |
See Workflow Services Java API Reference for Oracle SOA Suite, which is available in the documentation library. |
Understanding the API usage. |
N/A |
|
Using |
N/A |
It is possible to write a |
You receive |
Not all required JAR files are in the client class path. |
- |
Creating a routing slip for simple patterns to use with a simple approval task or to dynamically route a task during task initiation. |
N/A |
See |
Newly created task does not appear in Task UI |
|
Deploy |
Oracle JDeveloper Data Control / Form Generation Issues
Table B-13 describes symptoms, possible causes, and possible solutions for Oracle JDeveloper data control/form generation issues.
Table B-13 Troubleshooting Oracle JDeveloper Data Control / Form Generation Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
|
Created an initiator task based on an XSD element and tried to autogenerate the task form. |
This file is not required except for the BPM initiator task. If you encounter it, create an <?xml version='1.0' encoding='UTF-8'?><jsp:root xmlns:jsp="http://java.sun.com/JSP/ Page" version="2.1"xmlns:f="http://java.sun.com/jsf/core" xmlns:h="http://java.sun.com/jsf/html" xmlns:af="http://xmlns.oracle.com/a df/faces/rich"> <jsp:directive.page contentType="text/html;charset=UTF-8"/> <f:view> <af:document id="d1"> <af:form id="f1"></af:form> </af:document> </f:view> </jsp:root> |
Human Workflow Service/ System MBean Browser Issues
Table B-14 describes symptoms, possible causes, and possible solutions for human workflow service/System MBean Browser issues.
Table B-14 Troubleshooting Human Workflow Service/ System MBean Browser Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
Setting commonly used human workflow configuration parameters. |
N/A |
Use the Workflow Task Service and Workflow Notification pages of Oracle Enterprise Manager Fusion Middleware Control:
For more information, see Configuring Human Workflow Notification Properties and Configuring Human Workflow Task Service Properties. |
Setting human workflow configuration parameters not available in the Oracle Enterprise Manager Fusion Middleware Control properties pages. |
N/A |
Use the System MBean Browser in Oracle Enterprise Manager Fusion Middleware Control:
|
The System MBean Browser does not reflect my changes after editing the human workflow configuration MBeans. |
The System MBean Browser is showing a previously cached version of beans. |
Click the refresh cached tree data button in the System MBean Browser. |
Human workflow services are not locating resource bundles or classes located at the workflow customizations class path URL. |
The protocol is not specified in the URL, or the URL is missing a trailing forward slash ( |
Ensure that the configured URL is formatted correctly, and specifies a protocol. If the class path points to a directory (rather than a JAR file), it is important that the URL has a trailing forward slash character. For example: file:///home/wstallar/wfcustomizati ons/ |
Manually setting the URL used for displaying task details for a particular task component. |
N/A |
Use the Administration page in Oracle Enterprise Manager Fusion Middleware Control for the human task service component. See Managing the URI of the Human Workflow Service Component Task Details Application for instructions. You can edit or delete existing task display URL entries, and add new entries. For task display URLs used from Oracle BPM Worklist, the application name must be set to worklist. |
AMX Extension Issues
Table B-15 describes symptoms, possible causes, and possible solutions for AMX extension issues.
Table B-15 Troubleshooting AMX Extension Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
The dynamic approval group class is not found. |
The class file is not accessible in the Oracle SOA Suite class path. |
To make the dynamic approval group class accessible, the class file must be placed in the following directory: $FMW_HOME/SOA_HOME/soa/modules/ora
cle.soa.ext_11.1.1/classes This directory is part of the SOA class path. The Oracle WebLogic Server must be restarted. |
During design time at runtime, while defining a rule based on the Approval Group list builder, a message keeps appearing indicating that the group does not exist. |
The Approval Group name is not enclosed in quotes (" "). |
Enclose the name in quotes (for example, "Sample Approval Group Name"). |
In a ruleset, many rules defined are applicable for a transaction. It appears that the correct constraints are not getting applied; therefore, the generated approver list is not correct. |
Only one rule from a ruleset must be applicable for a transaction. In case many rules are true, the actions associated with the applicable rule with the highest priority are executed. In case multiple applicable rules have the same priority, the first rule in the list is picked and its actions are executed. |
Check that all rules in the ruleset have priorities defined so that multiple rules with the same priority are not applicable for the same transaction. |
Oracle BPM Worklist/Task Region Issues
Table B-16 describes symptoms, possible causes, and possible solutions for Oracle BPM Worklist/task region issues.
Table B-16 Troubleshooting Oracle BPM Worklist/Task Region Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
You receive the following exception message in the logs: <Warning> <oracle.adf.controller.internal. metadata.MetadataService> <BEA-000000><ADFc: /META-INF/adfc-config.xml: > <Warning> <oracle.adf.controller.internal. metadata.MetadataService><ADFC-5 2024> <ADFc: Duplicate managed bean definition for 'aleCompBindings' detected.> |
The shared library |
These JARs should not be packaged inside the web application. They should only be referenced as a shared library. Do not package these JARs in the web application. |
You receive the following exception message in the logs: Duplicate default server in client configuration. Configuration needs to have only one default server in client configuration. Specify one default server in client configuration. |
Two default servers are specified in the client configuration file or in the JAXB object passed to the task flow. |
Mark only one server as the default in the client configuration file or in the JAXB object passed. |
You receive the following exception message in the logs: The default server is not specified |
The default server is not specified in the client configuration file or in the JAXB object passed to the task flow. |
Ensure that the default server is marked in the client configuration file or the JAXB object. |
You receive the following exception message in the logs: Invalid display column. The display column COLUMN NAME is not a valid Task column. Specify a valid column name. |
The column name passed to the task flow parameter |
Ensure that you pass the correct column name to the task flow parameter. |
You receive the following exception message in the logs: java.lang.IllegalStateException: Attempt to validate an already invalid RegionSite: |
This is a generic exception that sometimes appears in server logs (for example, |
See the real exception in the diagnostic logs (for example, |
You receive the following exception message in the logs: Caused by: oracle.adf.controller.Controller Exception: ADFC-02001: The ADF Controller cannot find '/WEB-INF/taskList-task-flow-def inition.xml' |
The Oracle BPM Worklist JARs are not provided in the class path, either by referring to the shared library |
Ensure either the JARs are referred through the shared library or packaged inside the application. |
Filters for the task list are removed when the task list is refreshed. |
Because an inbox is not a persisted view, filters set on it are removed when rendering the page again or refreshing the task list. |
Instead of setting filters on the task list, create a user view with the required set of filters and pass the |
You have set the |
If you specified the |
You have to use both parameters with the <parameter id="taskTypesFilterList" value="http://xmlns.oracle.com/H elpDeskRequestSOAApp/HelpDeskReq uestComposite/HelpDeskRequestHum anTask,[ http://xmlns.oracle.com/Vacation RequestApp/VacationRequest/Vacat ionRequestTask]"/> <parameter id= "attributesFilterOperator" value="and"/> <parameter id= "attributesFilterList" value="state=ASSIGNED"/> |
You receive the following exception message in the logs: [AdminServer] [NOTIFICATION] [J2EE JSP-00008] [oracle.j2ee.jsp] [tid: [ACTIVE].ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 17011f2a001d6b0e:7e22d6ce:12e344 4eb1b:-8000-0000000000002f0a,0] [APP: FederatedApp_ application1] unable to dispatch JSP page: The following exception occurred:.[[ java.lang.RuntimeException: Cannot find FacesContext at javax.faces.webapp.UIComponentCl assicTagBase.getFacesContext(UIC omponentClassicTagBase.java:2122) |
This is a common mistake that is generic in nature and is not related to any components you are using. You forgot to put http://server:port/FederatedApp/test.jspx |
Put http://server:port/FederatedApp/ faces/test.jspx |
You receive the following exception message in the logs: [AdminServer] [TRACE] [] [] [tid: [ACTIVE].ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 17011f2a001d6b0e:7e22d6ce:12e344 4eb1b:-8000-0000000000001d39,0] [SRC_CLASS: oracle.bpel.services.workflow.cl ient.config.ClientConfigurationU til] [APP: FederatedApp_ application1] [SRC_METHOD: getClientConfiguration] WorkflowServiceClientContext: Cannot find client configuration file: wf_client_ config.xml |
There are three possible causes for this issue:
|
Ensure either the client configuration file |
You receive the following exception message in the logs: <Error> <Deployer> <BEA-149265> <Failure occurred in the execution of deployment request with ID '1297964056778' for task '3'. Error is: 'weblogic.management.DeploymentE xception: [J2EE:160149]Error while processing library references. Unresolved application library references, defined in weblogic-application.xml: [Extension-Name: oracle.soa.workflow.wc, exact-match: false].' weblogic.management.DeploymentEx ception: [J2EE:160149]Error while processing library references. Unresolved application library references, defined in weblogic-application.xml: [Extension-Name: oracle.soa.workflow.wc, exact-match: false]. |
Shared library |
Ensure that this shared library is deployed on the server on which you are deploying your application. The shared library may be deployed, but not targeted, for that server. |
You cannot see the mapped attributes mapped columns. Note: Starting with Release 11g R1 (11.1.1.4), Oracle BPM Worklist flex fields are now known as mapped attributes. |
The right set of parameters is not being passed to the task list task flow. |
The correct set of parameters to be passed is as follows: <parameter id= "displayColumnsList" value="assignees,creator, assignedDate,state, textAttribute1,textAttribute2"/>
You must specifically pass the fully qualified value to parameter For example: <parameter id="taskTypesFilterList" value="http://xmlns.oracle.com/Hel pDeskRequestSOAApp/HelpDeskRequest Composite/HelpDeskRequestHumanTask "/> |
Test-to-Production Issues
Table B-17 through Table B-20 describe symptoms, possible causes, and possible solutions for test-to-production issues.
Table B-17 Troubleshooting Test-to-Production Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
Finding the default realm name for a SOA server. |
N/A |
The
This retrieves the default realm name. Here is a sample answer from an invocation: <env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap /envelope/"> <env:Header/> <env:Body> <realmName xmlns="http://xmlns.oracle.com/bpel/servi ces/IdentityService">jazn.com</realmName> </env:Body> </env:Envelope> |
Table B-18 Troubleshooting Test-to-Production Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
The following user authentication error (in this example, the user is [java] Error in workflow service Web service operation invocation. The error is ORA-30501:Error in authenticating user. [java] Error in authenticating and creating a workflow context for user jazn.com/FMW_USERID. [java] Verify that the user credentials and identity service configurations are correct. |
This occurs if the given user is not seeded and available in the LDAP provider. To identify if the user is seeded properly, try to log in to Oracle BPM Worklist from a browser as this user. If the user can log in to Oracle BPM Worklist, that means the user is seeded. |
If Oracle Internet Directory or another LDAP provider is used, ensure the configuration of the LDAP provider is completed correctly. Otherwise, you cannot get past this error. |
While importing task payload mapped attribute mappings (previously known as flex fields) into the target SOA server, you may encounter the following error in the console logs: [java] Caused by: java.sql.SQLIntegrityConstrain tViolationException: ORA-02291: integrity constraint (UAT_ SOAINFRA.SYS_C0018364) violated - parent key not found* * |
The importing of task payload mapped attribute mappings into the target SOA server is a two-step process. Even before the import of task payload mapped attribute mappings into the target SOA server operation is attempted, there is a prerequisite step that must be performed. This is the import of attribute labels into the target SOA server operation. |
To be successful, perform the following operations (in the correct order) with the human workflow test-to-production migration tool:
See Moving Human Workflow Data from a Test to a Production Environment for more details. |
Table B-19 Troubleshooting Test-to-Production Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
Assume you encounter the following error during rule migration: [java] Error encountered during migration. [java] Exception in thread "main"[java] UserConfigDataMigrationException:[java] faultString:Invalid parameters for RULE. [java] Invalid user and group: both parameters can not have null values.[java] To migrate User Rules, provide the 'user' parameter only. [java] To migrate Group Rules, provide the 'group' parameter only.[java] oracle.bpel.services.workflow.util. tools.wfUserConfigDataMigrator. UserConfigDataMigrationE xception [java] at oracle.bpel.services.workflow. util.tools.wfUserConfigDataMigrator. implhwfMigrator.parseParametersNode |
There are two properties in the
During any rule migration (whether export or import) operation, at most one of them ( |
Set values for at most one of them. To perform user rule migration, set the |
Assume you encounter the following error during rule migration: [java] Error encountered during migration.[java] Exception in thread "main"[java] UserConfigDataMigrationExcepti on: [java] faultString: Invalid parameters for RULE. [java] Invalid user and group: both parameters can not have values. [java] To migrate User Rules, provide the 'user'parameter only. [java] To migrate Group Rules, provide the 'group' parameter only. [java] oracle.bpel.services. workflow.util.tools.wfUserConfigData Migrator.UserConfigDataMigrationE xception |
This is similar to the previous explanation in this table. During any rule migration (whether export or import) operation, both |
Provide values for at most one of them. |
Table B-20 Troubleshooting Test-to-Production Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
Testing the health of the installed server. |
N/A |
Before performing a test-to-production migration, it is useful to test the health of the server. From a browser, you can test some SOAP services. The following list provides a subset of human workflow services and Oracle BPM Worklist: http://host:port/integration/worklistapp/ http://host:port/integration/services /TaskQueryService/TaskQueryService http://host:port/integration/services/IdentityService/configuration http://host:port/integration/services/IdentityService/identity http://host:port/integration/services/RuntimeConfigService/RuntimeConfigService You can randomly test some operations in these services, and verify that the operation yields results. Similarly, you can log in as a user to Oracle BPM Worklist and see if everything is fine. |
For more information about test-to-production issues, see Moving Human Workflow Data from a Test to a Production Environment.
Identity Service Issues
Table B-21 and Table B-22 describe symptoms, possible causes, and possible solutions for identity service issues.
Table B-21 Troubleshooting Identity Service Issues
Symptoms | Possible Cause | Possible Solution |
---|---|---|
In a clustered environment, you can receive the following error when you attempt to act on an approval task in Oracle BPM Worklist: Token Decryption Failed |
You are using a file-based repository, instead of an LDAP-based repository. This causes the application roles to not be synchronized across the nodes. |
For repository configuration information in a clustered environment, you must follow the instructions in the Enterprise Deployment Guide for Oracle SOA Suite. |
Only a subset of users in LDAP can log in to Oracle BPM Worklist. |
The user base DN is not configured properly. |
Mention the user base under which all the groups are seeded. This can be performed in two ways: Add the base DN under which all the required groups are seeded. For instance, if users are seeded under: UserDN 1 : cn=users1,dc=us,dc=oracle,dc=com UserDN 2 : cn=users2,dc=us,dc=oracle,dc=com UserDN 3 : cn=users3,dc=us,dc=oracle,dc=com Then mention the group base DN as follows: dc=us,dc=oracle,dc=com This is the common DN. If only some user DNs are required (for example, <serviceInstance name="idstore.ldap" provider="idstore.ldap.provider"> <property name="idstore.config.provider" value="oracle.security.jps.wls.internal.idstore.WlsLd apIdStoreConfigProvider"/> <property name="CONNECTION_POOL_CLASS" value="oracle.security.idm.providers.stdldap.JNDIPool"/> <extendedProperty> <name>user.search.bases</name> <values> <value>cn=users1,dc=us,dc=oracle,dc=com</value> <value>cn=users2,dc=us,dc=oracle,dc=com</value> </values> </extendedProperty> </serviceInstance> |
Users and groups seeded only in the first authenticator are visible, but not from the other authenticators. |
By default, users and groups from the first authenticator are authorized. |
Starting with Release 11.1.1.4, you can authorize users and groups from multiple authenticators. For more information, see Configuring Multiple Authentication Providers. |
Table B-22 Troubleshooting Identity Service Issues
Symptom | Possible Cause | Possible Solution |
---|---|---|
The following exception appears when Exception seen : Service" Unknown macro: {0} "in configuration" Unknown macro: {1} " could not be initialized. Error in initializing service "Authentication" in configuration "myrealm". |
The human workflow identity service uses the identity context that is set in the |
To change the realm name, the
|
After configuring LDAP with Oracle WebLogic Server, the users are visible in the Oracle WebLogic Server Administration Console, but the following error is thrown: No Role found matching the criteria |
The group's base DN is not configured properly. Either the group that is being looked up is not present in LDAP or it may be seeded outside the group base DN that is mentioned while configuring LDAP. |
Mention the group base under which all the groups are seeded. This can be performed in two ways. Add the base DN under which all the required groups are seeded. For instance, if groups are seeded under: GroupDN 1 : cn=groups1,dc=us,dc=oracle,dc=com GroupDN 2 : cn=groups2,dc=us,dc=oracle,dc=com GroupDN 3 : cn=groups3,dc=us,dc=oracle,dc=com Then mention the group base DN as follows: dc=us,dc=oracle,dc=com This is the common DN. If only some group DNs are required (for example, <serviceInstance name="idstore.ldap" provider="idstore.ldap.provider"> <property name="idstore.config.provider" value="oracle.security.jps.wls.internal.idstore.WlsL dapIdStoreConfigProvider"/> <property name="CONNECTION_POOL_CLASS" value="oracle.security.idm.providers.stdldap.JNDIPool"/> extendedProperty> <name>group.search.bases</name> <values> <value>cn=groups1,dc=us,dc=oracle,dc=com</value> <value>cn=groups2,dc=us,dc=oracle,dc=com</value> </values> </extendedProperty> </serviceInstance> |
Business Events and Event Delivery Network Troubleshooting
This section describes how to troubleshoot business event and event delivery network (EDN) issues.
Increasing the JMS Adapter Connection Pool Size
EDN uses the connection factories shown in Table B-23 for the JMS adapter to communicate with the underlying JMS topic for event publishing and subscription.
Table B-23 Connection Factories for the JMS Adapter
JMS Type | JMS Connection Factories for the JMS Type |
---|---|
Oracle WebLogic Server JMS |
|
AQ JMS |
|
In environments in which there are a large number of concurrent event publishers, a large number of event subscribers, or both, the default connection pool size (200
) for the JMS Adapter can be exceeded. This can cause ResourceLimitException
errors similar to that shown below:
The JCA Binding Component was unable to establish an outbound JCA CCI connection due to the following issue: javax.resource.spi.ApplicationServerInternalException: Unable to get a connection for pool = "eis/wls/EDNxaDurableTopic", weblogic.common.resourcepool.ResourceLimitException: Configured maximum limit of (0) on number of threads allowed to wait for a resource reached for pool eis/wls/EDNxaDurableTopic Please make sure that the JCA connection factory and any dependent connection factories have been configured with a sufficient limit for max connections. Please also make sure that the physical connection to the backend EIS is available and the backend itself is accepting connections.
When this occurs, you must increase the connection pool Max Capacity value of the corresponding JMS connection factory for the JMS adapter in Oracle WebLogic Server Administration Console. Table B-24 identifies the use of each JMS connection factory for the JMS adapter.
Table B-24 Uses of Each JMS Connection Factory for the JMS Adapter
JMS Connection Factory | Used for Outbound Event Publishing? | Used for Inbound Event Subscription? |
---|---|---|
|
Yes, for event publishing with a global transaction. |
Yes, for durable subscriptions with One And Only One (OAOO) consistency. |
|
No. |
Yes, for nondurable subscriptions with OAOO consistency. |
|
Yes, for event publishing with a local transaction. |
Yes, for durable subscriptions with guaranteed consistency. |
|
No. |
Yes, for nondurable subscriptions with guaranteed consistency. |
|
Yes, for event publishing with global transaction. |
Yes, for durable subscriptions with OAOO consistency. |
|
No. |
Yes, for nondurable subscriptions with OAOO consistency. |
|
Yes, for event publishing with local transaction. |
Yes, for durable subscriptions with guaranteed consistency. |
|
No. |
Yes, for nondurable subscriptions with guaranteed consistency. |
When calculating connections used, you must combine the number of events publishing with the event subscriptions if the corresponding JMS connection factory is used for both event publishing and event subscription.
For example, if there are 200
concurrent BPEL process event publishers that publish events in a global transaction, and the default SOAInternalProcessing_maxThreads
is set to 150
, then these publishers use all 150
threads, which takes 150
connections from eis/wls/EDNxaDurableTopic
. There are also a total of 12
SOA event subscribers with ThreadsPerSubscriber
set to 5
. These subscribers use 12
x 5
= 60
threads, which takes 60
connections from eis/wls/EDNxaDurableTopic
.
Because 150
+ 60
= 210
> 200
(the default setting), you receive a ResourceLimitException
error.
In this case, you may need to increase the size of the Max Capacity field of the connection pool of eis/wls/EDNxaDurableTopic
to a larger value (for example, 300
).
To increase the connection pool size:
Tuning Recommendations For Publishing Many BPEL Process Events with Oracle WebLogic Server JMS
When publishing many BPEL process events (for example, one million) with Oracle WebLogic Server JMS, you can receive numerous recoverable errors (for example, 20% of the one million events) if the EDN is not properly tuned. For example, assume you have the following use case scenario:
-
200 threads
-
5000 iterations
After running for some time (for example, 40 minutes), you can receive the error shown below:
<Jan 8, 2014 10:08:33 AM PST> <Error> <oracle.soa.adapter.jms.outbound> <BEA-000000> <JMSMessageProducer_produce: [default destination = jms/fabric/EDNTopic]:weblogic.messaging.kernel.QuotaException: Quota blocking time exceeded and no quota available> <Jan 8, 2014 10:08:33 AM PST> <Error> <oracle.soa.adapter.jms.outbound> <BEA-000000> <JMSMessageProducer_produce: [default destination = jms/fabric/EDNTopic]:weblogic.messaging.kernel.QuotaException: Quota blocking time exceeded and no quota available>
In addition, subscriptions are slow (there are many database, Coherence cache, and transaction-related operations for the EDN threads) and the accumulated messages slow down publishing.
If this occurs, tune the following areas:
-
The EDN quota is automatically defined on the default EDN topic on Oracle WebLogic Server JMS. This EDN quota has a maximum message limit of 50,000. You may need to define a quota with a different maximum message limit based on your use case scenario. For more information, see Tuning WebLogic JMS in Tuning Performance of Oracle WebLogic Server.
-
Tune the send timeout value on the EDN connection factories based on your use case scenario. The following EDN JMS connection factories have a default value of 10,000 (ms) as the send timeout value:
-
EDNConnectionFactory
-
xaEDNConnectionFactory
For more information, see Blocking Senders During Quota Conditions in I.
-
-
When there are heavy loads, messages accumulate in the JMS topic. The EDN client receives a
QuoteException
when the number of messages exceeds the quota limit. If the EDN client is a composite with a BPEL process service component, the BPEL process persists the messages into the Error Hospital. There may be a big delay in delivery to the subscribers. The messages in the Error Hospital can be recovered manually from the Error Hospital or automatically from the Recovery tab of the BPEL process service engine. For more information, see Recovering From Faults in the Error Hospital and Performing BPEL Process Service Engine Message Recovery.
Tuning EDN Event Bus and Delivery
You can customize the following EDN properties in the System MBean Browser:
Events Are Consumed by Multiple Revisions of the Same Composites
By design, different composites can subscribe to the same event. However, it is an error if multiple revisions of the same composite subscribe to the same event. This occurs when you have multiple active versions of the composite that may be a result of a patch failure.
Perform the following steps to determine whether there are multiple, active revisions of the same composite and to retire the composite revision that should not be active:
Business Event Is Picked Up Twice (Or More) By SOA Server
Business events may be raised from J2EE applications and picked up more than once. Expected behavior is that they must be picked up only once.
If a patch fails, it may be possible for two versions of a given composite to be active (that is, the older version has not been retired). When multiple versions of a composite are active, they all become subscribers and the event is picked up more than once. This should not happen under normal scenarios. Reapply the patch and ensure that the deployed composite only has one active version.
Some Messages Are Lost Between EDN and Composites or Composites Across Clusters
Messages primarily may get lost for the following reasons:
-
The EDN message was not delivered.
-
The target asynchronous service did not respond.
-
The message was roll backed (though it was not lost; you still see it in Oracle Enterprise Manager Fusion Middleware Control).
This issue typically requires you to identify where the messages are supposed to be and to diagnose the path.
-
Check if the Oracle Web Services Manager Policy Manager security configurations allow the client to invoke the SOA composite.
-
Verify that the transaction is not transient (nonpersistent).
Oracle BPEL Process Manager uses the dehydration store database to maintain long-running, asynchronous processes and their current state information in a database while they wait for asynchronous callbacks. Storing the process in a database preserves the process and prevents any loss of state or reliability if a system shuts down or a network problem occurs. There are two types of processes in Oracle BPEL Process Manager. These processes impact the dehydration store database in different ways.
-
Transient processes: This process type does not incur any intermediate dehydration points during process execution. If there are unhandled faults or there is system downtime during process execution, the instances of a transient process do not leave a trace in the system. Instances of transient processes cannot be saved in-flight (whether they complete normally or abnormally). Transient processes are typically short-lived, request-response style processes. The synchronous process you design in Oracle JDeveloper is an example of a transient process.
-
Durable processes: This process type incurs one or more dehydration points in the database during execution because of the following activities:
-
Receive activity
-
OnMessage branch of a pick activity
-
OnAlarm branch of a pick activity
-
Wait activity
Instances of durable processes can be saved in-flight (whether they complete normally or abnormally). These processes are typically long-living and initiated through a one-way invocation. Because of out-of-memory and system downtime issues, durable processes cannot be memory-optimized.
-
-
-
If a business flow instance is not visible in Oracle Enterprise Manager Fusion Middleware Control and the SOA Infrastructure is running, check that the Audit Level is not set to Off on the SOA Infrastructure Common Properties page. This can be checked in Oracle Enterprise Manager Fusion Middleware Control:
-
Right-click soa-infra (SOA_cluster_name).
-
Select SOA Administration > Common Properties.
-
-
If the business flow instance is available, check the oneWayDeliveryPolicy BPEL property value. You can check the value in Oracle Enterprise Manager Fusion Middleware Control:
-
In the navigation pane, expand soa-infra (SOA_cluster_name).
-
Expand the partition, and select the composite.
The Dashboard page for the composite is displayed.
-
In the upper right corner, click the Show XML Definition icon.
The contents of
composite.xml
for that composite are displayed.
If this is set to async.cache, you may lose messages. Set it to async.persist for reliable messages. This is typically specified in the BPEL process service component section of the
composite.xml
file. Therefore, this can be set for custom composites. If the value is not set incomposite.xml
, the value for oneWayDeliveryPolicy in the System MBean Browser in Oracle Enterprise Manager Fusion Middleware Control is used. The following values are possible:-
async.persist: Messages are persisted in the database hash map.
-
async.cache: Messages are stored in memory.
-
sync: Direct invocation occurs on the same thread.
For more information about these settings, see the "Deployment Descriptor Properties" appendix and the "Transaction and Fault Propagation Semantics in BPEL Processes" chapter of Developing SOA Applications with Oracle SOA Suite.
For information about setting the oneWayDeliveryPolicy property in the System MBean Browser, see Configuring BPEL Process Service Engine Properties.
It is also possible that the transaction has invoked a target asynchronous service that has not responded back. In this case, the business flow instance shows the call to the target asynchronous service.
-
-
Check if the instance has rolled back and the message is in recovery.
-
Log in to Oracle Enterprise Manager Fusion Middleware Control.
-
Right-click soa-infra (SOA_cluster_name), and select Service Engines > BPEL.
-
Click the Recovery tab.
This may occur if any external references receiving the message are not reachable (for example, an external web service, enterprise applications such as Siebel, and so on).
-
Performance Troubleshooting
This section describes how to troubleshoot performance issues.
Resolving Issue Where BPEL Compiler Does Not Terminate or Report Exceptions
If a large SOA composite application containing many BPEL process service components is compiled, the BPEL compiler may not terminate and report any exceptions. If this occurs, increase the JVM heap size in the ant-sca-compile.xml
file as follows:
-
Change
<jvmarg value="-Xmx1024m"/>
-
To
<jvmarg value="-Xmx4g"/>
The code sample below provides details:
<scac input="${scac.input}" outXml="${scac.output}"
error="${scac.error}" appHome="${scac.application.home}"
failonerror="true" displayLevel="${scac.displayLevel}"
reportErrors="${scac.reportSchemaValidationErrors}">
<jvmarg
value="-Djavax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom=oracle.
xml.xpath.JXPathFactory"/>
<jvmarg value="-Xms128m"/>
<jvmarg value="-Xmx4g"/>
<jvmarg value="-XX:PermSize=32m"/>
<jvmarg value="-XX:MaxPermSize=256m"/>
<jvmarg value="-Doracle.xdkjava.compatibility.version=11.1.1"/>
<jvmarg value="-Doracle.soa.compatibility.version=11.1.1"/>
<!-- jvmarg
value="-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005"/ -->
</scac>
</target>
Resolving Message Failure Caused by Too Many Open Files
You can receive the following error at runtime or compilation time, depending on the number of JAR files being used, the use of file descriptors by JDK 6/JRE, or both.
Message send failed: Too many open files
To resolve this error, increase the number of file descriptors to at least 4096
.
Resolving MaxMessageSizeExceededException Errors Caused By Large Payloads
MaxMessageSizeExceededException
error occurs when you provide a large payload (for example, 200vu*500iter which is 100,000 requests) to your deployed SOA composite application. MaxMessageSize configuration must be configured based on the load. Click View XML Document in the audit trail to view the payload, and check for MaxMessageSizeExceededException
errors. To resolve the error either increase the MaxMessageSize to 80000000, or decrease the session-flush-threashold to 50.
Note:
This values depend on the load. The default value of MaxMessageSize is 10000000. Change the values according to you load.Increasing MaxMessageSize
To increase the MaxMessageSize:
-
Login to admin console.
-
Navigate to servers, protocols, general, max message size.
-
Changed the value to 80000000. (both the admin and the soa).
-
Navigate to servers, configuration, server start, arguments, and set-Dweblogic.MaxMessageSize = 80000000.
weblogic.MaxMessageSize
property. To change the value by adding the property:
-
Open the following file:
-
On UNIX operating systems, open
$MIDDLEWARE_HOME/user_projects
/domains/
domain_name
/bin/setDomainEnv.sh
. -
On Window operating systems, open
MIDDLEWARE_HOME
\user_projects\domains\
domain_name
\bin\setDomainEnv.bat
.
-
-
Add the
weblogic.MaxMessageSize
property with the following value:EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Dweblogic.MaxMessageSize=80000000" export EXTRA_JAVA_PROPERTIES
-
Restart the server.
Decreasing session-flush-threashold
Navigate to console cluster, soa_cluster, Configuration, Replication, Advanced, and change the Session Flush Threshold value to 50 (smaller value).
Extending Tablespaces to Avoid Problems at Runtime
If the database tablespace is not extended, runtime processing can be impacted. Messages are not processed or persisted, and exception errors similar to that shown in Example 1 and Example 2 below can appear in the log files. This is because Oracle BPEL Process Manager relies on the database to store instance data. If the database is not available, runtime processing is impacted.
Example 1
INFO: MediatorServiceEngine returning after processing the request for operation = processResponse [EL Warning]: 2009.01.14 11:46:16.783--UnitOfWork(32372128)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.1 (Build SNAPSHOT-20081007)): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.BatchUpdateException: ORA-01691: unable to extend lob segment SH_SOAINFRA.SYS_LOB0000145067C00007$$ by 1024 in tablespace SH_SOAINFRA Error Code: 1691 Query: InsertObjectQuery(com.collaxa.cube.persistence.dto.AuditTrail@199b33d) [EL Warning]: 2009.01.14 11:46:16.782--UnitOfWork(32372128)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.1 (Build SNAPSHOT-20081007)): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.BatchUpdateException: ORA-01691: unable to extend lob segment SH_SOAINFRA.SYS_LOB0000145067C00007$$ by 1024 in tablespace SH_SOAINFRA . . . . . .
Example 2
<bpelFault><faultType>0</faultType><subLanguageExecutionFault xmlns="http://schemas.oracle.com/bpel/extension"><part name="summary"><summary>XPath expression failed to execute. An error occurs while processing the XPath expression; the expression is ora:readBinaryFromFile('/scratch/vbasaval/dropbox/LFile/PO_5Gig.zip'). The XPath expression failed to execute; the reason was: java.sql.SQLException: ORA-01652: unable to extend temp segment by 128 in tablespace DEV12_IAS_TEMP
Ensure that you set a tablespace to automatically extend itself by a specified amount when it reaches its size limit. If you do not enable autoextend, ensure that you respond when alerted that the tablespace is reaching its critical or warning threshold size. You can respond to size alerts by manually increasing the tablespace size.
To manually add data files in SQL*Plus:
SQL> alter tablespace SOA12_SOAINFRA add datafile '/scratch/db12101/Oracle/oradata/SOA12_soainfra2.dbf' size 1000M autoextend on next 30M maxsize unlimited; ALTER DATABASE TEMPFILE '/scratch/db12101/Oracle/oradata/SOA12_iastemp.dbf' RESIZE 100M; ALTER DATABASE TEMPFILE '/scratch/db12101/Oracle/oradata/SOA12_iastemp.dbf' AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_mds.dbf' RESIZE 100M; ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_mds.dbf' AUTOEXTEND ON NEXT 50M MAXSIZE UNLIMITED; ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_soainfra.dbf' RESIZE 2048M; ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_soainfra.dbf' AUTOEXTEND ON NEXT 50M MAXSIZE UNLIMITED; ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_svctbl.dbf' RESIZE 100M; ALTER DATABASE DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_svctbl.dbf' AUTOEXTEND ON NEXT 50M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA1.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA2.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA3.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA4.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA5.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA6.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA7.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA8.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRA9.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAA.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAB.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAC.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAD.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAE.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAF.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAG.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAH.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAI.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAJ.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED; ALTER TABLESPACE "C0123_SOAINFRA" ADD DATAFILE '/scratch/db12101/Oracle/oradata/SOA12_SOAINFRAK.dbf' SIZE 100M AUTOEXTEND ON NEXT 30M MAXSIZE UNLIMITED;
Resolving Database Growth Issues Caused by a High Volume of Transactions
If Oracle SOA Suite transactions fail and the logs indicate that the database is running out of space, the dehydration store or MDS store may be running out of space due to a high volume of transactions. In the latter example, you see errors such as the following:
java.sql.SQLException: ORA-01653: unable to extend table SH_MDS.CUBE_INSTANCE by 16 in tablespace FUSION_TS_TOOLS
This indicates that the tablespace is full and the database cannot extend it.
Perform the following tasks.
For more details, see Resolving Message Failure Caused by Too Many Open Files.
Observing Slow Application Performance Such as Longer Time to Serve Pages or Finish Transactions
You can observe slow application performance and/or memory trashing. For example, it may take longer to load and serve pages or to complete composite transactions. Response time may seem slower compared to normal behavior.
There are various reasons for slow performance. It may be due to a large number of servers running on the same host, or there may be a large number of records/sessions/locking in the database. Thread contention can also be a reason for poor performance.
Note:
All configuration changes in Oracle Enterprise Manager Fusion Middleware Control may be unavailable due to this problem.
To diagnosis this issue:
-
Check the CPU utilization to see if it is saturated due to a heavy load or too many processes in relation to CPU capacity.
If CPU utilization is at 100% during normal load hours (the target should be 70-80%), you have no capacity to handle a peak load and the hardware resources are insufficient. Add scale-out servers to handle the additional load.
-
Check applications using Oracle Enterprise Manager Fusion Middleware Control to report on performance. Check the performance of services and the invoke/response times in the BPEL process audit trail.
-
To maximize performance, it is recommended that you not set the logging level higher than the default INFO level.
For debugging purposes, you must set the logging level to the FINEST level. However, once issues are resolved, reset the logging level to the default level for best performance. It is also recommended that you set Audit Level to Production on the SOA Infrastructure Common Properties page. This can be set in Oracle Enterprise Manager Fusion Middleware Control as follows:
-
Right-click soa-infra (SOA_cluster_name).
-
Select SOA Administration > Common Properties.
-
Set Audit Level to Production.
-
-
Purge periodically based on retention requirements to maintain any service level agreements (SLAs).
To identify the tables where data growth may lead to performance issues, see Section "Identifying Tables Impacted By Instance Data Growth" of Tuning Performance.
For information about using the purge scripts, see Managing Database Growth .
-
Check the memory/IO/paging/swapping/CPU usage load statistics using Top or Glance or another monitoring tool.
-
Optimize the JVM to avoid full garbage collection or out-of-memory errors.
Frequent garbage collection can be either due to higher memory usage or memory leaks.
-
Ensure that the sum of the maximum heap size of all the JVMs running on your system does not exceed the amount of available physical RAM to avoid operating system level paging activity.
-
Use the JRockit mission control memory profiling tools to get thread dumps and memory snapshots, which helps Oracle Support Services debug any code issues. For more information about thread dumps, see Diagnosing Problems with SOA Composite Applications.
-
-
Optimize threads to avoid contention. Get a thread dump to investigate and submit it to Oracle Support Services.
-
Check the JVM and thread dumps for methods invoking the database to identify if database performance is a bottleneck.
-
Run database AWR snapshots to identify causes of database performance issues.
-
Ensure that database statistics are updated at regular intervals and other tunable parameters for memory, tablespace, and partitions are used effectively to obtain maximum performance.
Here are some common tuning recommendations. For more information, see the "Tuning Database Parameters" section of Tuning Performance.
-
Put indexes and tables in as physically separate disk areas, if possible.
-
Never put rollback segments with data or index segments.
-
Separate highly active tables and indexes into their own tablespaces.
-
Partition high activity tables and indexes to help balance disk I/O and prevent hot disks.
-
Have processes in place to generate database table statistics at regular intervals.
-
-
Tune database tables to control the high watermark (HWM) contention of large objects. Tune database advanced queues (AQ) to control HWM contention and ensure consistent performance of producing and consuming messages from AQ.
The EDN AQ names to be aware of are
EDN_EVENT_QUEUE_TABLE
andEDN_OAOO_DELIVERY_TABLE
. -
Tune the BPEL process properties to reduce overhead (for example, disable the ValidateXML and StatsLastN (statistics gathering batch size) properties on the BPEL Service Engine Properties page in Oracle Enterprise Manager Fusion Middleware Control), if they are not needed.
-
In case of integration with packaged applications (for example, Siebel), check if the issue lies with the legacy applications.
For more information about performance tuning the various components, see the "Top Performance Areas" chapter and the "SOA Suite Components" part in Tuning Performance.
For information about setting properties on the BPEL Service Engine Properties page, see Configuring BPEL Process Service Components and Engines.
Observing Incoming Message Rates Exceeding Outgoing Message Rates
When the incoming rate of messages exceeds the processing rate of messages (that is, the outgoing rate), you can receive the warning message shown below:
[2012-04-09T16:49:40.342-04:00] [Soa_server1] [WARNING] [] [oracle.soa.bpel.engine.dispatch] [tid: weblogic.work.j2ee.J2EEWorkManager$WorkWithListener@e3d0e3d ] [userId: <anonymous>] [ecid: 0000JSnNKVTBh4Z5ln1FiZ1Fdnmd001gWL,0] [APP: soa-infra] BPEL service engine is overloaded, the inflow is higher than the outflow, there are "100000" invoke messages pending in the dispatcher in-memory cache, the configured threshold is "102",the new incoming invoke messages will not be dispatched or processed immediately, they are going to be persisted in the dehydration store and can be recovered later using recovery.
To resolve this issue, either increase the processing rate or lower (that is, throttle down) the incoming rate. The relatively low processing rate may be due to any or all of the following issues:
-
Not enough invoke threads
-
Not enough JVM capacity
-
The external partners that the BPEL process is calling cannot scale, which causes the messages to back up in the BPEL process.
To identify the exact causes and resolve them, perform the following steps:
- Check the
oneWayDeliveryPolicy
property value in thecomposite.xml
file. Is it set toasync.persist
,async.cache
, orsync
? For information about this property, see Deployment Descriptor Properties in I. - Ensure that the outgoing invocations from the BPEL process to external services are not backed up due to low scalability of the external services. If the bottleneck is in an external service, any tuning you perform in Oracle SOA Suite (that is, Step 3) does not help. You must first resolve the bottleneck.
- Monitor your JVM garbage collection (GC). If the GC frequency appears higher than normal and the JVM appears to be near capacity, increase the JVM heap size.
Server Troubleshooting
This section describes how to troubleshoot server issues.
For more information about server startup issues, see Stopping and Starting the Managed Server and SOA Infrastructure.
Best Practices for Starting and Stopping a Managed Server
As a best practice, it is always recommended that you start and stop a managed server through one, but not both, of the following methods. Do not mix these methods, such as starting the managed server from the command line and stopping it from Oracle Enterprise Manager Fusion Middleware Control, or vice versa.
-
Oracle Enterprise Manager Fusion Middleware Control
With this method, the node manager must be up and running. The node manager tracks all managed server startups and shutdowns performed from Oracle Enterprise Manager Fusion Middleware Control. With this method, the server state is not an issue.
-
Command line
With this method, the node manager does not track the server state. Therefore, if you start the server from the command line and shut it down from Oracle Enterprise Manager Fusion Middleware Control, the Oracle WebLogic Administration Server accesses the node manager to determine its status, which returns a state of unknown.
Perform the following steps to stop and start the server from Oracle Enterprise Manager Fusion Middleware Control.
- Expand the WebLogic domain.
- Select the managed server (for example, named soa_server1).
- Select Control > Shut Down.
- Select Control > Start Up.
For information on starting and stopping managed servers from the command line, see Installing and Configuring Oracle SOA Suite and Business Process Management.
Diagnosing SOA Server Startup Problems
If the SOA server (soa_infra
) does not start, there can be various reasons for server startup issues. A discovery-based approach to finding the root cause is required. Check the server and diagnostic logs as a first attempt to diagnose the issue. In addition, check the following.
To diagnose server startup issues:
-
Check if the database is not available or there are not enough connections available. Some failures with Oracle SOA Suite runtime can result from database outage/connectivity issues. Perform the following steps:
-
Log in to Oracle WebLogic Server Administration Console.
-
In the Domain Structure, view the status by selecting Services > Data Sources > SOADatasource > Monitoring > Testing to test the data source. You can also check this from a SQL prompt.
-
-
Check the list of ports used for port conflicts.
-
Check if the MDS repository cannot load shared documents (incorrect MDS configuration or the database that holds the MDS schema is not reachable).
-
Check the Oracle Coherence configuration if using an Oracle SOA Suite cluster (use of unicast versus multicast).
This issue only applies after provisioning is complete. Clustering of SOA servers may fail if there are port conflicts (used by Oracle Coherence). For example, one SOA server may be picking up the deployment, but the other server is not. In some cases, this may only present the following error:
[soa_server1] [ERROR] [] [Coherence] [tid: Logger@352821903 3.6.0.4] [ecid: 46f620208907e045:63f295ec:12dd091ec2e:-8000-0000000000000003,1:27187] [APP: soa-infra] 2011-01-28 23:06:19.463/414.816 Oracle Coherence GE 3.6.0.4 <Error> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster[[...
In this example, Oracle Coherence timeouts are prominently available. However, sometimes it presents itself with an unrelated error, such as the following:
Error creating bean with name 'SensorManager' defined in ServletContext resource
The root cause of this is still primarily related to Oracle Coherence configuration.
-
Check if both the administration server and managed server ports are open and accessible.
-
Check if managed server startup failed because the administration server is not reachable.
-
Check network issues (for example, IP routing filtering/rules that may be causing issues).
-
Check Oracle WebLogic Server LDAP security corruption.
The managed server may report that policies for the application System MBeans Browser already exist or do not exist.
Specifying the Proxy Server
To use system properties to specify the proxy server, write your client application in the standard way, and then specify Java system properties when you execute the client application. The code sample below provides details:
setenv PROXY_SETTINGS "-DproxySet=true -Dhttp.proxyHost=www-myproxy.us.mycompany.com -Dhttp.proxyPort=80 -Dhttp.nonProxyHosts=localhost|*.us.mycompany.com |0:0:0:0:0:0:0:1|fe80:0:0:0:250:56ff:fe31"
Note:
When you specify values for proxy properties such as http.proxyHost
and http.proxyPort
, also specify the http.nonProxyHosts
property.
Flow Diagram Does Not Display The First Time on Some Lower End Hosts
The flow diagram for an instance ID of a deployed SOA composite application in Oracle Enterprise Manager Fusion Middleware Control may not display the first time on some lower end hosts. Instead, you receive a failed
to
load
resource
message.
As a workaround, close the flow trace page and click the instance ID to return to the flow trace page.
Accessing Oracle Enterprise Manager Fusion Middleware Control on Dual Stack Hosts that Support IPv4 and IPv6
If you run Oracle SOA Suite on a dual stack host that supports both IPv4 and IPv6, you must update the etc/hosts
file as shown in Table B-25 for IPv4 clients to access IPv6 URLs in Oracle Enterprise Manager Fusion Middleware Control.
Table B-25 IPv4 and IPv6 Settings in etc/hosts File
On The... | Edit the etc/hosts File as Follows.... |
---|---|
On the IPv4 client: |
xx.xxx.xxx.xxx myhost10-ipv6 where |
On the IPv6 client |
2001:0db8:db0a::0:1 myhost10-ipv6 myhost10-ipv6.us.example.com Note: Replace |
Browser Troubleshooting
This section describes how to troubleshoot browser issues.
Limitation on Using the Safari Browser to View WSDL File Content
If you are using the Safari browser, note the following limitation and workaround for viewing WSDL file contents in Oracle Enterprise Manager Fusion Middleware Control. Note also that Mozilla Firefox works correctly and does not require this workaround.
Additional Troubleshooting Documentation
Table B-26 describes documentation to see for additional troubleshooting issues.
Table B-26 Additional Troubleshooting Documentation
For Information About... | See... |
---|---|
SOA Infrastructure startup and shutdown issues |
Stopping and Starting the Managed Server and SOA Infrastructure |
Using WebLogic Diagnostic Framework (WLDF) and Diagnostics Framework (DFW) to diagnosis Oracle SOA Suite problems |
|
Managing large documents and metadata and for managing environments with large numbers of instances |
Managing Large Documents and Large Numbers of Instances in Developing SOA Applications with Oracle SOA Suite |
Deployment and compilation errors |
Testing and Troubleshooting in Developing SOA Applications with Oracle SOA Suite |
Oracle Web Services Manager Policy Manager errors |
Diagnosing Problems with Oracle Web Services Manager in Securing Web Services and Managing Policies with Oracle Web Services Manager |
Oracle Business Rules |
Oracle Business Rules Troubleshooting in Designing Business Rules with Oracle Business Process Management |
Unable to Get Pool Connection
In scenarios where there are a large number of concurrent event publishers and a large number of event subscribers, (for example, PSR stress tests), ResourceLimitException with following messages or similar can occur.
The JCA Binding Component was unable to establish an outbound JCA CCI connection due to the following issue:
javax.resource.spi.ApplicationServerInternalException:
Unable to get a connection for pool = "eis/wls/EDNDurableTopic",
weblogic.common.resourcepool.ResourceLimitException:
Configured maximum limit of (0) on number of threads allowed to wait for a resource reached for pool eis/wls/EDNDurableTopic
Please make sure that the JCA connection factory and any dependent connection factories have been configured with a sufficient limit for max connections.
Please also make sure that the physical connection to the backend EIS is available and the backend itself is accepting connections.
Causes
EDN 12c uses following JMS adapter conneciton factories to communicate with underlying JMS topic for both event publish and/or event subscription.
JMS Type | JMS Connection Factories for JMS Adapter |
---|---|
WLJMS |
|
AQJMS |
|
When there are many publishers and subscribers, the default Connection Pool Size of JmsAdapter, which is 200, may be all used up.
Solutions
Increase the "Max Capacity" of Connection Pool of corresponding JMS connection factory for JMS adapter. The following table indicates the uses of each of the JMS connection factories for JMS adapter:
JMS Connection Factory | Used for Outbound Event Publish? | Used for Inbound Event Subscription? |
---|---|---|
eis/wls/EDNxaDurableTopic |
No |
Yes, for durable subscription with 'OAOO' consistency |
eis/wls/EDNxaTopic |
Yes, for event publish with global transaction |
Yes, for non-durable subscription with 'OAOO' consistency |
eis/wls/EDNLocalTxDurableTopic |
No |
Yes, for durable subscription with 'Guaranteed' consistency |
eis/wls/EDNLocalTxTopic |
Yes, for event publish with local transaction |
Yes, for non-durable subscription with 'Guaranteed' consistency |
eis/aqjms/EDNxaDurableTopic |
No |
Yes, for durable subscription with 'OAOO' consistency |
eis/aqjms/EDNxaTopic |
Yes, for event publish with global transaction |
Yes, for non-durable subscription with 'OAOO' consistency |
eis/aqjms/EDNLocalTxDurableTopic |
No |
Yes, for durable subscription with 'Guaranteed' consistency |
eis/aqjms/EDNLocalTxTopic |
Yes, for event publish with local transaction |
Yes, for non-durable subscription with 'Guaranteed' consistency |
When calculating JMS connections used, you need to add the number of event publishing to that of event subscription if the corresponding JMS connection factory is used for both event publishing and event subscription.
For example, if there are 200 concurrent BPEL event publishers that publish events in global transaction, and the default SOAInternalProcessing_maxThreads is 150 (SOAMaxThreadsConfig = 300 and InternalProcessingPercentage = 50%), then these publishers would use all 150 threads, which takes up 150 JMS connections from "eis/wls/EDNDurableTopic".
Suppose there are also a total of 8 SOA event subscribers with "durable" option and subscription consistency being "Guaranteed". The ThreadsPerSubscriber is also set to 10. Then these subscribers could use up to 8 * 10 = 80 consumer threads. Since the actual consumer threads is limited by the EDN WorkManager's max-threads-constraint, which is SOAIncoming_maxThreads which is 60 by default, the consumer threads would take away 60 connections from "eis/wls/EDNDurableTopic". Since 150 + 60 = 210 > 200 (200 is the default adapter connection pool size of "eis/wls/EDNDurableTopic"), user would get ResourceLimitException. In this case, user needs to increase the "Max Capacity" of Connection Pool of "eis/wls/EDNDurableTopic" to a larger value, for example, 300.
Note:
When a resource is accessed by multiple requests in a cluster environment, resource busy message logs likeorg.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLException: ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired
may be logged. These logs are warning messages and require no action if they do not affect the functionality.
Handling ConnectException and PeerGoneException
When a large number of task approvals occur, ConnectException
and PeerGoneException
exceptions can result, as follows:
<Sep 14, 2018 3:18:48,987 PM PDT> <Error> <HTTP Session> <BEA-100099> <An
error occurred while looking up the session with
ID:eEzaKdYeD4_xq3bUnOM8JZ5HIO74dk46xwD44S6XCl8boV34phXI!2139240963!-38703078
from primary server:soa_server3 and secondary server:soa_server2.
java.rmi.ConnectException: This RJVM has already been shutdown
-3651827968604015139S:10.232.32.47:[8002,8002,-1,-1,-1,-1,-1]:base_domain:soa_
server2
at weblogic.rjvm.RJVMImpl.connectionCheck(RJVMImpl.java:843)
at weblogic.rjvm.RJVMImpl.getOutputStreamByURL(RJVMImpl.java:464)
at weblogic.rjvm.RJVMImpl.getMsgAbbrevOutputStream(RJVMImpl.java:802)
at weblogic.rjvm.RJVMImpl.getRequestStream(RJVMImpl.java:793)
at weblogic.rjvm.RJVMImpl.getOutboundRequest(RJVMImpl.java:1052)
Truncated. see log file for complete stacktrace
>
<Sep 14, 2018 3:18:48,988 PM PDT> <Error> <HTTP Session> <BEA-100099> <An
error occurred while looking up the session with
ID:yCLaKdE3f_kC4szmeLBTg07sA_gsMXYQy4Llgd7uZFQesVP8Mnkj!2139240963!-38703078
from primary server:soa_server3 and secondary server:soa_server2.
weblogic.rjvm.PeerGoneException: ; nested exception is:
weblogic.socket.MaxMessageSizeExceededException: Incoming message of
size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes
for protocol: 't3'
at weblogic.rjvm.RJVMImpl.gotExceptionReceiving(RJVMImpl.java:1230)
at
weblogic.rjvm.ConnectionManager.gotExceptionReceiving(ConnectionManager.java:1
206)
at
weblogic.rjvm.MsgAbbrevJVMConnection.gotExceptionReceiving(MsgAbbrevJVMConnect
ion.java:582)
at
weblogic.rjvm.t3.MuxableSocketT3.hasException(MuxableSocketT3.java:663)
at
weblogic.socket.SocketMuxer.deliverExceptionAndCleanup(SocketMuxer.java:847)
Truncated. see log file for complete stacktrace
Caused By: weblogic.socket.MaxMessageSizeExceededException: Incoming message
of size: '10000080' bytes exceeds the configured maximum of: '10000000' bytes
for protocol: 't3'
at
weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMu
xableSocket.java:246)
at
weblogic.socket.BaseAbstractMuxableSocket.incrementBufferOffset(BaseAbstractMu
xableSocket.java:207)
at
weblogic.rjvm.t3.MuxableSocketT3.incrementBufferOffset(MuxableSocketT3.java:67
9)
at weblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:1010)
at
weblogic.socket.NIOSocketMuxer.readFromSocket(NIOSocketMuxer.java:782)
Truncated. see log file for complete stacktrace
>
Solutions
- Change
idle-periods-until-timeout
of all SOA servers to10
:Log in to the Admin Console, navigate to server name, Configuration, Tuning, Advanced, Idle Periods Until Timeout, and change it to 10 from 4.
- Change
Session Flush Threshold
of the cluster to10
:Log in to the WebLogic Console, navigate to soa_cluster, Configuration, Replication, Advanced Settings, and change Session Flush Threshold to 10 from 10000.