40 Processing Large Documents

This chapter describes best practices and limitations for processing large documents in Oracle SOA Suite.

This chapter includes the following sections:

40.1 Introduction to Processing Large Documents

This document provides the best practices for processing large XML document payloads in Oracle SOA Suite service engines. Limitations on using large payloads are also described.

Oracle recommends that you follow these best practices before developing and executing large payloads.

40.2 Best Practices for Handling Large Documents

This section describes the following scenarios of handling large documents and the best practice approach for each scenario:

40.2.1 Setting Audit Levels from Oracle Enterprise Manager for Large Payload Processing

For large payload processing, turn off audit level logging for the specific composite. You can set the settings/composite audit level option to Off or Production in Oracle Enterprise Manager Fusion Middleware Control Console. If you set the settings/composite audit level option to Development, then it serializes the entire large payload into an in-memory string and can lead to an out-of-memory error.

For more information about setting audit levels, see Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite.

40.2.2 Using the Assign Activity in BPEL/Mediator

When using the assign activity, in BPEL or Mediator, to manipulate large payloads, do not assign the complete message. Instead, assign only the part of the payload that you need.

In addition, when using the assign activity in BPEL, Oracle recommends using local variables instead of process variables wherever possible. Local variables are limited to the scope of the BPEL process. These get deleted from memory and from the database after you close the scope. However, the life cycle of a global variable is tied with the instance life cycle. These variables stay in memory or disk until the instance completes. Thus, local variables are preferred to process or global variables.

40.2.3 Using XSLT Transformations for Repeating Structures

In scenarios where the repeating structure is of smaller payloads compared to the overall payload size, Oracle recommends using XSLT transformation because the current XSLT implementation materializes the entire DOM in memory. For example, PurchaseOrder.LineItem.Supplier (a subpart of a large payload).

You can also substitute it with the assign activity, as it performs a shadow copy. Although a shadow copy does not materialize DOM, it creates a shadow node to point to the source document.

You can also use the following optimized translation functions while performing transforms/translations of large payloads:

  • ora:doTranslateFromNative

  • ora:doTranslateToNative

  • ora:doStreamingTranslate

For more information about the usage of these functions, see Oracle Fusion Middleware User's Guide for Technology Adapters.

40.2.4 Using Adapter Support for Streaming Large Payloads

The streaming feature exposed by the File/FTP adapters is used for processing large payloads. Only the Files/FTP adapters support large payload processing for both inbound and outbound processing. The other adapters do not have explicit support for both.

Note:

Only the Files/FTP adapters and web services binding support streaming. You should consider alternative strategies for handling large documents with other binding.

For more information about how the streaming feature is used for large payloads, see Section 4.5.4, "Oracle File Adapter Scalable DOM" in the Oracle Fusion Middleware User's Guide for Technology Adapters.

40.2.5 Using Correct Settings for Large Payload Scenarios

Uncomment the following line in setDomainEnv.sh for JAVA_OPTIONS, and then restart the server. Without this setting, the large payload scenarios fails with ResourceDisabledException for the dehydration data source.

-Dweblogic.resourcepool.max_test_wait_secs=30

40.2.6 Processing Large Documents in Oracle B2B

For processing large documents in Oracle B2B, various parameters such as MDSInstance Cache Size, Protocol Message Size, Number of threads, StuckThread Max Time, and Tablespace must be tuned. The following sections describe the parameters you must set for processing large documents in Oracle B2B:

40.2.6.1 MDSInstance Cache Size

To set MDSInstance cache size, the property and value must be added in the $DOMAIN_HOME/config/soa-infra/configuration/b2b-config.xml file as mentioned in the example below:

<property>
 <name>b2b.mdsCache</name>
 <value>200000</value>
 <comment>MDS Instance cache size </comment>
</property>

40.2.6.2 Protocol Message Size

If Oracle B2B wants to send or receive more than 10 MB of message or import/export configuration is more than 10 MB, then the following settings must be changed accordingly in the Oracle WebLogic Server Administration Console:

Environment > servers > soa_server > protocols > General > change Maximum Message Size

This setting can be added/modified in the $DOMAIN_HOME/config/config.xml file next to the server name configuration, as shown below:

<name>soa_server1</name>
<max-message-size>150000000</max-message-size>

Note:

By default, max-message-size is not available in this config.xml file.

40.2.6.3 Number of Threads

This parameter helps to improve the message processing capability of Oracle B2B and must be set in the $DOMAIN_HOME/config/soa-infra/configuration/b2b-config.xml file.

<property>
  <name>b2b.inboundProcess.threadCount</name>
  <value>5</value>
  <comment></comment>
</property>
<property>
  <name>b2b.inboundProcess.sleepTime</name>
  <value>10</value>
  <comment></comment>
</property>
<property>
  <name>b2b.outboundProcess.threadCount</name>
  <value>5</value>
  <comment></comment>
</property>
<property>
  <name>b2b.outboundProcess.sleepTime</name>
  <value>10</value>
  <comment></comment>
</property>
<property>
  <name>b2b.defaultProcess.threadCount</name>
  <value>5</value>
  <comment></comment>
</property>
<property>
  <name>b2b.defaultProcess.sleepTime</name>
  <value>10</value>
  <comment></comment>
</property>

40.2.6.4 StuckThread Max Time

The StuckThread Max Time parameter checks the number of seconds that a thread must be continually working before this server considers the thread stuck. You must change the following settings in the Oracle WebLogic Server Administration Console:

Environment > servers > soa_server > Configuration > Tuning > change Stuck Thread Max Time

40.2.6.5 Tablespace

If you must store more than a 150 MB configuration in the data file, then you must extend or add the data file to increase the tablespace size, as follows:

ALTER TABLESPACE sh_mds add DATAFILE 'sh_mds01.DBF' SIZE 100M autoextend on next
 10M maxsize unlimited;
ALTER TABLESPACE sh_ias_temp add TEMPFILE 'sh_ias_temp01.DBF' SIZE 100M autoextend
 on next 10M maxsize unlimited;

40.2.7 Setting the Default JTA Timeout in for Large Documents

Oracle recommends that the default JTA Timeout parameter be increased from the current 30 to an appropriate value for processing large documents.

40.2.8 Using Large Number of Activities in BPEL Processes (Without FlowN)

To deploy BPEL processes that have a large number of activities, for example, 50000, the following settings are required:

Set MEM_ARGS: -Xms512m -Xmx1024m -XX:PermSize = 128m -XX:MaxPermSize = 256m

Number of Concurrent Threads = 20

Number of Loops = 5 Delay = 100 ms

The above settings enable you to deploy and execute BPEL processes, which use only while loops without the flowN activities, successfully.

40.2.9 Using Large Number of Activities in BPEL Processes (With FlowN)

To deploy BPEL processes that have large number of activities, for example, 50000, the following settings are required:

Set USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=256m

Number of Concurrent Threads= 10

Number of Loops=5 Delay=100 ms

Set the StatsLastN property to -1 in the System MBean Browser of Oracle Enterprise Manager Fusion Middleware Control Console.

The above settings enable you to deploy and execute BPEL processes, which use the flowN activities, successfully.

For more information, see Chapter 9, "Using Parallel Flow in a BPEL Process."

40.2.10 Boundary on the Processing of Large Numbers of Activities in a BPEL Process

There is a limit to the number of activities that can be executed in a BPEL process. When you exceed this limit, system memory fills up, which can cause timeouts to occur. For example, with the following parameters, two fault instances occur due to a timeout:

  • 100 threads

  • 1 second of think time

  • 1000 incoming request messages

Try to keep the number of incoming request messages at a proper level to ensure system memory stability.

40.2.11 Using a Flow With Multiple Sequences

BPEL processes, which have large numbers of activities up to 7000, can be deployed and executed successfully with the following settings:

Set USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=256m

Note:

If you deploy BPEL processes with more than 8000 activities, then BPEL compilation throws errors.

40.2.12 Using a Flow with One Sequence

BPEL processes, which have large number of activities up to 7000, can be deployed and executed successfully with the following settings:

Set USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=512m

Note:

If you deploy BPEL processes with more than 10,000 activities, then the BPEL compilation fails.

40.2.13 Using Flow with No Sequence

You can deploy and execute BPEL processes, which have large number of activities, for example, up to 5000, successfully.

There is a probability that the BPEL compilation could fail for 6000 activities.

40.2.14 Large Numbers of Mediators in Composites

Oracle recommends that you not have more than 50 mediators in a single composite. The JTA Transaction timeout should be increased to an appropriate high value based on the environment.

40.2.15 Using XSLT Transformations on Large Payloads (For BPEL and Mediator)

Oracle recommends that you not apply the XSLT Transformation on large payloads as this would result in out-of-memory errors when XSLT must traverse the entire document.

40.3 Limitations on Concurrent Processing of Large Documents

This section describes the limitations on concurrent processing of large documents. This section includes the following topics:

40.3.1 Opaque Schema for Processing Large Payloads

There is a limitation when you use an opaque schema for processing large payloads. The entire data for the opaque translator is converted to a single Base64-encoded string. An opaque schema is generally used for smaller data. For large data, use the attachments feature instead of the opaque translator.

For more information about the usage of these functions, see Oracle Fusion Middleware User's Guide for Technology Adapters.

40.3.2 Streaming MTOM Attachments

The incoming requests for streaming MTOM attachments that are passed through the Service Infrastructure are normalized, and the processing of such messages are not optimized inside the Service Infrastructure layer.

40.3.3 Importing Large Data Sets in Oracle B2B

Oracle recommends that you not use browsers for large data set imports and to use the command-line utility. The following utility commands are recommended for large data configuration:

  • purge - This command is used to purge the entire repository.

  • import - This command is used to import the specified ZIP file.

  • deploy - This command is used to deploy an agreement with whatever name is specified. If no name is specified, then all the agreements are deployed.

However, the purgeimportdeploy option is not recommended to be used for transferring or deploying Oracle B2B configuration.