This chapter describes best practices and limitations for processing large documents in Oracle SOA Suite.
This chapter includes the following sections:
This document provides the best practices for processing large XML document payloads in Oracle SOA Suite service engines. Limitations on using large payloads are also described.
Oracle recommends that you follow these best practices before developing and executing large payloads.
This section describes the following scenarios of handling large documents and the best practice approach for each scenario:
Section 53.2.1, "Setting Audit Levels from Oracle Enterprise Manager for Large Payload Processing"
Section 53.2.2, "Using the Assign Activity in BPEL/Mediator"
Section 53.2.3, "Using XSLT Transformations for Repeating Structures"
Section 53.2.4, "Using Adapter Support for Streaming Large Payloads"
Section 53.2.5, "Using Correct Settings for Large Payload Scenarios"
Section 53.2.7, "Setting the Default JTA Timeout in for Large Documents"
Section 53.2.8, "Using Large Number of Activities in BPEL Processes (Without FlowN)"
Section 53.2.9, "Using Large Number of Activities in BPEL Processes (With FlowN)"
Section 53.2.14, "Using XSLT Transformations on Large Payloads (For BPEL and Mediator)"
For large payload processing, turn off audit level logging for the specific composite. You can set the settings/composite audit level option to Off or Production in Oracle Enterprise Manager Fusion Middleware Control Console. If you set the settings/composite audit level option to Development, then it serializes the entire large payload into an in-memory string and can lead to an out-of-memory error.
For more information about setting audit levels, see Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite.
When using the assign activity, in BPEL or Mediator, to manipulate large payloads, do not assign the complete message. Instead, assign only the part of the payload that you need.
In addition, when using the assign activity in BPEL, Oracle recommends using local variables instead of process variables wherever possible. Local variables are limited to the scope of the BPEL process. These get deleted from memory and from the database after you close the scope. On the other hand, the life cycle of a global variable is tied with the instance life cycle. These variables stay in memory or disk until the instance completes. Thus, local variables are preferred to process or global variables.
In scenarios where the repeating structure is of smaller payloads compared to the overall payload size, Oracle recommends using XSLT transformation because the current XSLT implementation materializes the entire DOM in memory. For example, PurchaseOrder.LineItem.Supplier (a subpart of a large payload).
You can also substitute it with the assign activity, as it performs a shadow copy. Although a shadow copy does not materialize DOM, it creates a shadow node to point to the source document.
You can also use the following optimized translation functions while performing transforms/translations of large payloads:
ora:doTranslateFromNative
ora:doTranslateToNative
ora:doStreamingTranslate
For more information about the usage of these functions, see Oracle Fusion Middleware User's Guide for Technology Adapters.
The streaming feature exposed by the File/FTP adapters is used for processing large payloads. Only the Files/FTP adapters support large payload processing for both inbound and outbound processing. The other adapters do not have explicit support for both.
Note:
Only the Files/FTP adapters and web services binding support streaming. You should consider alternative strategies for handling large documents with other binding.For more information about how the streaming feature is used for large payloads, see Section 4.5.4, "Oracle File Adapter Scalable DOM" in the Oracle Fusion Middleware User's Guide for Technology Adapters.
Add -Dweblogic.resourcepool.max_test_wait_secs=30 in setDomainEnv.sh for JAVA_OPTIONS, and then restart the server. Without this setting, the large payload scenarios fails with ResourceDisabledException for the dehydration data source.
For processing large documents in Oracle B2B, various parameters such as MDSInstance Cache Size, Protocol Message Size, Number of threads, StuckThread Max Time, and Tablespace must be tuned. The following sections describe the parameters you must set for processing large documents in Oracle B2B:
To set MDSInstance cache size, the property and value must be added in the $DOMAIN_HOME/config/soa-infra/configuration/b2b-config.xml file as mentioned in the example below:
<property> <name>b2b.mdsCache</name> <value>200000</value> <comment>MDS Instance cache size </comment> </property>
If Oracle B2B wants to send or receive more than 10 MB of message or import/export configuration is more than 10 MB, then the following settings must be changed accordingly in the Oracle WebLogic Server Administration Console:
Environment > servers > soa_server > protocols > General > change Maximum Message Size
This setting can be added/modified in the $DOMAIN_HOME/config/config.xml file next to the server name configuration, as shown below:
<name>soa_server1</name> <max-message-size>150000000</max-message-size>
Note:
By default,max-message-size is not available in this config.xml file.This parameter helps to improve the message processing capability of Oracle B2B and must be set in the $DOMAIN_HOME/config/soa-infra/configuration/b2b-config.xml file.
<property> <name>b2b.inboundProcess.threadCount</name> <value>5</value> <comment></comment> </property> <property> <name>b2b.inboundProcess.sleepTime</name> <value>10</value> <comment></comment> </property> <property> <name>b2b.outboundProcess.threadCount</name> <value>5</value> <comment></comment> </property> <property> <name>b2b.outboundProcess.sleepTime</name> <value>10</value> <comment></comment> </property> <property> <name>b2b.defaultProcess.threadCount</name> <value>5</value> <comment></comment> </property> <property> <name>b2b.defaultProcess.sleepTime</name> <value>10</value> <comment></comment> </property>
The StuckThread Max Time parameter checks the number of seconds that a thread must be continually working before this server considers the thread stuck. You must change the following settings in the Oracle WebLogic Server Administration Console:
Environment > servers > soa_server > Configuration > Tuning > change Stuck Thread Max Time
If you must store more than a 150 MB configuration in the data file, then you must extend or add the data file to increase the tablespace size, as follows:
ALTER TABLESPACE sh_mds add DATAFILE 'sh_mds01.DBF' SIZE 100M autoextend on next 10M maxsize unlimited; ALTER TABLESPACE sh_ias_temp add TEMPFILE 'sh_ias_temp01.DBF' SIZE 100M autoextend on next 10M maxsize unlimited;
Oracle recommends that the default JTA Timeout parameter be increased from the current 30 to an appropriate value for processing large documents.
To deploy BPEL processes that have a large number of activities, for example, 50000, the following settings are required:
Set MEM_ARGS: -Xms512m -Xmx1024m -XX:PermSize = 128m -XX:MaxPermSize = 256m
Number of Concurrent Threads = 20
Number of Loops = 5 Delay = 100 ms
The above settings enable you to deploy and execute BPEL processes, which use only while loops without the flowN activities, successfully.
To deploy BPEL processes that have large number of activities, for example, 50000, the following settings are required:
Set USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=256m
Number of Concurrent Threads= 10
Number of Loops=5 Delay=100 ms
Set <statsLastN>-1</statsLastN> in the bpel-config.xml file.
The above settings enable you to deploy and execute BPEL processes, which use the flowN activities, successfully.
For more information, see Chapter 10, "Using Parallel Flow in a BPEL Process."
BPEL processes, which have large numbers of activities up to 7000, can be deployed and executed successfully with the following settings:
Set USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=256m
Note:
If you deploy BPEL processes with more than 8000 activities, then BPEL compilation throws errors.BPEL processes, which have large number of activities up to 7000, can be deployed and executed successfully with the following settings:
Set USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=512m
Note:
If you deploy BPEL processes with more than 10,000 activities, then the BPEL compilation fails.You can deploy and execute BPEL processes, which have large number of activities, for example, up to 5000, successfully.
There is a probability that the BPEL compilation could fail for 6000 activities.
Oracle recommends that you not have more than 50 mediators in a single composite. The JTA Transaction timeout should be increased to an appropriate high value based on the environment.
This section describes the limitations on concurrent processing of large documents. This section includes the following topics:
There is a limitation when you use an opaque schema for processing large payloads. The entire data for the opaque translator is converted to a single Base64-encoded string. An opaque schema is generally used for smaller data. For large data, use the attachments feature instead of the opaque translator.
For more information about the usage of these functions, see Oracle Fusion Middleware User's Guide for Technology Adapters.
The incoming requests for streaming MTOM attachments that are passed through the Service Infrastructure are normalized, and the processing of such messages are not optimized inside the Service Infrastructure layer.
Oracle recommends that you not use browsers for large data set imports and to use the command-line utility. The following utility commands are recommended for large data configuration:
purge - This command is used to purge the entire repository.
import - This command is used to import the specified ZIP file.
deploy - This command is used to deploy an agreement with whatever name is specified. If no name is specified, then all the agreements are deployed.
However, the purgeimportdeploy option is not recommended to be used for transferring or deploying Oracle B2B configuration.