This chapter describes the best practices for managing large documents and metadata and for managing environments with large numbers of instances in Oracle SOA Suite. It also describes use cases for handling large documents, limitations on concurrent processing of large documents, and tuning recommendations.
This chapter includes the following sections:
For more information about Oracle SOA Suite tuning and performance, see Oracle Fusion Middleware Performance and Tuning Guide.
For information about troubleshooting Oracle SOA Suite issues, see Chapter "Troubleshooting Oracle SOA Suite and Oracle BPM Suite" of Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
For information about using Oracle Data Integrator to perform fast bulk data movement and handle complex data transformations, visit the following URL:
http://www.oracle.com/technetwork/middleware/data-integrator
This section describes the following scenarios for handling large documents and the best practice approach for each scenario. Oracle recommends that you follow these best practices before developing and executing large payloads.
This section describes use cases for handling large documents.
This section describes use cases for passing binary objects as Base64-encoded text in the XML payload.
In this use case, the binary attachments (for example, an image) are Base64-encoded as text and then passed inline in the XML document. Table 45-1 provides details.
| Capability | Description |
|---|---|
|
Security |
Supported. |
|
Filter/Transformation/Assign |
Use of transformations may lead to slower performance, out-of-memory errors, or both. |
|
Fanout |
Supported. |
|
Binding |
WS binding sends it as a document object model (DOM). |
|
Oracle BPEL Process Manager/Oracle Mediator |
Can be decoded in a BPEL process using Java |
In this use case, the binary attachments (for example, an image) are Base64-encoded as text and then passed as a Message Transmission Optimization Mechanism (MTOM) document. Table 45-2 provides details.
| Capability | Description |
|---|---|
|
Security |
Supported. |
|
Filter/Transformation/Assign |
Assign activities are supported. |
|
Fanout |
Supported. |
|
Binding |
WS binding materializes the attachment sent as MTOM and puts it inside in Base64-encoded format (streaming is not supported). |
|
Oracle BPEL Process Manager/Oracle Mediator |
No additional work is required. |
In this use case, the binary attachments (for example, an image) are Base64-encoded as text and then passed inline in the XML document. Table 45-3 provides details.
| Capability | Description |
|---|---|
|
Security |
Not supported. |
|
Filter/Transformation/Assign |
Pass through. |
|
Fanout |
Supported. |
|
Binding |
Adapter encodes it to Base64 format. |
|
Oracle BPEL Process Manager/Oracle Mediator |
Supported. Opaque content cannot be manipulated in an assign or a transformation activity. |
In this use case, the binary attachments (for example, an image) are Base64-encoded as text encoded. Table 45-4 provides details.
| Capability | Description |
|---|---|
|
Security |
Not supported. |
|
Filter/Transformation/Assign |
Pass through. |
|
Fanout |
Supported. |
|
Oracle B2B |
Oracle B2B encodes the native payload to Base64 format. For this scenario, you must configure the Oracle B2B binding document definition handling to be opaque. |
This section describes use cases for end-to-end streaming of attachments.
Note:
Direct Internet Message Encapsulation (DIME) attachments are not supported in Oracle SOA Suite 11g.
In this use case, the binary attachments (for instance, an image) are passed end-to-end as a stream. Table 45-5 provides details.
| Capability | Description |
|---|---|
|
Security |
Not supported. |
|
Filter/Transformation/Assign |
Pass through. You must use an XPath extension function in Oracle BPEL Process Manager. |
|
Binding |
WS binding creates stream iterators for the SOAP attachment. |
|
Oracle BPEL Process Manager/Oracle Mediator |
Oracle Mediator can perform a pass through without materializing it. Oracle BPEL Process Manager persists it. |
|
Tuning |
Manage the database tablespace when using with Oracle BPEL Process Manager. |
|
WSDL code for defining SOAP with attachments |
<mime:part> <mime:content part="bin" type=“image/jpeg"/> </mime:part> |
Notes:
You cannot stream attachments as part of a web service callback response.
The spring service component does not support processing MIME attachments. Only MTOM attachments are supported.
You can use various binding components such as direct binding, web services, and so on to process large attachments. However, processing large attachments with direct binding is not recommended and results in out-of-memory errors.
Working with Streaming Attachments
Oracle Fusion Middleware web services enable you to pass large attachments as a stream. Unlike the JAX-RPC API, which treats attachments as if they are entirely in memory, streams make the programming model more efficient to use. Streams also enhance performance and scalability because there is no need to load the attachment into memory before service execution.
As with embedded attachments, streamed attachments conform to the multipart MIME binary format. Embedded attachments refer to inlined/encoded attachments.
On the wire, messages with streamed attachments are identical to any other SOAP message with attachments.
Example 45-1 provides a sample message with a streamed attachment. The first part in the message is the SOAP envelope (<SOAP-ENV:Envelope...). The second part is the attachment (for this example, myImage.gif).
Example 45-1 Sample Message with a Streamed Attachment
MIME-Version: 1.0 Content-Type: Multipart/Related; boundary=MIME_boundary; type=text/xml; Content-Description: This is the optional message description. --MIME_boundary Content-Type: text/xml; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-ID: NotSure/DoesntMatter <?xml version='1.0' ?> <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"> <SOAP-ENV:Body> . . . <DocumentName>MyImage.gif</DocumentName> . . . </SOAP-ENV:Body> </SOAP-ENV:Envelope> --MIME_boundary Content-Type: image/gif Content-Transfer-Encoding: binary Content-ID: AnythingYoudLike ...binary GIF image... --MIME_boundary--
Creating Composites that Use MIME Attachments
Perform the following procedures to create composites that use MIME attachments.
To create composites that use MIME attachments:
Create a composite using a payload schema (for example, an inbound web service wired to an Oracle Mediator wired to an outbound web service).
Within the WSDL file of Oracle Mediator, perform the following steps:
From the WSDL designer, open the Oracle Mediator WSDL file.
Drag and drop bindings into the middle swimlane.
Select the RPC binding.
Enter a name.
Go to Source view of the WSDL and modify the WSDL input and WSDL output with MIME multiparts.
<wsdl:input>
<mime:multipartRelated>
<mime:part>
<soap:body parts="payload" use="literal"/>
</mime:part>
<mime:part>
<mime:content part="bin" type="application/octet-stream"/>
</mime:part>
</mime:multipartRelated>
</wsdl:input>
Add the MIME part in the request/response message.
<wsdl:message name="BPELProcess1RequestMessage">
<wsdl:part name="payload" element="ns1:purchaseOrder" />
<!--add below part-->
<wsdl:part name="bin" type="xsd:base64Binary"/>
</wsdl:message>
Add a namespace in the WSDL definitions.
xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/">
When complete, the WSDL that references a MIME attachment is displayed.
<wsdl:definitions name="PhotoCatalogService" targetNamespace="http://examples.com/PhotoCatalog" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:types="http://examples.com/PhotoCatalog/types" xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/" xmlns:tns="http://examples.com/PhotoCatalog"> <wsdl:message name="addPhotoRequest"> <wsdl:part name="photo" type="xsd:hexBinary"/> </wsdl:message> <wsdl:message name="addPhotoResponse"> <wsdl:part name="status" type="xsd:string"/> </wsdl:message> <wsdl:message name="replacePhotoRequest"> <wsdl:part name="oldPhoto" type="xsd:string"/> <wsdl:part name="newPhoto" type="xsd:hexBinary"/> </wsdl:message> <wsdl:message name="replacePhotoResponse"> <wsdl:part name="status" type="xsd:string"/> </wsdl:message> <wsdl:portType name="PhotoCatalog"> <wsdl:operation name="addPhoto"> <wsdl:input message="tns:addPhotoRequest"/> <wsdl:output message="tns:addPhotoResponse"/> </wsdl:operation> <wsdl:operation name="replacePhoto"> <wsdl:input message="tns:replacePhotoRequest"/> <wsdl:output message="tns:replacePhotoResponse"/> </wsdl:operation> </wsdl:portType> <wsdl:binding name="PhotoCatalogBinding" type="tns:PhotoCatalog"> <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/> <wsdl:operation name="addPhoto"> <wsdl:input> <mime:multipartRelated> <mime:part> <soap:body use="literal"/> </mime:part> <mime:part> <mime:content part="photo" type="image/jpeg"/> </mime:part> </mime:multipartRelated> </wsdl:input> <wsdl:output> <mime:multipartRelated> <mime:part> <soap:body use="literal"/> </mime:part> <mime:part> <mime:content part="status" type="text/plain"/> <mime:content part="status" type="text/xml"/> </mime:part> </mime:multipartRelated> </wsdl:output> </wsdl:operation> <wsdl:operation name="replacePhoto"> <wsdl:input> <mime:multipartRelated> <mime:part> <soap:body parts="oldPhoto" use="literal"/> </mime:part> <mime:part> <mime:content part="newPhoto" type="image/jpeg"/> </mime:part> </mime:multipartRelated> </wsdl:input> <wsdl:output> <soap:body parts="status" use="literal"/> </wsdl:output> </wsdl:operation> </wsdl:binding> </wsdl:definitions>
Performance Overhead and Pass Through Attachments
Because Oracle Mediator is stateless, there is no performance overhead with pass through attachments. However, Oracle BPEL Process Manager dehydrates attachments and has performance overhead, even for pass through attachments. Using Oracle BPEL Process Manager for attachments over a period, the SOA Infrastructure schema can grow to its maximum size and encounter memory issues. It is recommended that you extend the database tablespace appropriately for the SOA Infrastructure schema to accommodate large attachments. Simultaneously, you can use purge scripts to purge completed instances along with the attachments table. For information on purge scripts, see Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
In scenarios in which one BPEL process calls a second BPEL process within the same composite, the second BPEL process does not dehydrate the same attachment again.
In scenarios in which one BPEL process from composite 1 invokes a second BPEL process from composite 2 and optimization is disabled, composite 1 makes a SOAP call to composite 2. The second BPEL process does dehydrate attachments.
Properties for Streaming Attachments
To stream attachments, add the following properties in the composite.xml file. If optimization is enabled, then a native call is used instead of a SOAP call. Example 45-2 provides details.
Example 45-2 Properties for Streaming Attachments
<binding.ws port="http://services.otn.com#wsdl.endpoint(MIMEService/MIMEService)" xmlns:ns="http://xmlns.oracle.com/sca/1.0" streamIncomingAttachments="true" streamOutgoingAttachments="true"> <!--Add this prop to reference bindings to make a SOAP call. --> <property name="oracle.webservices.local.optimization">false</property> </binding.ws>
For information about the oracle.webservices.local.optimization property, see Section "Policy Attachments and Local Optimization in Composite-to-Composite Invocations" and Section "Configuring Local Optimization" in the Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
Note:
Oracle Web Services Manager (OWSM) does not inspect or enforce policies on streamed attachments. For more information about OWSM, see Oracle Fusion Middleware Security and Administrator's Guide for Web Services.
Reading and Encoding SOAP Attachment Content
The ora:getAttachmentContent function reads SOAP attachment content and encodes that data in Base64 format in a BPEL process by providing the BPEL variable as an argument, which has an href of the SOAP attachment. Example 45-3 shows how to use this function:
Example 45-3 ora:getAttachmentContent Function
<copy>
<from expression="ora:getAttachmentContent('input','bin')"/>
<to variable="initiateTaskInput" part="payload"
query="/taskservice:initiateTask/task:task/task:attachment/task:content"/>
</copy>
Example 45-3 copies the attachment content, which has its href stored in the "input/bin" variable, to the content variable in Base64-encoded format.
Oracle BPEL Process Manager supports sending the attachment stream to multiple receivers. For Oracle BPEL Process Manager to send a stream to multiple receivers, it must read the attachment stream from the database using the readBinaryFromFile XPath function and pass the stream to the appropriate targets.
With the default configuration, Oracle Mediator can pass an attachment stream to only one target receiver, which can be another component or a web service/adapter. The second target cannot receive the attachment. When you define the persistStreamAttachment property for the Oracle Mediator component, Oracle Mediator can pass an attachment stream to multiple target receivers.
Oracle Mediator requires the persistStreamAttachment property for streaming attachments where the source message that contains the attachment is shared by multiple target receivers. Set this property to true in composite.xml to enable the streaming of attachments to multiple targets. Example 45-4 provides details.
Example 45-4 Oracle Mediator Streaming Attachments Property
component name="Mediator1">
<implementation.mediator src="Mediator1.mplan"/>
<property name="persistStreamAttachment">true</property>
</component>
Overriding Pass Through Settings for Attachments in Oracle Mediator
Oracle Mediator automatically propagates attachments to target receivers for Oracle Mediator components that are pass through (that is, they do not contain a transformation or assign rule), and it does not propagate attachments for Oracle Mediator components that are not pass through. The passThroughAttachment property lets you override the pass through settings just for attachments. Setting this property to true copies all attachments to the target receiver implicitly.
Use this property to propagate attachments when the Oracle Mediator component is not a pass through, or use it to block attachments when the Oracle Mediator component is pass through. To implement the pass through attachment override, add the property to the project's composite.xml file in the component element for the Oracle Mediator component. Set the property to true to override an Oracle Mediator component that is not pass through; set it to false to override a pass through component. Example 45-5 provides details.
Example 45-5 Setting Attachments to Pass Through
<component name="Mediator">
<implementation.mediator src="Mediator.mplan"/>
<property name="passThroughAttachment">true</property>
</component>
Sharing Attachments Using Synchronous Flows
When Oracle BPEL Process Manager-based composites share attachments using synchronous flows, it is necessary to use the same end-to-end transaction. This is applicable to composites that are colocated and use local/optimized calls. This can be achieved by setting the property shown in Example 45-6 on all the called BPEL components (callees) in the call chain.
Example 45-6 bpel.config.transaction Property
<property name="bpel.config.transaction" many="false" type="xs:string">required</property>
If such composites do not execute as part of the same transaction context, the attachment data saved by the first BPEL component in the call chain is not visible to the other BPEL components in the call chain. In addition, they incur database locking and timeout exceptions:
"ORA-02049: timeout: distributed transaction waiting for lock"
In this use case, the adapter streams the binary data to a database store and publishes an href to the service engine (Oracle BPEL Process Manager or Oracle Mediator). Table 45-6 provides details.
| Capability | Description |
|---|---|
|
Security |
N/A. |
|
Filter/Transformation/Assign |
Filters and transformations on the attachment are not supported. |
|
Fanout |
Supported. |
|
Binding |
The adapter streams the non-XML to the database as a binary large object (BLOB) and passes the key to the service engines. |
|
Oracle BPEL Process Manager/Oracle Mediator |
Supported. |
|
Tuning |
|
|
Documentation |
See Oracle Fusion Middleware User's Guide for Technology Adapters. |
Writing Attachments Using an Outbound File Adapter
Example 45-7 shows a sample schema that can be used by the file adapter to write attachments to disk.
Example 45-7 Schema for Writing Attachments to Disk
<?xml version="1.0" encoding="windows-1252" ?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns="http://xmlns.oracle.com/attachment"
targetNamespace="http://xmlns.oracle.com/attachment"
elementFormDefault="qualified">
<xsd:element name="attach">
<xsd:complexType>
<xsd:attribute name="href" type="xsd:string"/>
</xsd:complexType>
</xsd:element>
</xsd:schema>
Use Oracle Mediator in the flow to map the attachment part from the source (Oracle Mediator) to the target (file adapter) using an Oracle Mediator assign.
If you use Oracle BPEL Process Manager, the attachment is written to the dehydration store, which slows down the process.
Transforming Attachments with the ora:doStreamingTranslate XPath Function
Use of the ora:doStreamingTranslate XPath function is only recommended while transforming attachments within an Oracle BPEL Process Manager or Oracle Mediator service component. This function expects the attachment location to be a relative path on the server. This function cannot translate incoming attachment streams.
For more information about this function, see Section B.2.7, "doStreamingTranslate."
In this use case, Oracle B2B stores the binary data to a database and publishes an href to the service engine (Oracle BPEL Process Manager or Oracle Mediator) based on an Oracle B2B-defined XSD. Oracle B2B protocols define the attachment. Table 45-7 provides details.
| Capability | Description |
|---|---|
|
Security |
N/A. |
|
Filter/Transformation/Assign |
Filters and transformations on the attachment are not supported. |
|
Fanout |
Supported. |
|
Binding |
Oracle B2B passes it as an |
|
Tuning |
Extend the database tablespace for the Oracle SOA Suite schema. |
Within a SOA composite application, you must attach the Oracle WS-MTOM policy to service and reference binding components to receive and send MTOM (MIME binary) optimized messages. When a service binding component (defined under binding.ws in the composite.xml file) is configured with an Oracle WS-MTOM policy, Oracle SOA Suite's MTOM message handling feature is used. When a reference binding component (also defined under binding.ws in the composite.xml file) is configured with an Oracle MTOM policy, Oracle SOA Suite sends MTOM-optimized messages.
Note the following issues with MTOM attachments:
When attachments are inline and encoded, Oracle recommends that you not use the file adapter to write attachments to a file.
The default mtomThreshold value is 1024 bytes and cannot be modified. If an attachment is less than 1024 bytes, for outbound configurations, Oracle SOA Suite sends it as an inline attachment. If the size is greater than 1024 bytes, then the attachment is sent as an attachment part with an href attribute in the message, and is sent as a WSDL-defined format on the wire. However, if the incoming request (for example, from a different web services provider) has an xop href node for small binary data (that is, size is less than 1024 bytes), Oracle SOA Suite uses the same href attribute in the payload in the flow trace. For example:
<xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include" href="cid:e29caf23dc8045908451fdfaafa26dce" />
If a service binding component of a composite does not include an Oracle WS-MTOM policy reference, this indicates that the service can accept non-MTOM messages. This indicates that the calling composite (the appropriate reference binding) does not have an Oracle WS-MTOM policy reference and can send out non-MTOM messages to that service.
MTOM streaming of attachments is not supported by Oracle SOA Suite.
MTOM attachments are supported only with web service bindings. Other bindings (for example, HTTP bindings) are not supported.
Oracle Mediator pass through scenarios are supported. If Oracle Mediator does not contain any transformation or assign statements, it is known as a pass through Oracle Mediator. The message and attachment received are propagated to the target without modifying the payload and attachment. Likewise, multiple MTOM attachments in the same message can be sent and received by Oracle SOA Suite.
Oracle recommends that you not use both streaming and the MTOM message handling feature for sending and receiving attachments. Use either streaming or the MTOM message handling feature.
Note:
If the input is of type text/xml, there is no significant decrease in file size when sending files in MTOM format.
As a best practice, Oracle recommends that you not use the XSLT Mapper to propagate binary data. Instead, use an assign activity. If you must use a style sheet to propagate binary data, it is recommended that you use the xsl:copy-of instruction (copy-of copies everything, including attributes) or use custom functions to copy attributes from source to target.
MTOM attachments should not be gigabytes in size. Instead, use the SOAP with attachments streaming feature for very large attachments. For more information, see Section 45.1.1.2.1, "SOAP with Attachments."
When a SOA composite application with a BPEL process receives an MTOM-optimized SOAP message, the attachment contents of each of the MTOM-optimized elements (the ones with an <xop href=””>) are stored in the dehydration store. Similarly, when receiving a SOAP message with attachments (SwA) message with one or more attachments, each attachment is stored in the dehydration store. These attachments can then be passed around by reference using an href attribute that identifies them in the database. In fact, all of the text content of these attachment elements is removed and replaced by this href attribute. For MTOM-optimized messages, the same value of the incoming href attribute from the <xop> element is reused. Similarly, for SwA, the href attributes of the attachment elements are reused.
The attachments are stored in the dehydration store when the message is delivered to the BPEL process service engine. (when the incoming message is saved into the DLV_MESSAGE table). Therefore, it is applicable only for one-way and asynchronous BPEL processes with bpel.config.oneWayDeliveryPolicy set to async.persist (the default value) in the composite.xml file.
Attachments are not persisted in the following use cases:
If the SOAP message was received by a synchronous BPEL process or a one-way/asynchronous BPEL process with bpel.config.oneWayDeliveryPolicy set to sync or async.cache.
Contents of all elements within the SOAP request with inline binary content are not persisted, but passed as-is. (That is, they do not have a child element <xop:Include>, but do have a base64 encoded string as a child.) An MTOM-optimized message can be a mix of one or more elements that have inline base64 data, and one or more elements that are XOP-packaged, at any level.
Note:
Even if the service binding component is MTOM-enabled, it does not automatically indicate that the service receives MTOM-optimized messages. The calling service/application must send MTOM-optimized messages over the wire to ensure the message is received. MTOM-enabled bindings can also receive ordinary non-MTOM messages. Therefore, when it receives one, the SOAP requests arriving into the service can have nonoptimized inline binary data elements that are not be persisted into the database.
Even though the content of the MTOM-optimized elements or SwA attachments have their value replaced by an href attribute at runtime, their design-time WSDLs still remain unaltered. You do not see these changes in Oracle JDeveloper. Their element type definitions do not change from hexBinary, base64Binary, and so on to that of a empty content with an href attribute.
However, this is transparent to you. For instance, when you use an assign activity to copy across their content, the href values are copied over at runtime. Similarly, when invoking an outbound reference such as a web service or an adapter, Oracle SOA Suite automatically resolves the href attribute to the actual data and executes the invocation.
This section describes use cases for processing large XML with repeating constructs.
In this use case, the inbound adapter splits a source document into multiple batches of records, each of which initiates a composite instance. Table 45-8 provides details.
| Capability | Description |
|---|---|
|
Security |
N/A. |
|
Filter/Transformation/Assign |
Supported. |
|
Fanout |
Supported. |
|
Binding |
The file/FTP adapter debatches it to a small chunk based on the native XSD (NXSD) definition. |
|
Oracle BPEL Process Manager/Oracle Mediator |
Supported. |
|
Tuning |
For repeating structures, XSLT is supported for scenarios in which the repeating structure is of smaller payloads compared to the overall payload size. Substitution with assign activities is preferred, as it performs a shadow copy. |
|
Documentation |
See Oracle Fusion Middleware User's Guide for Technology Adapters. |
In this use case, a loop within a BPEL process reads a chunk of records at a time and process (that is, cursor). Table 45-9 provides details.
| Capability | Description |
|---|---|
|
Security |
Supported. |
|
Filter/Transformation/Assign |
Supported. |
|
Fanout |
Supported. |
|
Oracle BPEL Process Manager/Oracle Mediator |
Supported only from Oracle BPEL Process Manager. |
|
Documentation |
See Oracle Fusion Middleware User's Guide for Technology Adapters. |
This section describes use cases for processing very large XML documents with complex structures.
In this use case, very large XML files are streamed through Oracle SOA Suite. Table 45-10 provides details.
| Capability | Description |
|---|---|
|
Security |
N/A. |
|
Filter/Transformation/Assign |
Supported, but must optimize to avoid issues. |
|
Fanout |
Supported. |
|
Binding |
The adapter streams the payload to a database as an SDOM and passes the key to the service engines. |
|
Documentation |
See Oracle Fusion Middleware User's Guide for Technology Adapters. |
In this use case, large XML files are passed by Oracle B2B to Oracle SOA Suite as an SDOM. This only occurs when a large payload size is defined in the Oracle B2B user interface. Table 45-11 provides details.
| Capability | Description |
|---|---|
|
Security |
N/A. |
|
Filter/Transformation/Assign |
Supported, but must optimize to avoid issues. |
|
Fanout |
Supported. |
|
Binding |
Oracle B2B streams the payload to a database as SDOM and passes the key to the service engines. |
|
Oracle BPEL Process Manager/Oracle Mediator |
Can use an XPath extension function to manipulate the payload. |
This section describes the limitations on concurrent processing of large documents.
There is a limitation when you use an opaque schema for processing large payloads. The entire data for the opaque translator is converted to a single Base64-encoded string. An opaque schema is generally used for smaller data. For large data, use the attachments feature instead of the opaque translator.
This section provides general tuning recommendations.
For more information about Oracle SOA Suite tuning and performance, see Oracle Fusion Middleware Performance and Tuning Guide.
This section provides general tuning recommendations.
Increase the JTA transaction timeout to 500 seconds in Oracle WebLogic Server Administration Console. For instructions, see section "Resolving Connection Timeouts" of Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
In Oracle Enterprise Manager Fusion Middleware Control, set the audit level to Off or Production at the SOA composite application level. See Section 45.1.3.2, "Setting Audit Levels from Oracle Enterprise Manager for Large Payload Processing" for additional information.
Uncomment the following line in setDomainEnv.sh (for Linux) or setDomainEnv.bat (for Windows) for JAVA_OPTIONS, and restart the server. If this line does not exist, add it. Without this setting, large payload scenarios fail with ResourceDisabledException error for the dehydration data source.
-Dweblogic.resourcepool.max_test_wait_secs=30
Update the heap size in setSOADomainEnv.sh or setDomainEnv.bat as follows:
DEFAULT_MEM_ARGS="-Xms1024m -Xmx2048m"
Use optimized translation functions, which are available while performing transformations and translations of large payloads (for example, ora:doTranslateFromNative, ora:doTranslateToNative, ora:doStreamingTranslate, and so on).
For information about these functions, see Appendix B, "XPath Extension Functions."
Extend data files for handling large attachments. For more information, see the Oracle Database Administrator's Guide.
If you are processing large documents and run into timeout errors, perform the following tasks:
Increase the timeout property value.
Increase the Stuck Thread Max Time property value.
Increase the timeout property value as follows:
Log in to Oracle Web Services Manager Administration Console.
Navigate to Deployments > soa-infra > EJBs.
Click each of the following beans, select Configuration, and increase the timeout value:
BpelEngineBean
BpelDeliveryBean
CompositeMetaDataServiceBean
Increase the Stuck Thread Max Time property value as follows:
Follow the instructions in Chapter "Using the WebLogic 8.1 Thread Pool Model" of Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server.
For large payload processing, turn off audit level logging for the specific composite. You can set the composite audit level option to Off or Production in Oracle Enterprise Manager Fusion Middleware Control. If you set the composite audit level option to Development, then it serializes the entire large payload into an in-memory string, which can lead to an out-of-memory error.
For more information about setting audit levels, see Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
When using the assign activity in Oracle BPEL Process Manager or Oracle Mediator to manipulate large payloads, do not assign the complete message. Instead, assign only the part of the payload that you need.
In addition, when using the assign activity in Oracle BPEL Process Manager, Oracle recommends using local variables instead of process variables, wherever possible. Local variables are limited to the scope of the BPEL process. These get deleted from memory and from the database after you close the scope. However, the life cycle of a global variable is tied with the instance life cycle. These variables stay in memory or remain on disk until the instance completes. Thus, local variables are preferred to process or global variables.
Until 11g Release 1 11.1.1.3, for XSLT operations in Oracle BPEL Process Manager, the result was cached into memory as a whole document in binary XML format. For large document processing, this caused out-of-memory errors. Starting with 11g Release 1 11.1.1.4, a the streamResultToTempFile property has been added. This property enables XSLT results to be streamed to a temporary file and then loaded from the temporary file. Set streamResultToTempFile to yes when processing large payload using XSLT. The default value is no.
This property is applicable when using the following BPEL XPath functions:
ora:processXSLT('template','input','properties'?)
ora:doXSLTransformForDoc ('template','input','name', 'value')
To configure large XML documents to be processed using XSLT:
Create a BPEL common properties schema. For example:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema targetNamespace ="http://schemas.oracle.com/service/bpel/common"
xmlns:common = "http://schemas.oracle.com/service/bpel/common"
xmlns:xs = "http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified" blockDefault="#all">
<xs:element name="serviceProperties" type="common:PropertiesType"/>
<xs:element name="anyProperties" type="common:ArrayOfNameAnyTypePairType"/>
<xs:complexType name="NameValuePairType">
<xs:sequence>
<xs:element name="name" type="xs:string"/>
<xs:element name="value" type="xs:string"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="ArrayOfNameValuePairType">
<xs:sequence>
<xs:element name="item" type="common:NameValuePairType"
maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="NameAnyTypePairType">
<xs:sequence>
<xs:element name="name" type="xs:string"/>
<xs:element name="value" type="xs:anyType"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="ArrayOfNameAnyTypePairType">
<xs:sequence>
<xs:element name="item" type="common:NameAnyTypePairType"
maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="PropertiesType">
<xs:sequence>
<xs:element name="property" type="common:NameValuePairType"
maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="ArrayOfAnyTypeType">
<xs:sequence>
<xs:element name="item" type="xs:anyType" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:schema>
Within a BPEL process, add the namespace in the import section:
xmlns:common = "http://schemas.oracle.com/service/bpel/common"
Create a global variable (for this example, named propertiesXMLVar):
<variable name="propertiesXMLVar" element="common:anyProperties"/>
Set the streamResultToTempFile property to yes. This assign activity should exist before performing an XSLT transformation.
<assign name="Assign_xsltprop">
<copy>
<from>
<common:anyProperties>
<common:item>
<common:name>streamResultToTempFile</common:name>
<common:value>yes</common:value>
</common:item>
</common:anyProperties>
</from>
<to variable="propertiesXMLVar"/>
</copy>
</assign>
Until 11g Release 1 11.1.1.3, for XSLT operations in Oracle Mediator, the result was cached into memory as a whole document in binary XML format. For large document processing, this caused out-of-memory errors. Starting with 11g Release 1 11.1.1.4, the streamResultToTempFile property is available. This property enables XSLT results to be streamed to a temporary file and then loaded from the temporary file. Set streamResultToTempFile to yes when processing large payloads using XSLT. The default value is no.
Note:
This property is recommended only for processing large payloads. Enabling this property could reduce performance for normal payloads.
To configure large XML documents to be processed using XSLT:
Create an Oracle SOA Suite project with an Oracle Mediator component.
Open the composite.xml file for the project in Source view.
In the composite.xml file, scroll to the component element that defines the Oracle Mediator component to process large XML documents, and add the streamResultToTempFile property. Set the property to yes as shown below.
<component name="Mediator1"> <implementation.mediator src="Mediator1.mplan"/> <property name="streamResultToTempFile">yes</property> </component>
Save and close the file.
In scenarios in which the repeating structure is of smaller payloads compared to the overall payload size, Oracle recommends using XSLT transformation because the current XSLT implementation materializes the entire DOM in memory. For example, use PurchaseOrder.LineItem.Supplier (a subpart of a large payload).
You can also substitute it with the assign activity, as it performs a shadow copy. Although a shadow copy does not materialize DOM, it creates a shadow node to point to the source document.
You can also use the following optimized translation functions while performing transformations/translations of large payloads:
ora:doTranslateFromNative or med:doTranslateFromNative
ora:doTranslateToNative or med:doTranslateToNative
ora:doStreamingTranslate or med:doStreamingTranslate
For more information about these functions, see Appendix B, "XPath Extension Functions" and Oracle Fusion Middleware User's Guide for Technology Adapters.
For processing large documents in Oracle B2B, tune the following parameters:
mdsCache
Cache Size
Protocol Message Size
Number of threads
Stuck Thread Max Time
Tablespace
The following sections describe the parameters you must set for processing large documents in Oracle B2B.
To set the Oracle Metadata Services (MDS) Repository instance cache size, the property and value must be added to the $DOMAIN_HOME/config/soa-infra/configuration/b2b-config.xml file, as shown in Example 45-8.
If Oracle B2B wants to send or receive more than 10 MB of message or the import/export configuration is more than 10 MB, then change the following setting accordingly at the Oracle WebLogic Server Administration Console:
To configure the protocol message size:
In the Domain Structure, select Environment > Servers.
In the Name column of the table, select soa_server.
Select the Protocols tab.
Change the value for Maximum Message Size.
This setting can also be added/modified in the $DOMAIN_HOME/config/config.xml file next to the server name configuration, as shown in Example 45-9.
Example 45-9 max-message-size Property
<name>soa_server1</name> <max-message-size>150000000</max-message-size>
Note:
By default, max-message-size is not available in the config.xml file.
This parameter improves the message processing capability of Oracle B2B and must be set in the $DOMAIN_HOME/config/soa-infra/configuration/b2b-config.xml file. Example 45-10 provides an example.
Example 45-10 Number of Threads
<property> <name>b2b.inboundProcess.threadCount</name> <value>5</value> <comment></comment> </property> <property> <name>b2b.inboundProcess.sleepTime</name> <value>10</value> <comment></comment> </property> <property> <name>b2b.outboundProcess.threadCount</name> <value>5</value> <comment></comment> </property> <property> <name>b2b.outboundProcess.sleepTime</name> <value>10</value> <comment></comment> </property> <property> <name>b2b.defaultProcess.threadCount</name> <value>5</value> <comment></comment> </property> <property> <name>b2b.defaultProcess.sleepTime</name> <value>10</value> <comment></comment> </property>
The Stuck Thread Max Time parameter checks the number of seconds that a thread must continually work before the server considers the thread stuck. You must change the following setting in the Oracle WebLogic Server Administration Console:
To configure the Stuck Thread Max Time parameter:
In the Domain Structure, select Environment > Servers.
In the Name column of the table, select soa_server.
Select the Tuning tab.
Change the value for Stuck Thread Max Time.
If you must store more than a 150 MB configuration in the data file, then you must extend or add the data file to increase the tablespace size, as shown in Example 45-11.
If you want to set a size restriction on inbound web service message size, configure the binding component property max-message-size in the composite.xml file. The property value is made available to the underlying web service infrastructure, which uses the value to test against the incoming message size. If the value specified is exceeded, an exception is thrown indicating that the message size is too large and the transaction is not processed. Example 45-12 provides details.
Example 45-12 max-message-size Setting in composite.xml
<composite name="LrgMsg" revision="1.0" label="2011-09-08_22-53-53_259"
mode="active" state="on">
<import namespace="http://xmlns.oracle.com/LargeMsg/LrgMsg/BPELProcess1"
location="BPELProcess1.wsdl" importType="wsdl"/>
<service name="bpelprocess1_client_ep" ui:wsdlLocation="BPELProcess1.wsdl">
<interface.wsdl
interface="http://xmlns.oracle.com/LargeMsg/LrgMsg/BPELProcess1# wsdl.interface
(BPELProcess1)"/>
<binding.ws port="http://xmlns.oracle.com/LargeMsg/LrgMsg/BPELProcess1
#wsdl.endpoint(bpelprocess1_client_ep/BPELProcess1_pt)">
<property name="max-message-size" type="xs:integer" many="false"
override="may">4</property>
</binding.ws>
</service>
<component name="BPELProcess1" version="1.1">
<implementation.bpel src="BPELProcess1.bpel"/>
</component>
<wire>
<source.uri>bpelprocess1_client_ep</source.uri>
<target.uri>BPELProcess1/bpelprocess1_client</target.uri>
</wire>
</composite>
You can use the following functions to write the results of large XSLT/XQuery operations to a temporary file in a directory system. The document is then loaded from the temporary file when needed. This eliminates the need for caching an entire document as binary XML in memory.
ora:processXSLT
ora:doXSLTransformForDoc
With the ora:processXSLT function, you use the properties argument to enable this functionality.
ora:processXSLT('template','input','properties'?)
You retrieve the value of this argument within your XSLT in a way similar to extracting data from XSL variables. The properties argument is an XML element of the structure shown in Example 45-13. For large payload results (for example, above 10 MB), set streamResultToTempFile to yes. For small payload results in which you do not need to write results to a temporary file, leave this property set to its default value of no.
<propertiesXMLVar>
<common:item xmlns:common="http://schemas.oracle.com/service/bpel/common">
<common:name>streamResultToTempFile</common:name>
<common:value>yes</common:value>
</common:item>
</propertiesXMLVar>
Within the XSLT, the parameters are accessible through the name of streamResultToTempFile and its value of yes.
In Oracle BPEL Process Manager, a literal assign is performed to populate the properties for ora:processXSLT('template','input','properties'?).
For more information on using this function, see Section B.2.52, "processXSLT."
With the ora:doXSLTransformForDoc function, you set the name and value properties to enable this functionality.
ora:doXSLTransformForDoc ('template','input','name', 'value')
With this function, the name of streamResultToTempFile and the value of yes are passed.
For more information on using the function, see Section B.2.11, "doXSLTransformForDoc."
This section provides recommendations for handling large metadata.
There is a limit to the number of activities that can be executed in a BPEL process. When you exceed this limit, system memory fills up, which can cause timeouts to occur. For example, with the following parameters, two fault instances occur due to a timeout:
100 threads
1 second of think time
1000 incoming request messages
Keep the number of incoming request messages at a proper level to ensure system memory stability.
To deploy BPEL processes that have a large number of activities (for example, 50,000), the following settings are required:
MEM_ARGS: -Xms512m -Xmx1024m -XX:PermSize = 128m -XX:MaxPermSize = 256m Number of Concurrent Threads = 20 Number of Loops = 5 Delay = 100 ms
The above settings enable you to deploy and execute BPEL processes, which use only while loops without the flowN activities, successfully.
To deploy BPEL processes that have a large number of activities (for example, 50,000), the following settings are required:
USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=256m Number of Concurrent Threads= 10 Number of Loops=5 Delay=100 ms
Set the StatsLastN property to -1 in the System MBean Browser of Oracle Enterprise Manager Fusion Middleware Control.
The above settings enable you to deploy and execute BPEL processes, which use the flowN activities, successfully.
For more information, see Section 10.3.1, "Customizing the Number of Flow Activities with the flowN Activity in BPEL 1.1" and Section "Configuring BPEL Process Service Engine Properties" of Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
BPEL processes that have up to 7000 activities can be deployed and executed successfully with the following settings:
USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=256m
Note:
If you deploy BPEL processes with more than 8000 activities, Oracle BPEL Process Manager compilation throws errors.
BPEL processes that have up to 7000 activities can be deployed and executed successfully with the following settings:
USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=512m
Note:
If you deploy BPEL processes with more than 10,000 activities, the process compilation fails.
You can deploy and execute BPEL processes that have a large number of activities (for example, up to 5000) successfully.
There is a probability that the BPEL process compilation may fail for 6000 activities.
Oracle recommends that you not have more than 50 Oracle Mediators in a single composite. Increase the JTA Transaction timeout to a high value based on the environment.
Oracle recommends that you do not use browsers for large data set imports, and that you use the command line utility. The following utility commands are recommended for large data configuration:
purge: Purges the entire repository.
import: Imports the specified ZIP file.
deploy: Deploys an agreement with whichever name is specified. If no name is specified, then all the agreements are deployed.
However, the purgeimportdeploy option is not recommended for transferring or deploying the Oracle B2B configuration.
For more information, see Oracle Fusion Middleware User's Guide for Oracle B2B.
This section provides recommendations for handling large numbers of instance and fault metrics.
Deleting thousands of instances and rejected messages in Oracle Enterprise Manager Fusion Middleware Control takes time and can result in a transaction timeout. If you must perform this task, use the PL/SQL purge script for instance and rejected message deletion.
For more information, see Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.
You can improve the loading of pages that display large amounts of instance and fault data in Oracle Enterprise Manager Fusion Middleware Control by setting two properties in the Data Display Options section of the SOA Infrastructure Common Properties page.
These two properties enable you to perform the following:
Disable the fetching of instance and fault count data to improve loading times for the following pages:
Dashboard pages of the SOA Infrastructure, SOA composite applications, service engines, and service components
Delete with Options: Instances dialog
These settings disable the loading of all metrics information upon page load. For example, on the Dashboard page for the SOA Infrastructure, the values that typically appear in the Running and Total fields in the Recent Composite Instances section and the Instances column of the Deployed Composites section are replaced with links. When these values are large, it can take time to load this page and other pages with similar information.
Specify a default time period used as part of the search criteria for retrieving recent instances and faults for display on the following pages:
Dashboard pages and Instances pages of the SOA Infrastructure, SOA composite applications, service engines, and service components
Dashboard pages of services and references
Faults and Rejected Messages pages of the SOA Infrastructure, SOA composite applications, services, and references
Faults pages of service engines and service components
For more information about setting these properties, see Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite.