1. Developing SOA Applications with Oracle SOA Suite
  2. Advanced Topics
  3. Managing Large Documents and Large Numbers of Instances

51 Managing Large Documents and Large Numbers of Instances

This chapter describes the best practices for managing large documents and metadata and managing environments with large numbers of instances in Oracle SOA Suite. It also describes use cases for handling large documents, limitations on concurrent processing of large documents, and tuning recommendations.

This chapter includes the following sections:

For more information about Oracle SOA Suite tuning and performance, see Tuning Performance.

For information about troubleshooting Oracle SOA Suite issues, see Chapter "Troubleshooting Oracle SOA Suite and Oracle BPM Suite" of Administering Oracle SOA Suite and Oracle Business Process Management Suite.

For information about using Oracle Data Integrator to perform fast bulk data movement and handle complex data transformations, visit the following URL:

http://www.oracle.com/technetwork/middleware/data-integrator

51.1 Best Practices for Handling Large Documents

This section describes the following scenarios for handling large documents and the best practice approach for each scenario. Oracle recommends that you follow these best practices before developing and executing large payloads.

51.1.1 Use Cases for Handling Large Documents

This section describes use cases for handling large documents.

51.1.1.1 Passing Binary Objects as Base64-Encoded Text in XML Payloads

This section describes use cases for passing binary objects as Base64-encoded text in the XML payload.

51.1.1.1.1 SOAP Inline

In this use case, the binary attachments (for example, an image) are Base64-encoded as text and then passed inline in the XML document. Table 51-1 provides details.

Table 51-1 Capabilities

Capability Description

Security

Supported.

Filter/Transformation/Assign

Use of transformations may lead to slower performance, out-of-memory errors, or both.

Fanout

Supported.

Binding

WS binding sends it as a document object model (DOM).

Oracle BPEL Process Manager/Oracle Mediator

Can be decoded in a BPEL process using Java exec.

51.1.1.1.2 SOAP MTOM

In this use case, the binary attachments (for example, an image) are Base64-encoded as text and then passed as a Message Transmission Optimization Mechanism (MTOM) document. Table 51-2 provides details.

Table 51-2 Capabilities

Capability Description

Security

Supported.

Filter/Transformation/Assign

Assign activities are supported.

Fanout

Supported.

Binding

WS binding materializes the attachment sent as MTOM and puts it inside in Base64-encoded format (streaming is not supported).

Oracle BPEL Process Manager/Oracle Mediator

No additional work is required.
51.1.1.1.3 Opaque Passed by File/FTP Adapters

In this use case, the binary attachments (for example, an image) are Base64-encoded as text and then passed inline in the XML document. Table 51-3 provides details.

Table 51-3 Capabilities

Capability Description

Security

Not supported.

Filter/Transformation/Assign

Pass through.

Fanout

Supported.

Binding

Adapter encodes it to Base64 format.

Oracle BPEL Process Manager/Oracle Mediator

Supported. Opaque content cannot be manipulated in an assign or a transform activity.
51.1.1.1.4 Opaque Passed by Oracle B2B

In this use case, the binary attachments (for example, an image) are Base64-encoded as text encoded. Table 51-4 provides details.

Table 51-4 Capabilities

Capability Description

Security

Not supported.

Filter/Transformation/Assign

Pass through.

Fanout

Supported.

Oracle B2B

Oracle B2B encodes the native payload to Base64 format. For this scenario, you must configure the Oracle B2B binding document definition handling to be opaque.
51.1.1.2 End-to-End Streaming with Attachments

This section describes use cases for end-to-end streaming of attachments.

Note:

Direct Internet Message Encapsulation (DIME) attachments are not supported.

51.1.1.2.1 SOAP with Attachments

In this use case, the binary attachments (for instance, an image) are passed end-to-end as a stream. Table 51-5 provides details.

Table 51-5 Capabilities

Capability Description

Security

Not supported.

Filter/Transformation/Assign

Pass through. You must use an XPath extension function in Oracle BPEL Process Manager.

Binding

WS binding creates stream iterators for the SOAP attachment.

Oracle BPEL Process Manager/Oracle Mediator

Oracle Mediator can perform a pass through without materializing it. Oracle BPEL Process Manager persists it.

Tuning

Manage the database tablespace when using with Oracle BPEL Process Manager.

WSDL code for defining SOAP with attachments

 
<mime:part>
   <mime:content part="bin" type=“image/jpeg"/>
</mime:part>

Note:

  • You cannot stream attachments as part of a web service callback response.

  • The spring service component does not support processing MIME attachments. Only MTOM attachments are supported.

  • You can use various binding components such as direct binding, web services, and so on to process large attachments. However, processing large attachments with direct binding is not recommended and results in out-of-memory errors.

51.1.1.2.2 Working with Streaming Attachments

Oracle Fusion Middleware web services enable you to pass large attachments as a stream. Unlike the JAX-RPC API, which treats attachments as if they are entirely in memory, streams make the programming model more efficient to use. Streams also enhance performance and scalability because there is no need to load the attachment into memory before service execution.

As with embedded attachments, streamed attachments conform to the multipart MIME binary format. Embedded attachments refer to inlined/encoded attachments.

On the wire, messages with streamed attachments are identical to any other SOAP message with attachments.

The following example provides a sample message with a streamed attachment. The first part in the message is the SOAP envelope (<SOAP-ENV:Envelope...). The second part is the attachment (for this example, myImage.gif). 

MIME-Version: 1.0
Content-Type: Multipart/Related; boundary=MIME_boundary; type=text/xml;
Content-Description: This is the optional message description.

--MIME_boundary
Content-Type: text/xml; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-ID: NotSure/DoesntMatter

<?xml version='1.0' ?>
<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Body>
. . .
<DocumentName>MyImage.gif</DocumentName>
. . .
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

--MIME_boundary
Content-Type: image/gif
Content-Transfer-Encoding: binary
Content-ID: AnythingYoudLike

...binary GIF image...
--MIME_boundary--
51.1.1.2.3 Creating Composites that Use MIME Attachments

Perform the following procedures to create composites that use MIME attachments.

To create composites that use MIME attachments:

  1. Create a composite using a payload schema (for example, an inbound web service wired to an Oracle Mediator wired to an outbound web service).

  2. Within the WSDL file of Oracle Mediator, perform the following steps:

    1. From the WSDL designer, open the Oracle Mediator WSDL file.

    2. Drag and drop bindings into the middle swimlane.

    3. Select the RPC binding.

    4. Enter a name.

    5. Go to Source view of the WSDL and modify the WSDL input and WSDL output with MIME multiparts. 

      <wsdl:input>
      <mime:multipartRelated>
          <mime:part>
              <soap:body parts="payload" use="literal"/>
          </mime:part>
          <mime:part>
              <mime:content part="bin" type="application/octet-stream"/>
          </mime:part>
      </mime:multipartRelated>
</wsdl:input>

    6. Add the MIME part in the request/response message.

      <wsdl:message name="BPELProcess1RequestMessage">
    <wsdl:part name="payload" element="ns1:purchaseOrder" />
    <!--add below part-->
    <wsdl:part name="bin" type="xsd:base64Binary"/>
</wsdl:message>

    7. Add a namespace in the WSDL definitions.

      xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/">

When complete, the WSDL that references a MIME attachment is displayed.

<wsdl:definitions
  name="PhotoCatalogService"
  targetNamespace="http://examples.com/PhotoCatalog"
  xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
  xmlns:types="http://examples.com/PhotoCatalog/types"
  xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/"
  xmlns:tns="http://examples.com/PhotoCatalog">
  <wsdl:message name="addPhotoRequest">
    <wsdl:part name="photo" type="xsd:hexBinary"/>
  </wsdl:message>
  <wsdl:message name="addPhotoResponse">
    <wsdl:part name="status" type="xsd:string"/>
  </wsdl:message>
  <wsdl:message name="replacePhotoRequest">
    <wsdl:part name="oldPhoto" type="xsd:string"/>
    <wsdl:part name="newPhoto" type="xsd:hexBinary"/>
  </wsdl:message>
  <wsdl:message name="replacePhotoResponse">
    <wsdl:part name="status" type="xsd:string"/>
  </wsdl:message>
  <wsdl:portType name="PhotoCatalog">
    <wsdl:operation name="addPhoto">
      <wsdl:input message="tns:addPhotoRequest"/>
      <wsdl:output message="tns:addPhotoResponse"/>
    </wsdl:operation>
    <wsdl:operation name="replacePhoto">
      <wsdl:input message="tns:replacePhotoRequest"/>
      <wsdl:output message="tns:replacePhotoResponse"/>
    </wsdl:operation>
  </wsdl:portType>
  <wsdl:binding name="PhotoCatalogBinding" type="tns:PhotoCatalog">
    <soap:binding style="document"
transport="http://schemas.xmlsoap.org/soap/http"/>
    <wsdl:operation name="addPhoto">
      <wsdl:input>
        <mime:multipartRelated>
          <mime:part>
            <soap:body use="literal"/>
          </mime:part>
          <mime:part>
            <mime:content part="photo"
                          type="image/jpeg"/>
          </mime:part>
        </mime:multipartRelated>
      </wsdl:input>
      <wsdl:output>
        <mime:multipartRelated>
          <mime:part>
            <soap:body use="literal"/>
          </mime:part>
          <mime:part>
            <mime:content part="status" type="text/plain"/>
            <mime:content part="status" type="text/xml"/>
          </mime:part>
        </mime:multipartRelated>
      </wsdl:output>
    </wsdl:operation>
    <wsdl:operation name="replacePhoto">
      <wsdl:input>
        <mime:multipartRelated>
          <mime:part>
            <soap:body parts="oldPhoto" use="literal"/>
          </mime:part>
          <mime:part>
            <mime:content part="newPhoto"
                          type="image/jpeg"/>
          </mime:part>
        </mime:multipartRelated>
      </wsdl:input>
      <wsdl:output>
            <soap:body parts="status" use="literal"/>
      </wsdl:output>
    </wsdl:operation>
  </wsdl:binding>
</wsdl:definitions>
51.1.1.2.4 Performance Overhead and Pass Through Attachments

Because Oracle Mediator is stateless, there is no performance overhead with pass through attachments. However, Oracle BPEL Process Manager dehydrates attachments and has performance overhead, even for pass through attachments. When using Oracle BPEL Process Manager for attachments over a period, the SOA Infrastructure schema can grow to its maximum size and encounter memory issues. It is recommended that you extend the database tablespace appropriately for the SOA Infrastructure schema to accommodate large attachments. Simultaneously, you can use purge scripts to purge completed instances along with the attachments table.

For information about purge scripts, see Administering Oracle SOA Suite and Oracle Business Process Management Suite.

For information about extending tablespaces, see Section "Extending Tablespaces to Avoid Problems at Runtime" of Administering Oracle SOA Suite and Oracle Business Process Management Suite.

In scenarios in which one BPEL process calls a second BPEL process within the same composite, the second BPEL process does not dehydrate the same attachment again.

In scenarios in which one BPEL process from composite 1 invokes a second BPEL process from composite 2 and optimization is disabled, composite 1 makes a SOAP call to composite 2. The second BPEL process does dehydrate attachments.

51.1.1.2.5 Properties for Streaming Attachments

To stream attachments, add the following properties in the composite.xml file. If optimization is enabled, then a native call is used instead of a SOAP call. The following example provides details. 

<binding.ws
port="http://services.otn.com#wsdl.endpoint(MIMEService/MIMEService)"
xmlns:ns="http://xmlns.oracle.com/sca/1.0"
streamIncomingAttachments="true" streamOutgoingAttachments="true">
<!--Add this prop to reference bindings to make a SOAP call. -->
<property name="oracle.webservices.local.optimization">false</property>
</binding.ws>

For information about the oracle.webservices.local.optimization property, see Policy Attachments and Local Optimization in Composite-to-Composite Invocations and Configuring Local Optimization in Administering Oracle SOA Suite and Oracle Business Process Management Suite.

Note:

Oracle Web Services Manager (OWSM) does not inspect or enforce policies on streamed attachments. For more information about OWSM, see Administering Web Services.

51.1.1.2.6 Streaming Attachments from the SOA Web Service Binding Layer

You can receive the error shown in the following example when steaming attachments from the SOA web service (WS) binding layer.

java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2271)
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
        at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
        at

To resolve this error, add the following properties in the composite.xml file for service and reference binding components.

  • streamIncomingAttachments="true"

  • streamOutgoingAttachments="true"

See the preceding section for information about setting these properties.

51.1.1.2.7 Reading and Encoding SOAP Attachment Content

The ora:getAttachmentContent function reads SOAP attachment content and encodes that data in Base64 format in a BPEL process by providing the BPEL variable as an argument, which has an href of the SOAP attachment. The following example shows how to use this function: 

<copy>
   <from expression="ora:getAttachmentContent('input','bin')"/>
   <to variable="initiateTaskInput" part="payload"
    query="/taskservice:initiateTask/task:task/task:attachment/task:content"/>
</copy>

The preceding example copies the attachment content, which has its href stored in the "input/bin" variable, to the content variable in Base64-encoded format.

51.1.1.2.8 Sending Attachment Streams

Oracle BPEL Process Manager supports sending the attachment stream to multiple receivers. For Oracle BPEL Process Manager to send a stream to multiple receivers, it must read the attachment stream from the database using the readBinaryFromFile XPath function and pass the stream to the appropriate targets.

With the default configuration, Oracle Mediator can pass an attachment stream to only one target receiver, which can be another component or a web service/adapter. The second target cannot receive the attachment. When you define the persistStreamAttachment property for the Oracle Mediator component, Oracle Mediator can pass an attachment stream to multiple target receivers.

Oracle Mediator requires the persistStreamAttachment property for streaming attachments where the source message that contains the attachment is shared by multiple target receivers. Set this property to true in composite.xml to enable the streaming of attachments to multiple targets. The following example provides details. 

component name="Mediator1">
   <implementation.mediator src="Mediator1.mplan"/>
   <property name="persistStreamAttachment">true</property>
</component>
51.1.1.2.9 Overriding Pass Through Settings for Attachments in Oracle Mediator

Oracle Mediator automatically propagates attachments to target receivers for Oracle Mediator components that are pass through (that is, they do not contain a transformation or assign rule), and it does not propagate attachments for Oracle Mediator components that are not pass through. The passThroughAttachment property lets you override the pass through settings just for attachments. Setting this property to true copies all attachments to the target receiver implicitly.

Use this property to propagate attachments when the Oracle Mediator component is not a pass through, or use it to block attachments when the Oracle Mediator component is pass through. To implement the pass through attachment override, add the property to the project's composite.xml file in the component element for the Oracle Mediator component. Set the property to true to override an Oracle Mediator component that is not pass through. Set it to false to override a pass through component. The following example provides details: 

<component name="Mediator">
    <implementation.mediator src="Mediator.mplan"/>
    <property name="passThroughAttachment">true</property>
</component>
51.1.1.2.10 Sharing Attachments Using Synchronous Flows

When Oracle BPEL Process Manager-based composites share attachments using synchronous flows, it is necessary to use the same end-to-end transaction. This is applicable to composites that are colocated and use local/optimized calls. This can be achieved by setting the property shown in the following example on all the called BPEL components (callees) in the call chain: 

<property name="bpel.config.transaction" many="false"
type="xs:string">required</property>

If such composites do not execute as part of the same transaction context, the attachment data saved by the first BPEL component in the call chain is not visible to the other BPEL components in the call chain. In addition, they incur database locking and timeout exceptions:

"ORA-02049: timeout: distributed transaction waiting for lock"
51.1.1.2.11 Attachment Options of File/FTP Adapters

In this use case, the adapter streams the binary data to a database store and publishes an href to the service engine (Oracle BPEL Process Manager or Oracle Mediator). Table 51-6 provides details.

Table 51-6 Capabilities

Capability Description

Security

N/A.

Filter/Transformation/Assign

Filters and transformations on the attachment are not supported.

Fanout

Supported.

Binding

The adapter streams the non-XML to the database as a binary large object (BLOB) and passes the key to the service engines.

Oracle BPEL Process Manager/Oracle Mediator

Supported.

Tuning

  • Extend the database tablespace for the Oracle SOA Suite schema.

  • Delete the attachments after message processing completion.

Documentation

See Understanding Technology Adapters.

Writing Attachments Using an Outbound File Adapter

The following example shows a sample schema that can be used by the file adapter to write attachments to disk:

<?xml version="1.0" encoding="windows-1252" ?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            xmlns="http://xmlns.oracle.com/attachment"
            targetNamespace="http://xmlns.oracle.com/attachment"
            elementFormDefault="qualified">
            <xsd:element name="attach">
              <xsd:complexType>
                 <xsd:attribute name="href" type="xsd:string"/>
              </xsd:complexType>
  </xsd:element>
</xsd:schema>

Use Oracle Mediator in the flow to map the attachment part from the source (Oracle Mediator) to the target (file adapter) using an Oracle Mediator assign.

If you use Oracle BPEL Process Manager, the attachment is written to the dehydration store, which slows down the process.

Transforming Attachments with the ora:doStreamingTranslate XPath Function

Use of the ora:doStreamingTranslate XPath function is only recommended while transforming attachments within an Oracle BPEL Process Manager or Oracle Mediator service component. This function expects the attachment location to be a relative path on the server. This function cannot translate incoming attachment streams.

For more information about this function, see doStreamingTranslate.

51.1.1.2.12 Oracle B2B Attachment

In this use case, Oracle B2B stores the binary data to a database and publishes an href to the service engine (Oracle BPEL Process Manager or Oracle Mediator) based on an Oracle B2B-defined XSD. Oracle B2B protocols define the attachment. Table 51-7 provides details.

Table 51-7 Capabilities

Capability Description

Security

N/A.

Filter/Transformation/Assign

Filters and transformations on the attachment are not supported.

Fanout

Supported.

Binding

Oracle B2B passes it as an href key to service engines.

Tuning

Extend the database tablespace for the Oracle SOA Suite schema.
51.1.1.3 Sending and Receiving MTOM-Optimized Messages to SOA Composite Applications

Within a SOA composite application, you must attach the Oracle WS-MTOM policy to service and reference binding components to receive and send MTOM (MIME binary) optimized messages. When a service binding component (defined under binding.ws in the composite.xml file) is configured with an Oracle WS-MTOM policy, Oracle SOA Suite's MTOM message handling feature is used. When a reference binding component (also defined under binding.ws in the composite.xml file) is configured with an Oracle MTOM policy, Oracle SOA Suite sends MTOM-optimized messages.

Note the following issues with MTOM attachments:

  • When attachments are inline and encoded, Oracle recommends that you not use the file adapter to write attachments to a file.

  • The default mtomThreshold value is 1024 bytes and cannot be modified. If an attachment is less than 1024 bytes, for outbound configurations, Oracle SOA Suite sends it as an inline attachment. If the size is greater than 1024 bytes, then the attachment is sent as an attachment part with an href attribute in the message, and is sent as a WSDL-defined format on the wire. However, if the incoming request (for example, from a different web services provider) has an xop href node for small binary data (that is, size is less than 1024 bytes), Oracle SOA Suite uses the same href attribute in the payload in the flow trace. For example: 

    <xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include"
 href="cid:e29caf23dc8045908451fdfaafa26dce" />

  • If a service binding component of a composite does not include an Oracle WS-MTOM policy reference, this indicates that the service can accept non-MTOM messages. This indicates that the calling composite (the appropriate reference binding) does not have an Oracle WS-MTOM policy reference and can send out non-MTOM messages to that service.

  • MTOM streaming of attachments is not supported by Oracle SOA Suite.

  • MTOM attachments are supported only with web service bindings. Other bindings (for example, HTTP bindings) are not supported.

  • Oracle Mediator pass through scenarios are supported. If Oracle Mediator does not contain any transformation or assign statements, it is known as a pass through Oracle Mediator. The message and attachment received are propagated to the target without modifying the payload and attachment. Likewise, multiple MTOM attachments in the same message can be sent and received by Oracle SOA Suite.

  • Oracle recommends that you not use both streaming and the MTOM message handling feature for sending and receiving attachments. Use either streaming or the MTOM message handling feature.

    Note:

    If the input is of type text/xml, there is no significant decrease in file size when sending files in MTOM format.

  • As a best practice, Oracle recommends that you not use the XSLT Map Editor to propagate binary data. Instead, use an assign activity. If you must use a style sheet to propagate binary data, it is recommended that you use the xsl:copy-of instruction (copy-of copies everything, including attributes) or use custom functions to copy attributes from source to target.

  • MTOM attachments should not be gigabytes in size. Instead, use the SOAP with attachments streaming feature for very large attachments. For more information, see SOAP with Attachments.

51.1.1.3.1 Scenarios for Storing SwA and MTOM-Optimized Attachments to the Database

When a SOA composite application with a BPEL process receives an MTOM-optimized SOAP message, the attachment contents of each of the MTOM-optimized elements (the ones with an <xop href="">) are stored in the dehydration store. Similarly, when receiving a SOAP message with attachments (SwA) message with one or more attachments, each attachment is stored in the dehydration store. These attachments can then be passed around by reference using an href attribute that identifies them in the database. In fact, all of the text content of these attachment elements is removed and replaced by this href attribute. For MTOM-optimized messages, the same value of the incoming href attribute from the <xop> element is reused. Similarly, for SwA, the href attributes of the attachment elements are reused.

The attachments are stored in the dehydration store when the message is delivered to the BPEL process service engine. (when the incoming message is saved into the DLV_MESSAGE table). Therefore, it is applicable only for one-way and asynchronous BPEL processes with bpel.config.oneWayDeliveryPolicy set to async.persist (the default value) in the composite.xml file.

Attachments are not persisted in the following use cases:

  • If the SOAP message was received by a synchronous BPEL process or a one-way/asynchronous BPEL process with bpel.config.oneWayDeliveryPolicy set to sync or async.cache.

  • Contents of all elements within the SOAP request with inline binary content are not persisted, but passed as-is. (That is, they do not have a child element <xop:Include>, but do have a base64 encoded string as a child.) An MTOM-optimized message can be a mix of one or more elements that have inline base64 data, and one or more elements that are XOP-packaged, at any level.

Note:

Even if the service binding component is MTOM-enabled, it does not automatically indicate that the service receives MTOM-optimized messages. The calling service/application must send MTOM-optimized messages over the wire to ensure the message is received. MTOM-enabled bindings can also receive ordinary non-MTOM messages. Therefore, when it receives one, the SOAP requests arriving into the service can have nonoptimized inline binary data elements that are not be persisted into the database.

Even though the content of the MTOM-optimized elements or SwA attachments have their value replaced by an href attribute at runtime, their design-time WSDLs still remain unaltered. You do not see these changes in Oracle JDeveloper. Their element type definitions do not change from hexBinary, base64Binary, and so on to that of an empty content with an href attribute.

However, this is transparent to you. For instance, when you use an assign activity to copy across their content, the href values are copied over at runtime. Similarly, when invoking an outbound reference such as a web service or an adapter, Oracle SOA Suite automatically resolves the href attribute to the actual data and executes the invocation.

51.1.1.4 Processing Large XML with Repeating Constructs

This section describes use cases for processing large XML with repeating constructs.

51.1.1.4.1 Debatching with the File/FTP Adapter

In this use case, the inbound adapter splits a source document into multiple batches of records, each of which initiates a composite instance. Table 51-8 provides details.

Table 51-8 Capabilities

Capability Description

Security

N/A.

Filter/Transformation/Assign

Supported.

Fanout

Supported.

Binding

The file/FTP adapter debatches it to a small chunk based on the native XSD (NXSD) definition.

Oracle BPEL Process Manager/Oracle Mediator

Supported.

Tuning

For repeating structures, XSLT is supported for scenarios in which the repeating structure is of smaller payloads compared to the overall payload size. Substitution with assign activities is preferred, as it performs a shadow copy.

Documentation

See Understanding Technology Adapters.

51.1.1.4.2 Chunking with the File/FTP Adapters

In this use case, a loop within a BPEL process reads a chunk of records at a time and process (that is, cursor). Table 51-9 provides details.

Table 51-9 Capabilities

Capability Description

Security

Supported.

Filter/Transformation/Assign

Supported.

Fanout

Supported.

Oracle BPEL Process Manager/Oracle Mediator

Supported only from Oracle BPEL Process Manager.

Documentation

See Understanding Technology Adapters.

51.1.1.5 Processing Large XML Documents with Complex Structures

This section describes use cases for processing very large XML documents with complex structures.

51.1.1.5.1 Streaming with the File/FTP Adapters

In this use case, very large XML files are streamed through Oracle SOA Suite. Table 51-10 provides details.

Table 51-10 Capabilities

Capability Description

Security

N/A.

Filter/Transformation/Assign

Supported, but must optimize to avoid issues.

Fanout

Supported.

Binding

The adapter streams the payload to a database as an SDOM and passes the key to the service engines.

Documentation

See Understanding Technology Adapters.

51.1.1.5.2 Oracle B2B Streaming

In this use case, large XML files are passed by Oracle B2B to Oracle SOA Suite as an SDOM. This only occurs when a large payload size is defined in the Oracle B2B user interface. Table 51-11 provides details.

Table 51-11 Capabilities

Capability Description

Security

N/A.

Filter/Transformation/Assign

Supported, but must optimize to avoid issues.

Fanout

Supported.

Binding

Oracle B2B streams the payload to a database as SDOM and passes the key to the service engines.

Oracle BPEL Process Manager/Oracle Mediator

Can use an XPath extension function to manipulate the payload.

51.1.2 Limitations on Concurrent Processing of Large Documents

This section describes the limitations on concurrent processing of large documents.

51.1.2.1 Opaque Schema for Processing Large Payloads

There is a limitation when you use an opaque schema for processing large payloads. The entire data for the opaque translator is converted to a single Base64-encoded string. An opaque schema is generally used for smaller data. For large data, use the attachments feature instead of the opaque translator.

51.1.3 JVM Memory Sizing Recommendations for SOA Composite Applications

Sending messages with payloads that are 100 MB or larger in size can exceed JVM heap size limits if not correctly tuned.

For example, when sending large payloads in the event delivery network (EDN) with Oracle advanced queueing (AQ) JMS, ensure that you set the maximum memory value by first testing with a typical message payload size and a maximum potential message size. Using a lesser memory value can result in an ORACLE.JMS.AQJMSEXCEPTION error. For example, to send a payload of 100 MB, it is recommended that you change the JTA time out and maximum memory to 5 GB.

For more information about tuning the JVM heap size, see General Tuning Recommendations and Section "Java HotSpot VM Heap Size Options" of Tuning Performance of Oracle WebLogic Server.

51.1.4 General Tuning Recommendations

This section provides general tuning recommendations.

For more information about Oracle SOA Suite tuning and performance, see Tuning Performance.

51.1.4.1 General Recommendations

This section provides general tuning recommendations.

  • Increase the JTA transaction timeout to 500 seconds in Oracle WebLogic Server Administration Console. For instructions, see section "Resolving Connection Timeouts" of Administering Oracle SOA Suite and Oracle Business Process Management Suite.

  • In Oracle Enterprise Manager Fusion Middleware Control, set the audit level to Off or Production at the SOA composite application level. See Setting Audit Levels from for Large Payload Processing for additional information.

  • Uncomment the following line in setDomainEnv.sh (for Linux) or setDomainEnv.bat (for Windows) for JAVA_OPTIONS, and restart the server. If this line does not exist, add it. Without this setting, large payload scenarios fail with a ResourceDisabledException error for the dehydration data source. 

    -Dweblogic.resourcepool.max_test_wait_secs=30

  • Update the heap size in setSOADomainEnv.sh or setDomainEnv.bat as follows: 

    DEFAULT_MEM_ARGS="-Xms1024m -Xmx2048m"

  • Use optimized translation functions, which are available while performing transformations and translations of large payloads (for example, ora:doTranslateFromNative, ora:doTranslateToNative, ora:doStreamingTranslate, and so on).

    For information about these functions, see XPath Extension Functions.

  • Extend data files for handling large attachments.

  • Increase the HTTP POST timeout for SocketException: Broken pipe errors in Oracle WebLogic Server Administration Console. See Increasing the HTTP POST Timeout

  • If you are processing large documents and run into timeout errors, perform the following tasks:

    • Increase the timeout property value.

    • Increase the Stuck Thread Max Time property value.

    For more information, see Increasing the Timeout Value.

51.1.4.1.1 Increasing the HTTP POST Timeout

Increase the HTTP POST timeout for SocketException: Broken pipe errors in Oracle WebLogic Server Administration Console.

  1. From the Domain Structure, select soainfra > servers > server_name > Protocols > HTTP.

  2. In the Post Timeout field, enter 120 (maximum).

51.1.4.1.2 Increasing the Timeout Value

Increase the timeout property value as follows:

  1. Log in to Oracle Web Services Manager Administration Console.

  2. Navigate to Deployments > soa-infra > EJBs.

  3. Click each of the following beans, select Configuration, and increase the timeout value:

    • BpelEngineBean

    • BpelDeliveryBean

    • CompositeMetaDataServiceBean

To increase the Stuck Thread Max Time property value:

Follow the instructions in Chapter "Using the WebLogic 8.1 Thread Pool Model" of Tuning Performance of Oracle WebLogic Server.

51.1.4.2 Setting Audit Levels from Oracle Enterprise Manager for Large Payload Processing

For large payload processing, turn off audit level logging for the specific composite. You can set the composite audit level option to Off or Production in Oracle Enterprise Manager Fusion Middleware Control. If you set the composite audit level option to Development, it serializes the entire large payload into an in-memory string, which can lead to an out-of-memory error.

For more information about setting audit levels, see Administering Oracle SOA Suite and Oracle Business Process Management Suite.

51.1.4.3 Using the Assign Activity in Oracle BPEL Process Manager and Oracle Mediator

When using the assign activity in Oracle BPEL Process Manager or Oracle Mediator to manipulate large payloads, do not assign the complete message. Instead, assign only the part of the payload that you need.

In addition, when using the assign activity in Oracle BPEL Process Manager, Oracle recommends using local variables instead of process variables, wherever possible. Local variables are limited to the scope of the BPEL process. These get deleted from memory and from the database after you close the scope. However, the life cycle of a global variable is tied with the instance life cycle. These variables stay in memory or remain on disk until the instance completes. Thus, local variables are preferred to process or global variables.

51.1.4.4 Using XSLT Transformations on Large Payloads (For Oracle BPEL Process Manager)

Until 11g Release 1 11.1.1.3, for XSLT operations in Oracle BPEL Process Manager, the result was cached into memory as a whole document in binary XML format. For large document processing, this caused out-of-memory errors. Starting with 11g Release 1 11.1.1.4, a the streamResultToTempFile property was added. This property enables XSLT results to be streamed to a temporary file and then loaded from the temporary file. Set streamResultToTempFile to yes when processing large payload using XSLT. The default value is no.

This property is applicable when using the following BPEL XPath functions:

  • ora:processXSLT('template','input','properties'?)

  • ora:doXSLTransformForDoc('template','input','name', 'value')

To configure large XML documents to be processed using XSLT:

  1. Create a BPEL common properties schema. For example:
    <?xml version="1.0" encoding="UTF-8"?>
<xs:schema targetNamespace ="http://schemas.oracle.com/service/bpel/common" 
          xmlns:common = "http://schemas.oracle.com/service/bpel/common"
          xmlns:xs = "http://www.w3.org/2001/XMLSchema"
          elementFormDefault="qualified" blockDefault="#all">
   
  <xs:element name="serviceProperties"  type="common:PropertiesType"/> 
  <xs:element name="anyProperties"  type="common:ArrayOfNameAnyTypePairType"/> 
     <xs:complexType name="NameValuePairType">
    <xs:sequence>
      <xs:element name="name" type="xs:string"/>
      <xs:element name="value" type="xs:string"/>
    </xs:sequence>
   </xs:complexType>
   <xs:complexType name="ArrayOfNameValuePairType">
    <xs:sequence>
      <xs:element name="item" type="common:NameValuePairType"
 maxOccurs="unbounded"/>
    </xs:sequence>
   </xs:complexType>
   <xs:complexType name="NameAnyTypePairType">
    <xs:sequence>
      <xs:element name="name" type="xs:string"/>
      <xs:element name="value" type="xs:anyType"/>
    </xs:sequence>
   </xs:complexType>
   <xs:complexType name="ArrayOfNameAnyTypePairType">
    <xs:sequence>
      <xs:element name="item" type="common:NameAnyTypePairType"
 maxOccurs="unbounded"/>
    </xs:sequence>
   </xs:complexType>  
   <xs:complexType name="PropertiesType">
    <xs:sequence>
      <xs:element name="property" type="common:NameValuePairType"
 maxOccurs="unbounded"/>
    </xs:sequence>
   </xs:complexType>
   <xs:complexType name="ArrayOfAnyTypeType">
    <xs:sequence>
      <xs:element name="item" type="xs:anyType" maxOccurs="unbounded"/>
    </xs:sequence>
   </xs:complexType>   
</xs:schema>
  2. Within a BPEL process, add the namespace in the import section:
    xmlns:common = "http://schemas.oracle.com/service/bpel/common"
  3. Create a global variable (for this example, named propertiesXMLVar):
    <variable name="propertiesXMLVar" element="common:anyProperties"/>
  4. Set the streamResultToTempFile property to yes. This assign activity should exist before performing an XSLT transformation.
    <assign name="Assign_xsltprop">
      <copy>
        <from>
           <common:anyProperties>
             <common:item>
                   <common:name>streamResultToTempFile</common:name>
                   <common:value>yes</common:value>
             </common:item>
           </common:anyProperties>
        </from>
        <to variable="propertiesXMLVar"/>
      </copy>
</assign>
51.1.4.5 Using XSLT Transformations on Large Payloads (For Oracle Mediator)

Until 11g Release 1 11.1.1.3, for XSLT operations in Oracle Mediator, the result was cached into memory as a whole document in binary XML format. For large document processing, this caused out-of-memory errors. Starting with 11g Release 1 11.1.1.4, the streamResultToTempFile property was added. This property enables XSLT results to be streamed to a temporary file and then loaded from the temporary file. Set streamResultToTempFile to yes when processing large payloads using XSLT. The default value is no.

Note:

This property is recommended only for processing large payloads. Enabling this property could reduce performance for normal payloads.

To configure large XML documents to be processed using XSLT:

  1. Create an Oracle SOA Suite project with an Oracle Mediator component.
  2. Open the composite.xml file for the project in Source view.
  3. In the composite.xml file, scroll to the component element that defines the Oracle Mediator component to process large XML documents, and add the streamResultToTempFile property. Set the property to yes as shown below.
    <component name="Mediator1">
   <implementation.mediator src="Mediator1.mplan"/>
   <property name="streamResultToTempFile">yes</property>
</component>
  4. Save and close the file.
51.1.4.6 Using XSLT Transformations for Repeating Structures

In scenarios in which the repeating structure is of smaller payloads compared to the overall payload size, Oracle recommends using XSLT transformations because the current XSLT implementation materializes the entire DOM in memory. For example, use PurchaseOrder.LineItem.Supplier (a subpart of a large payload).

You can also substitute it with the assign activity, as it performs a shadow copy. Although a shadow copy does not materialize DOM, it creates a shadow node to point to the source document.

You can also use the following optimized translation functions while performing transformations/translations of large payloads:

  • ora:doTranslateFromNative or med:doTranslateFromNative

  • ora:doTranslateToNative or med:doTranslateToNative

  • ora:doStreamingTranslate or med:doStreamingTranslate

For more information about these functions, see XPath Extension Functions and Understanding Technology Adapters.

51.1.4.7 Processing Large Documents in Oracle B2B

For processing large documents in Oracle B2B, tune the following parameters:

  • mdsCache

  • Cache Size

  • Protocol Message Size

  • Number of threads

  • Stuck Thread Max Time

  • Tablespace

  • Large payload size

The following sections describe the parameters you must set for processing large documents in Oracle B2B. For more information, see Section "Using Document Streams to Handle Large Payloads" of Using Oracle B2B.

51.1.4.7.1 MDSInstance Cache Size

To set the Oracle Metadata Services (MDS) Repository instance cache size, use Oracle Enterprise Manager Fusion Middleware Control. This property depends on the size of the metadata. Specify a value based on the metadata/endpoint count. The default value is 100000. For information, see Section "Setting B2B Configuration Properties in Fusion Middleware Control" of Using Oracle B2B.

51.1.4.7.2 Protocol Message Size

If Oracle B2B wants to send or receive more than 10 MB of message or the import/export configuration is more than 10 MB, then change the following setting accordingly at the Oracle WebLogic Server Administration Console:

To configure the protocol message size:

  1. In the Domain Structure, select Environment > Servers.
  2. In the Name column of the table, select soa_server.
  3. Select the Protocols tab.
  4. Change the value for Maximum Message Size.

This setting can also be added/modified in the $DOMAIN_HOME/config/config.xml file next to the server name configuration, as shown in the following example: 

<name>soa_server1</name>
<max-message-size>150000000</max-message-size>

Note:

By default, max-message-size is not available in the config.xml file.

51.1.4.7.3 Number of Threads

This parameter improves the message processing capability of Oracle B2B and must be set in the Oracle Enterprise Manager Fusion Middleware Control. For more information, see Section "Configuring Oracle B2B Server Properties" of Administering Oracle SOA Suite and Oracle Business Process Management Suite.

  • b2b.inboundThreadCount

  • b2b.inboundSleepTime

  • b2b.outboundThreadCount

  • b2b.outboundSleepTime

  • b2b.defaultThreadCount

51.1.4.7.4 Stuck Thread Max Time Parameter

The Stuck Thread Max Time parameter checks the number of seconds that a thread must continually work before the server considers the thread stuck. You must change the following setting in the Oracle WebLogic Server Administration Console:

To configure the Stuck Thread Max Time parameter:

  1. In the Domain Structure, select Environment > Servers.
  2. In the Name column of the table, select soa_server.
  3. Select the Tuning tab.
  4. Change the value for Stuck Thread Max Time.
51.1.4.7.5 Tablespace

If you must store more than a 150 MB configuration in the data file, then you must extend or add the data file to increase the tablespace size, as shown in the following example:

ALTER TABLESPACE sh_mds add DATAFILE 'sh_mds01.DBF' SIZE 100M autoextend on next
 10M maxsize unlimited;
ALTER TABLESPACE sh_ias_temp add TEMPFILE 'sh_ias_temp01.DBF' SIZE 100M autoextend
 on next 10M maxsize unlimited;
51.1.4.8 Setting a Size Restriction on Inbound Web Service Message Size

If you want to set a size restriction on inbound web service message size, configure the binding component property max-message-size in the composite.xml file. The property value is made available to the underlying web service infrastructure, which uses the value to test against the incoming message size. If the value specified is exceeded, an exception is thrown indicating that the message size is too large and the transaction is not processed. The following example provides details: 

<composite name="LrgMsg" revision="1.0" label="2011-09-08_22-53-53_259"
 mode="active" state="on">
  <import namespace="http://xmlns.oracle.com/LargeMsg/LrgMsg/BPELProcess1"
   location="BPELProcess1.wsdl" importType="wsdl"/>
  <service name="bpelprocess1_client_ep" ui:wsdlLocation="BPELProcess1.wsdl">
     <interface.wsdl
 interface="http://xmlns.oracle.com/LargeMsg/LrgMsg/BPELProcess1# wsdl.interface
(BPELProcess1)"/>

<binding.ws port="http://xmlns.oracle.com/LargeMsg/LrgMsg/BPELProcess1
    #wsdl.endpoint(bpelprocess1_client_ep/BPELProcess1_pt)">
  <property name="max-message-size" type="xs:integer" many="false"
    override="may">4</property>
</binding.ws>
</service>

  <component name="BPELProcess1" version="1.1">
    <implementation.bpel src="BPELProcess1.bpel"/>
  </component>

  <wire>
    <source.uri>bpelprocess1_client_ep</source.uri>
    <target.uri>BPELProcess1/bpelprocess1_client</target.uri>
  </wire>
</composite>
51.1.4.9 Using XPath Functions to Write Large XSLT/XQuery Output to a File System

You can use the following functions to write the results of large XSLT/XQuery operations to a temporary file in a directory system. The document is then loaded from the temporary file when needed. This eliminates the need for caching an entire document as binary XML in memory.

  • ora:processXSLT

  • ora:doXSLTransformForDoc

With the ora:processXSLT function, you use the properties argument to enable this functionality. 

ora:processXSLT('template','input','properties'?)

You retrieve the value of this argument within your XSLT in a way similar to extracting data from XSL variables. The properties argument is an XML element of the structure shown in the example that follows. For large payload results (for example, above 10 MB), set streamResultToTempFile to yes. For small payload results in which you do not need to write results to a temporary file, leave this property set to its default value of no. 

<propertiesXMLVar>
  <common:item  xmlns:common="http://schemas.oracle.com/service/bpel/common">
    <common:name>streamResultToTempFile</common:name>
    <common:value>yes</common:value>
  </common:item>
</propertiesXMLVar>

Within the XSLT, the parameters are accessible through the name of streamResultToTempFile and its value of yes.

In Oracle BPEL Process Manager, a literal assign is performed to populate the properties for ora:processXSLT('template','input','properties'?).

For more information about using this function, see processXSLT.

With the ora:doXSLTransformForDoc function, you set the name and value properties to enable this functionality. 

ora:doXSLTransformForDoc('template','input','name', 'value')

With this function, the name of streamResultToTempFile and the value of yes are passed.

For more information about using the function, see doXSLTransformForDoc.

51.2 Best Practices for Handling Large Metadata

This section provides recommendations for handling large metadata.

51.2.1 Boundary on the Processing of Large Numbers of Activities in a BPEL Process

There is a limit to the number of activities that can be executed in a BPEL process. When you exceed this limit, system memory fills up, which can cause timeouts to occur. For example, with the following parameters, two fault instances occur due to a timeout:

  • 100 threads

  • 1 second of think time

  • 1000 incoming request messages

Keep the number of incoming request messages at a proper level to ensure system memory stability.

51.2.2 Using Large Numbers of Activities in BPEL Processes (Without FlowN)

To deploy BPEL processes that have a large number of activities (for example, 50,000), the following settings are required: 

MEM_ARGS: -Xms512m -Xmx1024m -XX:PermSize = 128m -XX:MaxPermSize = 256m
Number of Concurrent Threads = 20
Number of Loops = 5 Delay = 100 ms

The above settings enable you to deploy and execute BPEL processes, which use only while loops without the flowN activities, successfully.

51.2.3 Using Large Numbers of Activities in BPEL Processes (With FlowN)

To deploy BPEL processes that have a large number of activities (for example, 50,000), the following settings are required:

USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=256m
Number of Concurrent Threads= 10
Number of Loops=5 Delay=100 ms

Set the StatsLastN property to -1 in the System MBean Browser of Oracle Enterprise Manager Fusion Middleware Control.

The above settings enable you to deploy and execute BPEL processes, which use the flowN activities, successfully.

For more information, see Customizing the Number of Flow Activities with the flowN Activity in BPEL 1.1 and Section "Configuring BPEL Process Service Engine Properties" of Administering Oracle SOA Suite and Oracle Business Process Management Suite.

51.2.4 Using a Flow With Multiple Sequences

BPEL processes that have up to 7000 activities can be deployed and executed successfully with the following settings:

USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=256m

Note:

If you deploy BPEL processes with more than 8000 activities, Oracle BPEL Process Manager compilation throws errors.

51.2.5 Using a Flow with One Sequence

BPEL processes that have up to 7000 activities can be deployed and executed successfully with the following settings:

USER_MEM_ARGS: -Xms2048m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=512m

Note:

If you deploy BPEL processes with more than 10,000 activities, the process compilation fails.

51.2.6 Using a Flow with No Sequence

You can deploy and execute BPEL processes that have a large number of activities (for example, up to 5000) successfully.

There is a probability that the BPEL process compilation may fail for 6000 activities.

51.2.7 Large Numbers of Oracle Mediators in a Composite

Oracle recommends that you not have more than 50 Oracle Mediators in a single composite. Increase the JTA Transaction timeout to a high value based on the environment.

51.2.8 Importing Large Data Sets in Oracle B2B

Oracle recommends that you do not use browsers for large data set imports, and that you use the command line utility. The following utility commands are recommended for large data configuration:

  • purge: Purges the entire repository.

  • import: Imports the specified ZIP file.

  • deploy: Deploys an agreement with whichever name is specified. If no name is specified, then all the agreements are deployed.

However, the purgeimportdeploy option is not recommended for transferring or deploying the Oracle B2B configuration.

For more information, see Using Oracle B2B.

51.3 Best Practices for Handling Large Numbers of Instances

This section provides recommendations for handling large numbers of instance and fault metrics.

51.3.1 Instance and Rejected Message Deletion with the Purge Script or Oracle Enterprise Manager Fusion Middleware Control

You can delete thousands of instances and rejected messages with the PL/SQL purge script or from the Auto Purge page in Oracle Enterprise Manager Fusion Middleware Control.

For more information, see Administering Oracle SOA Suite and Oracle Business Process Management Suite.