A Handling Large Payloads

This appendix describes how to use Oracle B2B to handle large payloads by using the SOA Infrastructure and JMS internal queues.

The appendix includes the following sections:

• Handling Large Payloads

• Using Document Streams to Handle Large Payloads

A.1 Handling Large Payloads

Oracle B2B can handle large payloads through the SOA Infrastructure and JMS internal queues. Examples of large payloads that B2b transactions can handle include a large batch file containing multiple purchase orders.

Note, however, that the definition of a large payload is user-defined and varies by customer and use case. Anything that is in the payload that is above a large payload is not stored in the database but in the file system and specified as part of the large payload configuration. This helps save database table space and optimizes runtime performance.

Additionally, with the Document Streaming feature available in 12.1.3, the document is processed through a stream rather than loading the whole payload into memory, if the document channel and type meet the streaming criteria. Please see the streaming documentation below for more details.

A.1.1 Introduction to Large Payload Support

Inbound Setup

Figure A-1 shows the properties to set for inbound cases. Go to Administration > Configuration.

Figure A-1 Large Payload Size

Description of "Figure A-1 Large Payload Size"

If a composite is deployed to handle the large payload, this is the only configuration needed. If B2B is not delivering the payload to a composite, set Use JMS Queue as default to true, as shown in Figure A-2. Go to Administration > Configuration.

Figure A-2 Use JMS Queue

Description of "Figure A-2 Use JMS Queue"

With Use JMS Queue as default set to true, the payload is delivered to B2B_IN_QUEUE, a JMS-based queue.

Outbound Setup

Figure A-3 shows the properties to set for the outbound case.

Figure A-3 Large Payload Directory

Description of "Figure A-3 Large Payload Directory"

A.2 Using Document Streams to Handle Large Payloads

Oracle B2B enables you to handle large payloads (whose size is specified by the user under Configuration settings) by using streams instead of loading them in-memory.

Earlier, when you specified the large payload size, Oracle B2B, instead of persisting the payload in the database, the payload is placed in the large payload directory. However, prior to streams processing being available, Oracle B2B would load the payload into memory to process the payload, which, for very large payloads, can cause the JVM to run out of Heap Space.

To avoid running out of JVM memory space in the heap for large payloads, Oracle B2B now supports stream-based processing of payloads.

Currently, the document types that are supported for stream-based processing are

• EDI X12

• EDI_EDIFACT

• HL7

Currently, the transport protocols supported for stream-based processing are

• File

• FTP

• SFTP

• MFT

Stream-based processing is particularly useful in the case of batching, where you have say, 10,000 payloads zipped in a 1 Gigabyte batch. Once the payloads are debatched, they put a good deal of load on the system.

A.2.1 How Streaming Works

From the transport perspective, File, FTP, SFTP, and MFT are the only protocols that are enabled to process the input payloads through streams once the large payload threshold is exceeded. This means if a payload received by the system through any of the preceding transports exceeds the large payload size, then the payload is not loaded in-memory; instead, the payload is sent to the file system (stored in the large payload directory). With HTTP transport, the payload is first loaded on to the memory, but during the time of persistence, the payload size is checked, and if the size is more than the large payload size, the payload is persisted in the file system and not in the database, but the payload will not be able to be processed with streams.

When a large payload is received, the Oracle B2B engine stores an input stream reference to the file, and does not load it to the memory. In the identification layer (where the trading partners or document types are identified), Oracle B2B identifies a payload as a large payload and it pushes the payload on to a stream rather than on to memory. In the processing layer, the same check is performed to identify whether a payload is a large payload, and Oracle B2B processes the payload as a stream.

However, if the payload is a non-EDI payload, such as an OAG payload, it will be loaded on to the memory. However, for EDI payloads, the payload is read from an input stream and is passed to an output stream during processing. The output stream is stored in a directory called the Processing directory. Typically, Oracle B2B creates a processing directory inside the large payload directory itself.

For example, Oracle B2B receives a 1 GB payload containing 10,000 batched messages (payloads) over a File channel, and the large payload size is specified as 2 MB. The File listening channel reads the payload and copies it to the large payload directory and creates a wire message that references the payload.

After the processing, the payload is then inserted into the business message and the application message, and is delivered to the back end; not all the 10,000 messages will be greater than 2 MB.

When the payload is inserted into the business message, a check is performed to identify if the payload is a large payload. If the payload is a large payload, then the payload file is copied from the processing directory, and an entry is created in the large payload directory. A reference to that entry is provided to the business message with a header identifying it as a large payload (large payload = true).

When you click that link reference in the business message, the payload is loaded from the respective location. Payloads that are lesser than the large payload size are copied on to the memory and then moved to the database. However, it may so happen that a payload whose size is less than the large payload size, after getting converted to XML, the size increases to more than the large payload size. In this case, the payload is not persisted in the database, and a file reference is passed on to the business message.

When you use streaming, B2B reads one transaction at a time and process it in memory until completion before loading the next transaction into memory. This enables the B2N engine to not run out of memory when processing EDI files that have a large number of transactions that is a large payload.

Taking the preceding example of 10,000 messages of less than 2 MB each, however, it is not possible to process the entire payload of 10,000 messages in a single transaction. You have to specify a partial commit size for the batch (say 200 messages) accordingly so that every time the partial commit size is reached, those 200 messages are committed to the database.

The limitation, if you try to process all the messages in a single transaction even if the JTA is set to a high value, is Oracle B2B will have to load all the 10000 into memory during delivery and persistence because they are small payloads, eventually causing a memory overflow.

A.2.1.1 When Does Streaming Not Occur

Streaming does not occur in the following circumstances:

• With HTTP transport, the payload is first loaded on to the memory, but during the time of persistence, the payload size is checked, and if the size is more than the large payload size, then the payload is persisted in the file system and not in the database, but the payload will not be able to be processed with streams.

• In the processing layer, when the check is performed to identify whether a payload is a large payload, and Oracle B2B processes the payload as a stream. In case of non-EDI/X12/HL7 payload, such as an OAG payload, it will be loaded on to the memory.

A.2.1.2 Using Document Streams from the Back End

You must also notify B2B that a service engine is sending a large payload. The change involves two steps:

The b2b.largePayload property must be set in the BPEL process when sending a large payload to Oracle B2B. For composite samples, which do not handle large payload, there is no change.

Note:

The behavior of this part of the setup from earlier releases is different. Earlier this directory was a plain directory, but from 12.1.3.0 is a treated as a stream store. The implication is that if you change the directory mid way during the processing, the application would previously work fine because the application maintained the entire file reference. However, these internally have been abstracted and the application now only maintains a Stream store id.

You need to migrate all payloads from 1 dir to another if there needs to be a change in location. You need to restart the server for the change in stream store location to take effect.

Code change in Oracle B2B to handle this flag.

1. Declare the Variable_largePayload variable in an outbound BPEL process in the <variables> section.

   <variable name="Variable_largePayload" type="xsd:string"/>
   

2. In the Assign activity, copy 'true' into the variable.

   <copy>
    <from expression="'true'"/>
    <to variable="Variable_largePayload"/>
   </copy>
   

3. Assign the variable to b2b.largePayload in the Invoke activity.

   <bpelx:inputProperty name="b2b.largePayload"
    variable="Variable_largePayload"/>
   

Note:

If BPEL is not sending a large payload to Oracle B2B, this property should not be set.

after the code is checked in, any Large Payload Sample must be updated to confirm to this.

In BPEL and Mediator, if b2b.largePayload is set to true, then largePayloadDir must be present (set it in Oracle B2B). If b2b.largePayload is not set, then this directory should not matter.

Oracle B2B retains the large payloads in the large payload processing directory, after sending the payload to corresponding endpoints.

About Large Payload Support

1. If you are doing large payload testing, set Log Payload on the Administration > Configuration tab to false.
2. If you are doing large payload testing, set Show Payload on the Administration > Configuration tab to false to avoid listing the payload in reports.
3. If an enqueue script is used when working with large payloads, add

   eventName=LARGE_PAYLOAD=true
   

4. Increase the maximum heap size to use -Xmx2048m.
5. Increase the database tablespace size for soadatasource to have autoextend on and increase the tablespace file size maximum limit.

   alter database datafile '/scratch/$user/auto_work/db230/oradata/db230/SH_soainfra.dbf' autoextend on next 10M maxsize 4096M

6. Set the transaction timeout in Oracle WebLogic Administration Server:

   • Weblogic Console Services -> JTA Timeout Seconds=720 seconds

   • Weblogic Console Services -> JDBC->DataSources->SOADataSource - increase XA timeout to 120-180 seconds

7. If Oracle B2B is used alone (without the SOA Infrastructure), the JTA timeout can be set in b2b.jtaTimeout by using Oracle Enterprise Manager Fusion Middleware Control. See Oracle Fusion Middleware Administrator's Guide for Oracle SOA Suite and Oracle Business Process Management Suite for more information.
8. For an outbound SOA composite, always select the Use file streaming option for the File Adapter, as shown in Figure A-4.

   Figure A-4 The File Adapter Use File Streaming Option

   
   Description of "Figure A-4 The File Adapter Use File Streaming Option"

A.2.1.3 Settings for a Large Dataset Scenario

The following suggested settings are based on a dataset with approximately 2,500 trading partners, an export ZIP file that is approximately 253 MB in size, and assumes a 6 GB computer. Using these settings can considerably reduce data upload time when using the Upgrade Assistant.

1. Use Oracle WebLogic Server Administration Console to increase the

   • JTA transaction timeout from 30 to 350

   • Maximum message size from the default size to 200000000

2. Add indices for better performance. Using Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production, with the Partitioning, OLAP, Data Mining and Real Application Testing options, do the following:

   SQL> create index idx_mds_attr on
   rc1_mds.MDS_ATTRIBUTES("ATT_VALUE","ATT_LOCALNAME");
   Index created.
   
   SQL> create index idx_mds_path on
   rc1_mds.MDS_PATHS("PATH_CONTENTID","PATH_PARTITION_ID");
   Index created.
   
   SQL> commit;
   

3. Start the managed server with the following updated memory setting:

   DEFAULT_MEM_ARGS="-Xms1024m -Xmx2048m"
   

4. Change ORACLE_HOME/bin/UA default memory from the default 256 to 2048. The default is

   $JAVA_HOME/bin/java ${JAVAMODE} -Xmx256m -classpath ${CLASSPATH}
   -Dua.home=$base_dir -Dice.pilots.html4.ignoreNonGenericFonts=true
   -Dsun.lang.ClassLoader.allowArraySyntax=true
   -Doracle.installer.oui_loc=$OUI_HOME oracle.ias.upgrade.UpgradeDriver
   $ARGUMENTS
   

   Change the default to

   $JAVA_HOME/bin/java ${JAVAMODE} -Xmx2048m -classpath ${CLASSPATH}
   -Dua.home=$base_dir -Dice.pilots.html4.ignoreNonGenericFonts=true
   -Dsun.lang.ClassLoader.allowArraySyntax=true
   -Doracle.installer.oui_loc=$OUI_HOME oracle.ias.upgrade.UpgradeDriver
   $ARGUMENTS
   

5. Change the value of Stuck Thread Max Time from 600 to 2000.

A.2.2 Limitations

The following limitations apply to payload handling and streaming:

• Only certain documentation types support streaming.

• Only certain transport types are supported.

• Functions such as XPatch extraction, tend to load the XML payloads into memory for extraction currently. (However, the impact tends to be less where there is batching, as the individual payloads are of lesser size and once the extraction is complete, the loaded references are cleared allowing for a immediate garbage collection,)

• Documents greater than 2 Gigabytes are not supported.

• When using HL7 custom delimiters, the native payload is loaded into memory.