test

Sun Adapter for Batch User's Guide

Streaming Data Between Components

Components in the Batch Adapter implement a feature for data streaming. This chapter explains data streaming, how it works, and how to use it with the adapter and eGate Integrator.

Introduction to Data Streaming

Data streaming provides a means for interconnecting any two components of the adapter by means of a data stream channel. This channel provides an alternate way of transferring the data between the Batch Adapter components. Streaming is available between BatchLocalFile and BatchFTP, or BatchLocalFile and BatchRecord.

Each OTD component in the adapter has a Payload node. This node represents the in-memory data and is used when the data is known to be relatively small in size or has already been loaded into memory. The node can represent, for example, the buffer in the record-processing OTD, as it is being built or parsed, or the contents of a file read into memory.

Instead of moving the data all at once between components in eGate’s memory, you can use a data-stream channel to provide for streaming the data between them a little at a time, outside of eGate.

Data streaming was designed primarily to handle large files, but you can use it for smaller data sizes as well.

Use the eGate Enterprise Designer’s Collaboration Rules Editor to set up data-streaming operations. The rest of this section explains the data streaming feature and how to set it up.

Note –

Payload-based and streaming-based transfers are mutually exclusive. You can use one or the other but not both for the same data.

Overcoming Large-file Limitations

The primary advantage of using data streaming is that it helps to overcome the limitations of dealing with large files. For example, if you have a 1-gigabyte file that contains a large number of records, you need a large amount of resources to load the payload into memory just to parse it.

Streaming allows you to read from the file, little by little, using a data-streaming mechanism. This way, you do not need to load the file into the eGate system’s memory. Using streaming is not as fast as using in-memory operations, but it is far less resource-intensive.

Using Data Streaming

Each data-streaming transfer involves two OTDs in a Collaboration as follows:

This section explains how the two data-streaming OTDs operate to effect the transfer of data.

Data-streaming Operation

Each of the OTDs in the Batch Adapter exposes stream adapter nodes that enable any OTD to participate in data-streaming transfers. The nodes are named InputStreamAdapter and OutputStreamAdapter. You can associate the stream adapters by using the drag-and-drop features of the eGate Enterprise Designer.

The InputStreamAdapter (highlighted) and OutputStreamAdapter nodes in the OTD are used for data streaming. This feature operates as follows:

The local file OTD is always the stream provider, and the FTP and record-processing OTDs are the consumers.

Note –

For an explanation of the adapter’s different types of OTDs, see Understanding Batch Adapter OTDs .

Data Streaming Versus Payload Data Transfer

Use of the InputStreamAdapter and OutputStreamAdapter nodes is an alternative to using the Payload node as follows:

All operations that, in payload data transfer, read from the Payload node require the InputStreamAdapter node when you are setting up data streaming. Using the same logic, all operations that, in payload data transfer, write to the Payload node require OutputStreamAdapter node for data streaming.

Do not confuse the stream adapter nodes with the get() and put() methods on the OTDs. For example, the BatchFTP OTD’s client interface get() method writes to the Payload node during a payload transfer, so it requires an OutputStreamAdapter node to write to for data streaming. In contrast, the record-processing OTD’s get() method reads from the Payload node during a payload transfer, so for data streaming, get() requires an inputStreamAdapter node to read from.

Data Streaming Scenarios

The four typical data-streaming scenarios are:

Consuming-stream Adapters

This section explains how to use consuming-stream adapters.

ProcedureTo obtain a stream

  1. Use the requestXXStream() method to obtain the corresponding XX stream.

ProcedureTo Use a Stream

  1. Perform the transfer using the methods provided by the stream.

ProcedureTo Dispose of a Stream

  1. Release any references to the stream.

  2. Release the stream (XX) using the releaseXXStream() method. Some of the OTDs support post-transfer commands. The Success parameter indicates whether these commands are executed. Do not close the stream.

Stream-adapter Interfaces

This section provides the Batch Adapter’s OTD stream-adapter Java interfaces. This information is only for advanced users familiar with Java programming, who want to provide custom OTD implementations for stream-adapter consumers or providers.

Inbound Transfers

The following Java programming-language interface provides support for inbound transfers from an external system:


public interface com.stc.eways.common.eway.streaming.InputStreamAdapter {
public java.io.InputStream requestInputStream() throws StreamingException;
public void releaseInputStream(boolean success) throwsStreamingException;
}

Outbound Transfers

The following Java interface provides support for outbound transfers to an external system:


public interface com.stc.eways.common.eway.streaming.OutputStreamAdapter {
public java.io.OutputStream requestOutputStream() throws StreamingException;
public void releaseOutputStream(boolean success) throws StreamingException;
}