Streaming Data Between Components with the Batch Adapter

Components in the Batch Adapter implement a feature for data streaming. This topic explains data streaming, how it works, and how to use it with the adapter and Java CAPS.

• Introduction to Data Streaming

• Overcoming Large-file Limitations

• Using Data Streaming

• Stream-adapter Interfaces

Introduction to Data Streaming

Data streaming provides a means for interconnecting any two components of the adapter by means of a data stream channel. This channel provides an alternate way of transferring the data between the Batch Adapter components. Streaming is available between BatchLocalFile and BatchFTP, or BatchLocalFile and BatchRecord.

Each OTD component in the adapter has a Payload node. This node represents the in-memory data and is used when the data is known to be relatively small in size or has already been loaded into memory. The node can represent, for example, the buffer in the record-processing OTD, as it is being built or parsed, or the contents of a file read into memory.

Instead of moving the data all at once between components in Java CAPS’s memory, you can use a data-stream channel to provide for streaming the data between them a little at a time, outside of Java CAPS. Data streaming was designed primarily to handle large files, but you can use it for smaller data sizes as well. Use the NetBeans IDE’s Collaboration Rules Editor to set up data-streaming operations. The rest of this section explains the data streaming feature and how to set it up.

Overcoming Large-file Limitations

The primary advantage of using data streaming is that it helps to overcome the limitations of dealing with large files. For example, if you have a 1-gigabyte file that contains a large number of records, you need a large amount of resources to load the payload into memory just to parse it.

Streaming allows you to read from the file, little by little, using a data-streaming mechanism. This way, you do not need to load the file into the Java CAPS system’s memory. Using streaming is not as fast as using in-memory operations, but it is far less resource-intensive.

Using Data Streaming

Each data-streaming transfer involves two OTDs in a Collaboration as follows:

• One provides the stream adapter.

• The other consumes the stream adapter to perform the data transfer.

  ---

  Caution - Implementing InputStream must support skip() with negative numbers as an argument.

  ---

This section explains how the two data-streaming OTDs operate to effect the transfer of data.

Data-streaming Operation

Each of the OTDs in the Batch Adapter exposes stream adapter nodes that enable any OTD to participate in data-streaming transfers. The nodes are named InputStreamAdapter and OutputStreamAdapter. You can associate the stream adapters by using the drag-and-drop features of the Java CAPS IDE.

The InputStreamAdapter and OutputStreamAdapter nodes in the OTD are used for data streaming. This feature operates as follows:

• Stream-adapter consumers: The FTP and the record-processing OTDs can only consume stream adapters. Therefore, their stream-adapter nodes are write-only. Their node values can be set (modified).

• Stream-adapter provider: The local file OTD can only provide stream adapters, so its stream-adapter nodes are read-only. Its node value can only be retrieved.

The local file OTD is always the stream provider, and the FTP and record-processing OTDs are the consumers.

Data Streaming Versus Payload Data Transfer

Use of the InputStreamAdapter and OutputStreamAdapter nodes is an alternative to using the Payload node as follows:

• Use these stream adapter nodes to transfer data if you want data streaming.

• Use the Payload node for a data transfer without data streaming (payload data transfer).

All operations that, in payload data transfer, read from the Payload node require the InputStreamAdapter node when you are setting up data streaming. Using the same logic, all operations that, in payload data transfer, write to the Payload node require OutputStreamAdapter node for data streaming.

Do not confuse the stream adapter nodes with the get() and put() methods on the OTDs. For example, the BatchFTP OTD’s client interface get() method writes to the Payload node during a payload transfer, so it requires an OutputStreamAdapter node to write to for data streaming. In contrast, the record-processing OTD’s get() method reads from the Payload node during a payload transfer, so for data streaming, get() requires an inputStreamAdapter node to read from.

Data Streaming Scenarios

The four typical data-streaming scenarios are:

• Transfer data from a local file system (BatchLocalFile) to a record-processing (BatchRecord) setup. This scenario uses the InputStreamAdapter OTD node (see Data Streaming Scenarios).

  

• Transfer data from a record-processing (BatchRecord) setup to a local file system (BatchLocalFile) using the OutputStreamAdapter OTD node (see Data Streaming Scenarios).

  

• Transfer data from a local file system (BatchLocalFile) to a remote FTP (BatchFTP) setup. This scenario uses the InputStreamAdapter OTD node (see Data Streaming Scenarios).

  

• Transfer data from a remote FTP server (BatchFTP) to a local file system (BatchLocalFile) using the OutputStreamAdapter OTD node (see Data Streaming Scenarios).

  

Consuming-stream Adapters

This section explains how to use consuming-stream adapters.

To Obtain a Stream

• Use the requestXXStream() method to obtain the corresponding XX stream.

To Use a Stream

• Perform the transfer using the methods provided by the stream.

To Dispose of a Stream

1. Release any references to the stream.
2. Release the stream (XX) using the releaseXXStream() method. Some of the OTDs support post-transfer commands. The Success parameter indicates whether these commands are executed. Do not close the stream.

Stream-adapter Interfaces

This section provides the Batch Adapter’s OTD stream-adapter Java interfaces. This information is only for advanced users familiar with Java programming, who want to provide custom OTD implementations for stream-adapter consumers or providers.

Inbound Transfers

The following Java programming-language interface provides support for inbound transfers from an external system:

public interface com.stc.adapters.common.adapter.streaming.InputStreamAdapter {
public java.io.InputStream requestInputStream() throws StreamingException;
public void releaseInputStream(boolean success) throwsStreamingException;
}

Outbound Transfers

The following Java interface provides support for outbound transfers to an external system:

public interface com.stc.adapters.common.adapter.streaming.OutputStreamAdapter {
public java.io.OutputStream requestOutputStream() throws StreamingException;
public void releaseOutputStream(boolean success) throws StreamingException;
}