Skip Navigation Links | |
Exit Print View | |
Oracle Java CAPS Adapter for Batch User's Guide Java CAPS Documentation |
Oracle Java CAPS Adapter for Batch User's Guide
Additional Licensing Considerations
Batch Adapter System Requirements
Installing Adapter Enterprise Manager plug-ins
Creating and Configuring Batch Adapters
Selecting a Batch External Application
Modifying the Adapter Properties
BatchFTP Adapter Connectivity Map Properties
Pre Transfer (BatchFTP Connectivity Map)
SOCKS (BatchFTP Connectivity Map)
FTP (BatchFTP Connectivity Map)
FTP Raw Commands (BatchFTP Connectivity Map)
Sequence Numbering (BatchFTP Connectivity Map)
Post Transfer (BatchFTP Connectivity Map)
Target Location (BatchFTP Connectivity Map)
SSH Tunneling (BatchFTP Connectivity Map)
Additional SSH-supporting Software
General Settings (BatchFTP Connectivity Map)
BatchFTP Adapter Environment Properties
General Settings (BatchFTP Environment)
SSH Tunneling (BatchFTP Environment)
Connection Pool Settings (BatchFTP Environment)
Connection Retry Settings (BatchFTP Environment)
BatchFTPOverSSL Adapter Connectivity Map Properties
Pre Transfer (BatchFTPOverSSL Connectivity Map)
FTP and SSL Settings (BatchFTPOverSSL Connectivity Map)
Post Transfer (BatchFTPOverSSL Connectivity Map)
Firewall Settings (BatchFTPOverSSL Connectivity Map)
Synchronization (BatchFTPOverSSL Connectivity Map)
BatchFTPOverSSL Adapter Environment Properties
FTP and SSL Settings (BatchFTPOverSSL Environment)
Firewall Settings (BatchFTPOverSSL Environment)
General Settings (BatchFTPOverSSL Environment)
Connection Pool Settings (BatchFTPOverSSL Environment)
Connection Retry Settings (BatchFTPOverSSL Environment)
BatchSCP Adapter Connectivity Map Properties
SCP Settings (BatchSCP Connectivity Map)
Firewall Settings (BatchSCP Connectivity Map)
Synchronization (BatchSCP Connectivity Map)
BatchSCP Adapter Environment Properties
SSH Settings (BatchSCP Environment)
Firewall Settings (BatchSCP Environment)
General Settings (BatchSCP Environment)
Connection Pool Settings (BatchSCP Environment)
Connection Retry Settings (BatchSCP Environment)
BatchSFTP Adapter Connectivity Map Properties
Pre Transfer (BatchSFTP Connectivity Map)
SFTP Settings (BatchSFTP Connectivity Map)
Post Transfer (BatchSFTP Connectivity Map)
Firewall Settings (BatchSFTP Connectivity Map)
Synchronization (BatchSFTP Connectivity Map)
BatchSFTP Adapter Environment Properties
SFTP Settings (BatchSFTP Environment)
Firewall Settings (BatchSFTP Environment)
General Settings (BatchSFTP Environment)
Connection Pool Settings (BatchSFTP Environment)
Connection Retry Settings (BatchSFTP Environment)
BatchLocalFile Connectivity Map Properties
Pre Transfer (BatchLocalFile Connectivity Map)
Sequence Numbering (BatchLocalFile Connectivity Map)
Post Transfer (BatchLocalFile Connectivity Map)
General Settings (BatchLocalFile Connectivity Map)
Target Location (BatchLocalFile Connectivity Map)
BatchLocalFile Environment Properties
General Settings (BatchLocalFile Environment)
Connection Pool Settings (BatchLocalFile Environment)
BatchRecord Connectivity Map Properties
General Settings (BatchRecord Connectivity Map)
Record (BatchRecord Connectivity Map)
BatchRecord Environment Properties
Connection Pool Settings (BatchRecord Environment)
BatchInbound Connectivity Map Properties
Settings (BatchInbound Connectivity Map)
BatchInbound Environment Properties
MDB Settings (BatchInbound Environment)
IBM IP Stack Required for MVS Sequential, MVS GDG, and MVS PD
Creating User Defined Heuristic Directory Listing Styles
Heuristics Configuration File Format
FTP Heuristics Configuration Parameters
Commands Supported by FTP Server
Header Indication Regex Expression
Trailer Indication Regex Expression
Directory Indication Regex Expression
File Link Indication Regex Expression
File Link Symbol Regex Expression
Valid File Line Minimum Position
Special Envelope For Absolute Path Name
Listing Directory Yields Absolute Path Names
Absolute Path Name Delimiter Set
Change Directory Before Listing
Directory Name Requires Terminator
FTP Configuration Requirements for AS400 UNIX (UFS)
Dynamic Configurable Parameters for Secure FTP OTDs
Configuration Parameters that Accept Integer Values
Understanding Batch Adapter OTDs
Code Conversion and Generation
Type Conversion Troubleshooting
Essential BatchFTP OTD Methods
Additional FTP File Transfer Commands
BatchFTPOverSSL OTD Node Functions
BatchLocalFile OTD Node Functions
BatchLocalFile Specific Features
Pre/Post File Transfer Commands
Essential BatchLocalFile OTD Methods
Operation Without Resume Reading Enabled
To Avoid Storing a Resume Reading State
Generating Multiple Files with Sequence Numbering
Example 1: Parsing a Large File
Example 2: Slow, Complex Query
Record-processing OTD Node Functions
Using the Record-processing OTD
Choosing the Parse or Create Mode
Using Record Processing with Data Streaming
Regular Expressions and the Adapter
Rules for Directory Regular Expressions
Restrictions for Using Regular Expressions as Directory Names
Regular Expression Directory Name Examples
Additional Batch Adapter Features
Streaming Data Between Components
Introduction to Data Streaming
Overcoming Large-file Limitations
SOCKS Configuration Properties
Additional Software Requirements
Components in the Batch Adapter implement a feature for data streaming. This chapter explains data streaming, how it works, and how to use it with the adapter and Java CAPS. it includes the following sections:
Data streaming provides a means for interconnecting any two components of the adapter by means of a data stream channel. This channel provides an alternate way of transferring the data between the Batch Adapter components. Streaming is available between BatchLocalFile and BatchFTP, or BatchLocalFile and BatchRecord.
Each OTD component in the adapter has a Payload node. This node represents the in-memory data and is used when the data is known to be relatively small in size or has already been loaded into memory. The node can represent, for example, the buffer in the record-processing OTD, as it is being built or parsed, or the contents of a file read into memory.
Instead of moving the data all at once between components in memory, you can use a data-stream channel to provide for streaming the data between them a little at a time, outside of Java CAPS.
Data streaming was designed primarily to handle large files, but you can use it for smaller data sizes as well.
Use the Java CAPS Collaboration Rules Editor in NetBeans to set up data-streaming operations. The rest of this section explains the data streaming feature and how to set it up.
Note - Payload-based and streaming-based transfers are mutually exclusive. You can use one or the other but not both for the same data.
The primary advantage of using data streaming is that it helps to overcome the limitations of dealing with large files. For example, if you have a 1-gigabyte file that contains a large number of records, you need a large amount of resources to load the payload into memory just to parse it.
Streaming allows you to read from the file, little by little, using a data-streaming mechanism. This way, you do not need to load the file into the Java CAPS system’s memory. Using streaming is not as fast as using in-memory operations, but it is far less resource-intensive.
Each data-streaming transfer involves two OTDs in a Collaboration as follows:
One provides the stream adapter.
The other consumes the stream adapter to perform the data transfer.
Caution - Implementing InputStream must support skip() with negative numbers as an argument. |
This section explains how the two data-streaming OTDs operate to effect the transfer of data.
Each of the OTDs in the Batch Adapter exposes stream adapter nodes that enable any OTD to participate in data-streaming transfers. The nodes are named InputStreamAdapter and OutputStreamAdapter. You can associate the stream adapters by using the drag-and-drop features of NetBeans.
The InputStreamAdapter (highlighted) and OutputStreamAdapter nodes in the OTD are used for data streaming. This feature operates as follows:
Stream-adapter consumers: The FTP and the record-processing OTDs can only consume stream adapters. Therefore, their stream-adapter nodes are write-only. Their node values can be set (modified).
Stream-adapter provider: The local file OTD can only provide stream adapters, so its stream-adapter nodes are read-only. Its node value can only be retrieved.
The local file OTD is always the stream provider, and the FTP and record-processing OTDs are the consumers.
Note - For an explanation of the adapter’s different types of OTDs, see Understanding Batch Adapter OTDs .
Use of the InputStreamAdapter and OutputStreamAdapter nodes is an alternative to using the Payload node as follows:
Use these stream adapter nodes to transfer data if you want data streaming.
Use the Payload node for a data transfer without data streaming (payload data transfer).
All operations that, in payload data transfer, read from the Payload node require the InputStreamAdapter node when you are setting up data streaming. Using the same logic, all operations that, in payload data transfer, write to the Payload node require OutputStreamAdapter node for data streaming.
Do not confuse the stream adapter nodes with the get() and put() methods on the OTDs. For example, the BatchFTP OTD’s client interface get() method writes to the Payload node during a payload transfer, so it requires an OutputStreamAdapter node to write to for data streaming. In contrast, the record-processing OTD’s get() method reads from the Payload node during a payload transfer, so for data streaming, get() requires an inputStreamAdapter node to read from.
The four typical data-streaming scenarios are:
Transfer data from a local file system (BatchLocalFile) to a record-processing (BatchRecord) setup. This scenario uses the InputStreamAdapter OTD node (see Data Streaming Scenarios).
Transfer data from a record-processing (BatchRecord) setup to a local file system (BatchLocalFile) using the OutputStreamAdapter OTD node (see Data Streaming Scenarios).
Transfer data from a local file system (BatchLocalFile) to a remote FTP (BatchFTP) setup. This scenario uses the InputStreamAdapter OTD node (see Data Streaming Scenarios).
Transfer data from a remote FTP server (BatchFTP) to a local file system (BatchLocalFile) using the OutputStreamAdapter OTD node (see Data Streaming Scenarios).
This section explains how to use consuming-stream adapters.
This section provides the Batch Adapter’s OTD stream-adapter Java interfaces. This information is only for advanced users familiar with Java programming, who want to provide custom OTD implementations for stream-adapter consumers or providers.
The following Java programming-language interface provides support for inbound transfers from an external system:
public interface com.stc.eways.common.eway.streaming.InputStreamAdapter { public java.io.InputStream requestInputStream() throws StreamingException; public void releaseInputStream(boolean success) throwsStreamingException; }
The following Java interface provides support for outbound transfers to an external system:
public interface com.stc.eways.common.eway.streaming.OutputStreamAdapter { public java.io.OutputStream requestOutputStream() throws StreamingException; public void releaseOutputStream(boolean success) throws StreamingException; }