An OTD contains a set of rules that define an object. The object encodes data as it travels through OTDs which are used as the basis for creating Collaboration Definitions for a Project.
Each OTD acts as a template with a unique set of features of the adapter. The Batch Adapter OTD template is not customizable and cannot be edited.
The four parts of an OTD are:
Element: An element is the highest level in the OTD tree. The element is the basic container that holds the other parts of the OTD. The element can contain fields and methods.
Field: Fields are used to represent data. A field can contain data in any of the following formats: string, boolean, int, double, or float.
Method: Method nodes represent actual Java methods.
Parameter: Parameter nodes represent the Java methods’ parameters.
A high-level view of the OTD folder structure shows methods and attributes you can use in creating Business Rules that invoke FTP, secure FTP, batch record, or local file data exchange.
The figure shows the specialized OTDs available with the adapter.
Table 1–1 Batch Adapter OTDs
OTD Name |
Description |
---|---|
BatchFTP |
Provides FTP access to remote systems. |
BatchFTPOverSSL |
Provides secure data transfer using FTP over SSL. |
BatchSCP |
Provides secure data transfer using Secure Copy Protocol with Secure Shell (SSH) as an underlying protocol. |
BatchSFTP |
Provides secure data transfer using SSH File Transfer Protocol or SFTP protocol. |
BatchLocalFile |
Provides easy access to local file systems. |
BatchRecord |
Allows the adapter to parse or create (or both) payloads of records in specified formats. |
BatchInbound |
Polls for input file, renames the file to a GUID, and triggers the Business Process or Collaboration. |
This chapter describes each of these OTDs and how to use them with the adapter.
OTDs provide the following functions:
OTD are used in Collaborations to provide platform, system, and program specific functionality that allow you to create Business Rules.
OTDs contain system specific parameters that can be configured using the Properties Editor.
OTDs provide access to the information required to interface with specific external application.
All OTDs must be configured and administered using NetBeans IDE. Any client components relevant to these OTDs have their own requirements. See the client system’s documentation for details.
For the BatchFTP, BatchLocalFile, and BatchRecord OTDs, only those nodes for which there is a corresponding section in the Environment or Connectivity Map properties (From the Properties Editor) are implemented on the OTD. The remaining nodes are not implemented and are reserved for potential future use.
For the BatchFTP, BatchLocalFile, and BatchRecord OTDs, only those configuration parameters which appear in the Environment or Connectivity Map properties (From the Properties Editor) are supported for use on the OTD. The remaining configuration parameters are not implemented, and are reserved for potential future use. Even though an implemented configuration parameter might be accessed and used from a non-implemented node, such use is not recommended.
The Batch Adapter includes four OTDs that allow you to perform FTP data-transfer functions. The BatchFTP OTD enables the Sun Enterprise Service Bus system to exchange data with other network hosts for the purpose of receiving and delivering objects stored in files. In addition, the BatchFTPOverSSL, BatchSCP, and BatchSFTP OTDs enable secure data transfer between the local host and a remote host using FTP over SSL and SSH.
The BatchFTP OTD contains three top-level nodes, Client, Configuration, Provider, State, and StateManager. Expand these nodes to reveal additional sub-nodes.
Each field sub-node in the Configuration node of the OTD corresponds to one of the adapter’s FTP configuration parameters.
This OTD includes two additional top-level nodes, the Client and Provider. These nodes implement their respective functionality interfaces in the adapter.
The client interface represents how the functionality of the provider is actually used.
The provider interface represents all of the general FTP operations that can be performed by the OTD.
The following list provides an explanation of various nodes in the BatchFTP OTD, including primary functions:
BatchFTP: Represents the OTD’s root node.
Configuration: Each field sub-node within this node corresponds to an adapter configuration parameter and contains settings information.
InputStreamAdapter and OutputStreamAdapter: Allow you to use and control the OTD’s data-streaming features.
This OTD has configuration parameters that can be regular expressions. See Using Regular Expressions With the Batch Adapter for details.
Client: This node contains the following sub-nodes, which implement the adapter’s client interface in the OTD (FtpFileClient):
Payload: An in-memory buffer that contains the payload or message data you want to transfer by FTP, in the form of a byte array.
UserProperties: Only used if you have provided a user-defined implementation of the FtpFileClient interface; in such cases, the node represents the properties specified in the configuration.
You can transfer data using the Payload node or by using data streaming (InputStreamAdapter and OutputStreamAdapter nodes), but you cannot use both methods in the same OTD.
ResolvedNamesForGet and ResolvedNamesForPut: Allow you to get the real file or directory name used during a transfer and perform an operation with it. For example, you could do a file transfer, with get() or put(), using the real name. You are able to retrieve the real file or directory name, even if these names have been expressed using regular expressions or special characters.
These nodes contain sub-nodes that allow you to resolve file and directory names for target destinations, as well as names for pre- and post-transfer commands. See the Pre/Post File Transfer Commands, Resolving Names for more information on these nodes. Also see Using Regular Expressions With the Batch Adapter for more information on regular expressions.
get(), put(), reset(), connect(), disconnect(), and isConnected(): See Essential BatchFTP OTD Methods.
Provider: The sub-nodes contained in this node are methods that implement the adapter’s provider interface in this OTD (FtpFileProvider). These methods allow you to do the general FTP operations that can be performed using the OTD.
The BatchFTP OTD nodes allow you to configure specific adapter configuration parameters for the Collaboration controlling the FTP process. Once you have set the configuration parameters, you do not have to define the same parameters in each corresponding adapter component that uses this Collaboration.
The Payload node in the BatchFTP OTD is predefined as a byte array (byte[]). This definition allows the adapter to handle both binary and character data.
For example, you could use another OTD (such as an OTD from another adapter or a user-defined OTD) where the “data” node has been defined as a String (java.lang.String). If you were to map that String to the BatchFTP OTD’s Payload node, the Collaboration Editor can do an automatic type conversion and create code similar to that shown in the following example.
You must use care with this feature. While it works in many situations, there can be occasions when the default encoding causes errors in the translation.
For example, in a string-to-byte array conversion (or vice versa), the generated Java code could be:
getoutput().setPayload(STCTypeConverter.toByteArray (getinput().getBlob())); |
or
getinput().setBlob(STCTypeConverter.toString (getoutput().getPayload())); |
If you define the blob data as a byte array, no type conversion is necessary. When there is a conversion, the Collaboration Editor uses the Java Virtual Machine (JVM) default encoding to do the conversion to code, as shown in the previous examples.
As explained previously, the default encoding and translation works for many situations. There are cases, however (for example, binary data such as a .zip file), when the encoding could cause errors in the translation. Depending on the data character set and JVM default encoding, you should choose the appropriate encoding. In most cases, using the encoding string “ISO-8859-1” is the best choice.
To use this encoding, you can modify the code manually by adding the encoding String. Taking the previous examples, the resulting code using “ISO-8859-1” is:
getoutput().setPayload(STCTypeConverter.toByteArray (getinput().getBlob(), "ISO-8859-1")); |
or
getinput().setBlob(STCTypeConverter.toString (getoutput().getPayload(), "ISO-8859-1")); |
Using this String solves this type conversion problem. For more information, see the appropriate JVM encoding reference manuals.
In addition to the field elements, the BatchFTP OTD’s Client node contains methods that extend the client interface functionality of the adapter. These methods are essential to the proper use of the OTD and require some additional explanation. They are:
get(): Retrieves a file from the remote FTP server then stores its contents as a data payload. The method retrieves the first matching file based on the Target Directory Name and Target File Name parameters and stores the contents as a data payload (a byte array). It then performs any Post Transfer Command.
After this method call, you can get the payload’s contents through the method getPayload().
If no qualified file is available for retrieving, you get the exception containing java.io.FTPFileException as a nested exception.
put(): Places the payload data on the FTP server, that is, it performs an append or put action from the Payload node to the remote FTP server and performs any Post Transfer Command.
If no qualified file is available for sending, you get the exception containing java.io.FTPFileException as a nested exception.
When you are using the adapter’s data-streaming feature, the get() and put() methods operate differently. See the Streaming Data Between Components section for details on this operation.
reset(): Allows you to return the Client node to its state immediately after the previous initialization.
The reset() method is available in the Batch FTP OTDs and BatchLocalFile OTD. The reset method must be called when the OTD has to be reused for another transfer during the same execution of executeBusinessRules() (for example, when you are using the Dynamic Configuration feature). Thereset() method resets the content of the Client node without resetting the whole OTD. If you attempt another transfer without calling reset() first, the system throws an exception and makes an entry in the adapter’s error log file.
restoreConfigValues(): Allows you to restore the configuration parameter defaults to the related adapter configuration.
connect(), disconnect(), and isConnected(): Perform connection-related operations with respect to the FTP server.
The sequence numbering feature allows you to set up the FTP target directory or file name to contain a sequence number. You can set the starting and maximum sequence numbers using the adapter configuration parameters for the OTD.
This parameter is used for the name pattern %#.
This feature is also available with the BatchLocalFile OTD. For more information on these configuration parameters, see Sequence Numbering (BatchFTP Connectivity Map).
The BatchFTP OTD also allows you to enter commands to be executed directly before and after the file transfer operation. See the Pre/Post File Transfer Commands section for details.
The Batch Secure FTP over SSL OTD (BatchFTPOverSSL) provides secure data transfer using Secure Sockets Layer (SSL) protocol.
For information about configuring external FTP servers, SSL servers, and so forth, refer to the application’s documentation.
The BatchFTPOverSSL OTD contains two top-level nodes, Client and Configuration. Expand these nodes to reveal additional sub-nodes.
The following list provides an explanation of various nodes in the BatchFTPOverSSL OTD, including primary functions:
BatchFTPOverSSL: Represents the OTD’s root node.
The BatchFTPOverSSL sub-nodes under the Configuration node correspond to BatchFTPOverSSL adapter’s Connectivity Map and Environment configuration parameters.
The Client node contains sub-nodes that implement the adapter’s client interface in the OTDs. The BatchFTPOverSSL Client node includes the following methods:
append(): transfers data to the remote in append mode. The source of the data is determined by the configuration parameters Local Directory and Local File. If both are empty, then the data is from pay load.
connect(): Connects to the FTP server and does authentication as configured.
deleteDir(String dir): Deletes the remote directory as specified by the argument.
deleteFile(String path): Deletes the remote file as specified by the argument.
disconnect(): Disconnects from the FTP server.
doRawCommands(String commands): Specifies the raw commands.
download(): Downloads data from the remote (specified in configuration parameters Remote Directory and Remote File) to the local (specified in configuration parameters Local Directory and Local File).
get(): Copies the file or directory specified by the configuration parameters Remote Directory and Remote File, to the local, as specified by the configuration parameters Local Directory and Local File. If the configuration parameter, Is Copy Recursive, is set to Yes, the copy will be recursive.
getEntry(): Gets the index entry in the current entry list.
getEntryCount(): Returns the count of directory entries, as the result of invoking listDir or listDirLong.
getLastReply(): Gets the FTP response code as a String.
getPayload(): Returns the payload.
hasEntry(): Returns false if there is more to the entry in the directory listing result list (when the result list is exhausted), and calls resetEntries to re-iterate the result list again.
getResolvedLocalDirectory(): Returns the resolved local directory name.
getResolvedLocalFile(): Returns the resolved local file name.
getResolvedRemoteDirectory(): Returns the resolved remote directory name.
getResolvedRemoteFile(): Returns the resolved remote file name.
hasEntry(): Returns whether the current entry list has entries.
isConnected(): Determines if the Java Integration Suite is connected to the FTP server.
listDir(): Returns the entry under the remote directory (specified in configuration parameters Remote Directory and Remote File). The entry only contains the name. After this method is invoked, use methods such as hasEntry, nextEntry, getEntry, getEntryCount, and so forth. to iterate the entry information.
listDirLong(): Returns the entries under the remote directory (specified in configuration parameters Remote Directory and Remote File). The entry contains details such as name, size, time stamp, is directory or not, and so forth. After this method is invoked, use methods such as hasEntry, nextEntry, getEntry, getEntryCount, and so forth, to iterate the entry information.
mkdir(String dir): Creates a directory on the remote. The name of the directory is specified in the configuration parameters.
nextEntry(): Returns the next entry in the result list.
put(): Same as get, but the data transfer is from local to remote.
renameFile(String newName): Renames the file specified by the configuration parameters Remote Directory and Remote File to a new name (arg).
reset(): Resets the internal life cycle methods, such as discard payload buffer.
resetEntries(): Resets the result list iterator so that it can be iterated again.
resolveLocalAsDestination(): Resolves the local directory name and local file name if they are patterns (used to generate real directory and file name for data transfer destination), upon the success of the resolution.
resolveLocalAsSource(): Resolves the local directory and file if they are regular expressions (filters for data transfer source); upon the success of the resolution.
resolveRemoteAsDestination(): Resolves the remote directory name and remote file name if they are patterns (used to generate real directory and file name for data transfer destination), upon the success of the resolution.
resolveRemoteAsSource(): Resolves the remote directory and file if they are regular expressions (filters for data transfer source); upon the success of the resolution.
setpayload(byte[] newPayload): Sets the payload as specified by the argument.
setResolvedLocalDirectory(String s): Sets the resolved local directory name.
setResolvedLocalFile(String s): Sets the resolved local file name.
setResolvedRemoteDirectory(String s): Sets the resolved remote directory name.
setResolvedRemoteFile(String s): Sets the resolved remote file name.
upload(): Uploads data from the local (specified in configuration parameters Local Directory and Local File) to the remote (specified in configuration parameters Remote Directory and Remote File).
See the Batch adapter Javadoc for a list of all exposed FTPOverSSLClient methods.
The Batch Secure SFTP OTD (BatchSFTP) provides secure data transfer using SSH File Transfer Protocol or SFTP. SFTP provides a range of operations on remote files, such as resuming interrupted transfers, directory listings, and remote file removal.
SFTP is one means of securely transferring computer files between a local and a remote host or between two remote hosts, using the Secure Shell (SSH) protocol. The BatchSFTP OTD uses SFTP to copy a file to or from a remote host.
The BatchSFTP OTD contains two top-level nodes, Client and Configuration (see the following figure). Expand these nodes to reveal additional sub-nodes.
The following list provides an explanation of various nodes in the BatchSFTP OTD, including primary functions:
BatchSFTP: Represents the OTD’s root node.
The BatchSFTP sub-nodes under the Configuration node correspond to the BatchSFTP adapter’s Connectivity Map and Environment configuration parameters.
The BatchSFTP OTD’s Client node includes the following methods:
cd(String dir): Changes the remote current directory to the specified path.
connect(): Connects to the remote SSH server and does authentication as configured.
delete(): Deletes remote files specified by configuration parameters RemoteDirectory and RemoteFile.
delete(String file): Deletes remote files specified by file.
disconnect(): Disconnects the client from the remote SSH server.
get(): Copies data from a remote SSH server (specified by configuration parameters RemoteDirectory and RemoteFile) to the local machine. Depending on the current status of the configuration, the remote data can be stored into the payload (in memory buffer) or a local file specified by configuration parameters LocalDirectory and LocalFile.
The remote could be a folder. In this case, if the configuration parameter Recursive is "Yes", the folder hierarchy is copied to the local destination.
getEntry(int index): Gets the index entry in the current entry list.
getEntryCount(): Returns the number of entries in the current entry list.
getPayload(): Returns the payload.
If you call getPayload() when using the BatchSFTP OTD, and the Local Directory and Local Filename are set, getPayload()returns null, even if a file has been retrieved.
getResolvedLocalDirectory(): Returns the resolved local directory name.
getResolvedLocalFile(): Returns the resolved local file name.
getResolvedRemoteDirectory(): Returns the resolved remote directory name.
getResolvedRemoteFile(): Returns the resolved remote file name.
hasEntry(): Returns whether the current entry list has entries.
isConnected(): Determines if the Java Integration Suite is connected to the SSH server.
lcd(String dir): Changes the local current directory.
listDir(): Lists all the entries under remote current directory.
lpwd(): Returns the local current path.
mkdir(): Creates a directory on the remote. The name of the directory is specified in the properties.
mkdir(String dir): Creates a directory on the remote using the name specified in the configuration parameters, Remote Directory and Remote File.
nextEntry(): Returns the next entry in the current entry list.
put(): Copies local data (specified by configuration parameters LocalDirectory and LocalFile) to the remote SSH server (specified by configuration parameters RemoteDirectory and RemoteFile). Depending on the current status of the configuration, the local data can be from either a payload or local file.
The local could be a folder. In this case, if the configuration parameter Recursive is "Yes", the folder hierarchy is copied to the remote destination.
pwd(): Returns the remote current path.
rename(String newPath): Renames the file or directory specified by the old name (first argument), to new name (second argument).
rename(String oldPath, String newPath): Renames the file or directory specified by configuration parameters, Remote Directory and Remote File, to new name (argument).
reset(): Resets the internal life cycle methods, such as discard payload buffer.
resetEntries(): Resets the current entry list so that next call to nextEntry() will return the first entry in the list.
resolveLocalAsDestination(): Resolves the local directory name and local file name if they are patterns (used to generate real directory and file name for data transfer destination), upon the success of the resolution.
resolveLocalAsSource(): Resolves the local directory and file if they are regular expressions (filters for data transfer source); upon the success of the resolution.
resolveRemoteAsDestination(): Resolves the remote directory name and remote file name if they are patterns (used to generate real directory and file name for data transfer destination), upon the success of the resolution.
resolveRemoteAsSource(): Resolves the remote directory and file if they are regular expressions (filters for data transfer source); upon the success of the resolution.
setpayload(byte[] newPayload): Sets the payload.
setResolvedLocalDirectory(String s): Sets the current resolved local directory name.
setResolvedLocalFile(String s): Sets the current local file name to s, should not be invoked directly from user Collaboration.
setResolvedRemoteDirectory(String s): Sets the current resolved remote directory name.
setResolvedRemoteFile(String s): Sets the current resolved remote file name.
See the Batch adapter Javadoc for a list of all exposed SFTPClient methods.
The Batch Secure SCP OTD (BatchSCP) provides secure data transfer using Secure Copy Protocol with Secure Shell (SSH) as an underlying protocol. SCP is similar to RCP (remote copy), but the file is copied over secure shell (SSH) rather than RSH (remote shell). When files are copied using SCP the data is encrypted during transfer.
For information about configuring external FTP servers, SSH servers, and so forth, refer to the application’s documentation.
The BatchSCP OTD contains two top-level nodes, Client and Configuration (see the following figure). Expand these nodes to reveal additional sub-nodes.
The following list provides an explanation of various nodes in the BatchSCP OTD, including primary functions:
BatchSCP: Represents the OTD’s root node.
The BatchSCP sub-nodes under the Configuration node correspond to BatchSCP adapter’s Connectivity Map and Environment configuration parameters.
The BatchSCP OTD’s Client node includes the following methods:
connect(): Connects to the SSH server and does authentication as configured.
disconnect(): Disconnects from the SSH server.
get(): Copies the file or directory specified by the configuration parameters, Remote Directory and Remote File, to the local, as specified by configuration parameters Local Directory and Local File. If the configuration parameter, Is Copy Recursive, is set to Yes, the copy will be recursive.
getPayload(): Returns the payload buffer as a byte array.
getRecursive(): Copies data from remote to local with configuration parameter Recursive set to "Yes".
isConnected(): Checks whether the client is connected to the SSH server.
put(): Copies local data (specified by configuration parameters LocalDirectory and LocalFile) to the remote SSH server (specified by configuration parameters RemoteDirectory and RemoteFile).
putRecursive(): Copies data from local to remote with the configuration parameter Recursive set to "Yes".
setpayload(byte[] newPayload): Sets the payload.
See the Batch adapter Javadoc for a list of all exposed SCPClient methods.
The BatchLocalFile OTD provides access to files on your local system. While file access is not always necessary in the Sun Enterprise Service Bus, it makes sense for the Batch Adapter to have this feature because file processing is one of its core functions.
Additional BatchLocalFile features include regular expressions for accessing files and a sequence-numbering scheme for creating files.
The BatchLocalFile OTD contains four top-level nodes, Client, Configuration, PersistentState, and State Manager. Expand these nodes to reveal additional sub-nodes.
As it iw with each of the Batch OTDs, each field sub-node under the Configuration node in the BatchLocalFile OTD corresponds to one of the adapter’s configuration parameters for that OTD. See the BatchLocalFile Connectivity Map Properties for details.
This OTD includes an additional top-level node, the Client. This node implements its respective functionality interface in the adapter.
The client interface represents how the functionality of the OTD is actually used. This functionality includes the basic operations and features of the OTD. The client interface provides the OTD’s the file services those who want to use them.
The following list explains the nodes in the BatchLocalFile OTD, including primary functions:
Configuration: The field sub-nodes within this node corresponds to an adapter configuration parameter and contains the corresponding settings information. See the BatchLocalFile Connectivity Map Properties section for details on these parameters and settings.
This OTD has configuration parameters that can be regular expressions. See Using Regular Expressions With the Batch Adapter for details.
Client: The following sub-nodes, contained in this node, implement the adapter’s client interface in the OTD (LocalFileClient):
ResolvedNamesToGet and ResolvedNamesToPut: Allow you to get the real file or directory name used during a transfer and perform an operation with it. For example, you could do a local file transfer, with get() or put(), using the real name. You are able to retrieve the real file or directory name, even if these names have been expressed using regular expressions or special characters.These nodes contain sub-nodes allowing you to resolve file and directory names for target destinations, as well as names for pre- and post-transfer commands (see the Pre/Post File Transfer Commands section for details).
See Using Regular Expressions With the Batch Adapter and Using Name Patterns With the Batch Adapter for more information on these features.
InputStreamAdapter and OutputStreamAdapter: Allow you to use and control the OTD’s data-streaming features; see the Streaming Data Between Components section for details.
Payload: An in-memory buffer that contains the payload or message data you want to transfer by local file, in the form of a byte array.
get(), put(), and reset(): See Essential BatchLocalFile OTD Methods.
ResumeReadingInProgress: This node allows you to resume a data-streaming file transfer operation that was interrupted for whatever reason. These transfers occur piece by piece and usually involve large files. This feature allows you to resume at the same point where the transfer left off when it stopped.
You can transfer data using the Payload node or by using data streaming (InputStreamAdapter and OutputStreamAdapter nodes), but you cannot use both methods in the same OTD.
This section explains how to use the BatchLocalFile OTD’s features.
There is no required particular order for the calls that can be made on the BatchLocalFile OTD. The only required call is reset() after a transfer, if it is used for more than one transfer per Collaboration Rule execution. An example of this would be a dynamic batch order with multiple files to be transferred.
If you are using a Java Collaboration to generate multiple files with sequence numbers, using the BatchLocalFile interface, you must call the reset() method to indicate the end of one file and the start of the next one.
Using the BatchLocalFile OTD to read records from a local file has the following advantages:
Data Streaming: Allows your application to stream data directly to and from a local file system when used together with the BatchFTP OTD or the record-processing OTD. This feature minimizes the required RAM when large files are read, because the entire file is never loaded in memory.
Resume Reading: Allows your application to read large files in a number of subsequent Business Rule executions, when you are using data streaming. This operation is achieved by persisting information about the current successful file read operation and resuming the next read operation from that last stored position.
The adapter has features that allow you to execute desired actions directly before or after the actual file transfer. You can enter these settings at the adapter configuration parameters or in the Configuration node of the desired OTD.
These features are available with both the BatchLocalFile OTD and the BatchFTP OTD.
Pre Commands
For an inbound transfer, the file can be made unavailable to other clients polling the target system with the same directory and file pattern or name (Rename). For an outbound transfer, you can perform an automatic backup of the existing file (Copy).
Your pre-transfer options are:
Rename: Rename the target file for protection or recovery; you must provide a desired directory and file name. The directory is created if it does not already exist.
Copy: Copy the target file for backup or recovery; you must enter a desired directory and file name.
None: Do nothing.
Each FTP server can behave differently when you are using Rename and a destination file already exists. For example, for some UNIX FTP servers, the destination file is overwritten without question. That is, no error or warning message is given. On other FTP servers, a Windows XP server for example, the system generates an error that results in exceptions being thrown in the called OTD method.Be sure you are familiar with the native behavior of the corresponding FTP server. If you are in doubt, try the action at the command prompt. If the action displays an error message, it is likely to result in the throwing of an exception in the Collaboration.
To gain proper protection, backup, or recovery, you must choose the appropriate setting that serves your purpose. For example, to recover from failures on an outbound appending transfer, use the Copy setting. When specifying file and directory names you can use regular expressions, special characters, or both.
Post Commands
For an inbound transfer, you can mark the transferred file as “consumed” by making an automatic backup (Rename) or by destroying it permanently (Delete). For an outbound transfer, you can make the transferred file available to other clients by renaming it. When specifying file and directory names you can use regular expressions, special characters, or both.
Your post-transfer options are:
Rename: Rename the transferred file; you must provide a desired directory and file name.
Delete: Delete the transferred file (inbound transfers only).
None: Do nothing.
For an outbound transfer (publishing), the directory is created if it does not already exist.
For more information on Pre and Post Transfer commands see the following:
BatchFTP OTD
Pre Transfer (BatchFTP Connectivity Map)
Post Transfer (BatchFTP Connectivity Map)
BatchLocalFile OTD
Pre Transfer (BatchLocalFile Connectivity Map)
Post Transfer (BatchLocalFile Connectivity Map)
In addition to the field elements, the BatchLocalFile OTD’s Client node contains methods that extend the client interface functionality of the adapter. These methods are essential to the proper use of the OTD. These methods include:
get(): Retrieves a local file then stores its contents as a data payload. The method retrieves the first matching file based on the Target Directory Name and Target File Name parameters and stores the contents as a data payload (a byte array). It then performs any Post Transfer Command.
After this method call, you can get the payload’s contents through the method getPayload().
put(): Stores the data payload (as a byte array) to a file. It then performs any Post Transfer Command.
Before using this method call, you must set the file contents using the method setPayload().
The method throws an exception (LocalFileException) if an error occurs.
reset(): Allows you to return the Client node to its state immediately after the previous initialization.
The reset() method is available in both BatchFTP and BatchLocalFile OTDs. It must be called when the OTD has to be reused for another transfer during the same execution of executeBusinessRules() (for example, when you are using the Dynamic Configuration feature). The reset() method resets the content of the Client node without resetting the whole OTD.
The purpose of this feature is to allow an application to read large files in parts instead of processing the whole file at once. Resume Reading allows your system to read files in a number of subsequent Business Rule executions, when you are using data streaming.
General Operation
The Resume Reading feature’s operation is achieved by keeping persistent information about the current successful file read operation, breaking, then resuming the next read operation from that last stored break position. As a result, the current file is read in parts, and the beginning and end of each part is determined by a predefined break condition.
You determine the break condition through the definition of your Business Rules. Since the Resume Reading feature operates based on reading one part of a file at a time per Business Rule, these rules must determine the break. Each Business Rule executes reading a part of the file, breaks, then passes to the next rule, which reads the next part up to the break, and so on, until the entire file is read.
A break condition can be any type of stopping point you determine in your Collaboration Rules. For example, this condition could be a fixed number of records, a delimiter, or reaching a specific character string.
The Client node in the OTD has a read-only property (ResumeReadingInProgress node) indicating whether there is a resume-reading operation in progress. This node is for informational purposes only. Also, the Resume Reading feature is available in the data-streaming mode only.
The feature has no special operational requirements besides setting the adapter configuration option. The adapter configuration has an option to enable or disable this feature. This option is also accessible at run time.
If this feature is enabled, the adapter always checks first for a resume-reading operation in progress. If this feature is not in progress, the adapter determines the next file based on the adapter configuration settings.
Step-by-step Operation
The figure shows how the Resume Reading feature operates along with pre- and post-file-transfer commands. In this example, the Collaboration executes the same Business Rule four times. Each execution causes the Collaboration to read another section of the file. When the Collaboration reads the final records, it executes the Post-Transfer commands
In this example, the reading happens in the following steps:
The adapter starts reading the file then reaches a break condition after a partial data read (the end of Part 1), the adapter’s pre-transfer commands have already been executed. The resume-reading state is stored, and no post-transfer commands are executed. The adapter is waiting for the next execution of the Business Rule.
The resume-reading operation is in progress but still attains only partial data reads. The adapter reads from one break condition to the next (Part 2 and Part 3 in the figure) The resume-reading state is stored in each case, and the adapter executes the Business Rule once per each part.
The resume-reading operation is in progress and completes its data read during the final execution of the Business Rule (Part 4). The adapter reads from a break condition to the end of a file. No resume-reading state is stored, and any post-transfer commands are then executed.
In all of the previous steps, the Business Rule is executed repeatedly, and the current read position in the file changes on each execution. If the file is smaller than Part 1 in the figure, the adapter does not reach a break condition. The operation is normal, and no resume-reading state is stored. The pre- and post-transfer commands are executed.
Operation Without Resume Reading Enabled
If the Resume Reading feature is not enabled:
Data-read Stop Then Restart: Any unread data at the end of the file is ignored.
Resume Reading in Progress: If there is a resume-reading operation in progress from a previous execution, an error is generated, and an exception is thrown.
If there is a resume-reading operation in progress it cannot be interrupted and must be completed. The executeBusinessRules() method must be called enough times to fully consume the file. In other words, do not discontinue processing the file before it has been completely consumed.
To Avoid Storing a Resume Reading State
Sometimes a partial data-stream read is necessary even when the Resume Reading feature is enabled. For example, there could be some application logic on top of the record parsers, which might abandon the rest of the file because of a corrupted record and close the file successfully after reading only part of the file’s content.
In this case, you must set the LocalFileOTD.Configuration.ResumeReading node to False before calling finish(). This setting tells the BatchLocalFile OTD to complete the operation without storing a resume-reading state. You can set up the Collaboration Rule to then send notifications or take other measures, as desired.
You can use the BatchLocalFile OTD to implement the adapter’s data streaming feature. This feature is also available with the FTP and record-processing OTDs. However, the BatchLocalFile OTD is a data stream-adapter provider, while the other two OTDs are only consumers. See Streaming Data Between Components for details on how to use the OTD’s data streaming feature.
Sequence numbering for the BatchLocalFile OTD operates similar to sequence numbering for the BatchFTP OTD. See Sequence Numbering for details.
Generating Multiple Files with Sequence Numbering
When using a Java Collaboration to generate multiple files that have sequence numbering using the BatchLocalFile interface, reset() must be called to signify the end of one file and the start of the next.
This feature in this OTD operates in the same way as type conversion for the BatchFTP OTD. See Handling Type Conversions for details.
To parse records or construct payloads in your Collaboration, we recommend that you use the record-processing BatchRecord OTD together with the BatchLocalFile OTD. Using the two OTD together provides a number of advantages over just using the BatchFTP OTD.
For example, you have set up a Collaboration Rule to parse a large file and submit the records to a database or a JMS IQ Manager. If something goes wrong during the parsing process, the whole file needs to be transmitted again from the FTP server.
In contrast, streaming from a local file system can avoid later FTP transfers of the same file in case of error. This approach has the advantage of allowing you to use data streaming and the Resume Reading feature with large files (see Streaming Data Between Components and Resume Reading Feature.
Another scenario could be a case where a slow, complex SQL query is used to retrieve a number of records. The Collaboration Rule packs them into a Payload node using the record-processing OTD then sends them through FTP to an external system. If the FTP transfer fails, the SQL query must be executed again.
In contrast, if the data payload has been stored locally with the BatchLocalFile OTD, the FTP transfer can be repeated without the need to re-execute the SQL query. In such cases, you can also use data streaming and local-file appending.
In both cases, the use of a data-streaming link can significantly reduce the memory requirements compared to the in-memory data-payload transfer used with the BatchFTP OTD.
The BatchLocalFile OTD supports mapped drives and NFS mounted drives. It does not, however, support the mapping of the drives. That is, the drive must already be mapped or mounted. The adapter itself does not perform any mapping or mounting.
The OTD supports Universal Reference Identifiers (URIs) but the scheme must be left off, for example,\\drive\directory\file_name.
BatchRecord, the Batch Adapter’s record-processing OTD, allows you to parse (extract) records from an incoming payload (payload data) or to create an outgoing payload consisting of records. Understanding the operation of this OTD and how to use it requires an explanation of some of these terms.
The word payload here refers to an in-memory buffer, that is, a sequence of bytes or a stream. Also, records in this context are not records in the database sense. Instead, a record simply means a sequence of bytes with a known and simple structure, for example, fixed-length or delimited records.
For example, each of the following record types can be parsed or created by this OTD:
A large data file that contains a number of SAP IDocs, each with 1024 bytes in length.
A data file that contains a large number of X12 purchase orders, each terminated by a special sequence of bytes.
The record-processing OTD can handle records in the following formats:
Fixed length: Each record in the payload is exactly the same size.
Delimited: Each record is followed by a specific sequence of bytes, for example, CR,LF.
Single: The entire payload is the record.
When using character delimiters with DBCS data, use single byte character(s) or equivalent hex values with hex values that do not coincide with either byte of the double byte character.
The BatchRecord OTD contains two top-level nodes, Client, Configuration, PersistentState, and StateManager (see the following figure). Expand these nodes to reveal additional sub-nodes.
Each field node under the Configuration node of the OTD, corresponds to one of the adapter’s record-processing configuration parameters.
The following list explains these primary nodes in the record-processing OTD, including their functions:
BatchRecord: Represents the OTD’s root node.
Configuration: Each sub-node within this node corresponds to an adapter configuration parameter and contains the corresponding settings information, except for the Parse or Create parameter. See BatchRecord Connectivity Map Properties for details.
InputStreamAdapter and OutputStreamAdapter: Allow you to use and control the data-streaming features of the OTD. For details on their operation, see Streaming Data Between Components.
You can transfer data using the Payload node or by using data streaming (InputStreamAdapter and OutputStreamAdapter nodes), but you cannot use both methods in the same OTD.
For the record-processing OTD, these configuration nodes are read-only. They are provided only for the purpose of accessing and checking the configuration information at run time.
Record: A properties node that represents either:
The current record just retrieved through the get() method, if the call succeeded
The current record to be added to the data payload when put() is called
Payload: The in-memory buffer containing the data payload byte array you are parsing or creating.
It is a good practice to use a byte array in all cases. Failure to do so can cause loss of data.
put(): Adds whatever is currently in the Record node to the data payload. The method returns true if the call is successful.
get(): Retrieves the next record from the data payload (or stream), and populates the Record node with the record retrieved. get() returns true if the call is successful.
finish(): Allows you to indicate a successful completion of either a parse or create loop for both put() and get().
Usereset() to indicate any errors and to allow the OTD to clean up any unneeded internal data structures.
This OTD has the following basic uses:
Parsing a payload: When the payload comes from an external system
Creating a payload: Before sending the payload to an external system
A single instance of the OTD is not designed to be used for both purposes at the same time in the same Collaboration. To enforce this restriction, there is a setting under the adapter’s General Settings parameters called Parse or Create Mode, for which you can select either Parse or Create.
The get() and put() methods are the heart of the OTD’s functionality. If you call either method, the record retrieved or added is assumed to be of the type specified in the adapter configuration, for example, fixed-length or delimited.
The get() method can throw an exception, but generally this action only happens when there is a severe failure. One such failure is an attempt to call get() before the payload data (or stream if you are streaming) has been set. However, the best practice is to code the Collaboration to check the return value from a get() call. A return of true means a successful get operation; a false means the opposite.
The adapter checks to ensure that the proper calls are made according to your mode setting. For example, calling put() in a parse-mode environment would cause the adapter to throw an exception with an appropriate error message explaining why. Calling get() in the create mode would also result in an error.
The adapter requires these restrictions because:
If you are processing an inbound payload, you are calling get() to extract records from the payload (parsing). In this situation it makes little sense to call put(). Doing so at this point would alter the payload while you are trying to extract records from it. Calling put() would overwrite the payload and destroy the data you are trying to obtain.
Conversely, when you are creating a payload by calling put(), you have no need to extract or parse data at this point. Therefore, you cannot call get().
As a result, you can place the OTD on the source or destination side of a given Collaboration, as desired, and use the OTD for either parsing or creating a payload. However, you cannot parse and create at the same time. Implement your OTD in a Collaboration using the Collaboration Rules Editor.
When you want the payload data sent to an external system, you can place the OTD on the outbound side of the Collaboration interfacing with that system. Successive calls to put() build up the payload data in the format defined in the adapter configuration.
Once all the records have been added to the payload, you can drag and drop the payload onto the node or nodes that represent the Collaboration’s outbound destination. Also, you can set an output stream as the payload’s destination (see Table 1–3 for details on payload streaming).
When you are building a data payload, you must take into account the type and format of the data you are sending. The adapter allows you to use the following formats:
Single Record: This type of payload represents a single record to be sent. Each successive call to put() has the effect of growing the payload by the size of the data being put, and the payload is one contiguous stream of bytes.
Fixed-size Records: This type of payload is made up of records, with each being exactly the same size. An attempt to put() a record that is not of the size specified causes an exception to be thrown.
Delimited Records: This type of payload is made up of records that have a delimiter at the end. Each record can be a different size. Do not add any delimiters to this data type when it is passed to put(). The delimiters are added automatically by the adapter.
User Defined: In this type of payload, the semantics are fully controlled by your own implementation.
To represent payload data inbound from an external system, you map the data to the payload node in the OTD (from the Collaboration Rules Editor). In addition, you can specify an input stream as a source.
Either way, each successive call to get() extracts the next record from the payload. The type of record extracted depends on the parameters you set in the adapter’s configuration, for example, fixed size or delimited.
You must design the parsing Collaboration with instructions on what to do with each record extracted. Normally, the record can be sent to another Collaboration where a custom OTD describes the record format and carries on further processing.
Fully Consuming a Payload
It is possible to fully consume a payload. That is, after a number of successive calls to get(), you can retrieve all the records in the payload. After this point, successive calls to get() return the Boolean false. You must design the business rules in the subject Collaboration to take this possibility into account.
If you are using the record-processing OTD with data streaming, you must be careful not to overwrite the output files. If the OTD is continually streaming to a BatchLocalFile OTD that uses the same output file name, the OTD can write over files on the output side.
To avoid this problem, you must use either file sequence numbering or change the output file names in the Collaboration Rules. Sequence numbering allows the BatchLocalFile OTD to distinguish individual files by adding a sequence number to them. If you use target file names, post-transfer file names, or both, you can change the name of the output file to a different file name.
For more information on how to use these features, see Sequence Numbering and Pre/Post File Transfer Commands.
The BatchInbound Input (Trigger) File adapter polls for an input file, renames the file with a GUID prefix, and triggers the Business Process or Collaboration.
The Batch adapter’s BatchInbound OTD acts similar to the inbound File adapter, in that it regularly polls an input directory for inbound target files. But unlike the File adapter, when a file with the appropriate name is received by the BatchInbound adapter, the target file is renamed with the following pattern: GUID.original_filename to ensure that the file is not over-written and is only sent once. A GUID (Globally Unique Identifier) provides a unique, formatted string that represents a 128-bit value.
The BatchInbound OTD does not read the file, but renames the file in such a way that it provides the name of the file that triggers the Business Process or Collaboration.
The BatchInbound OTD contains one top-level node, BatchAppconnMessage, with three fields, PathDirName, OriginalFileName, and GUIDFileName (see the following figure). These nodes provide the external input directory, original file name, and the GUID file name.
A regular expression is a character string in which some characters provide special meaning in regard to matching patterns. This section explains some basic guidelines on how to use regular expressions with the HTTPS.
Regular expressions allow you to specify patterns for file names and directory names. Regular expressions are used for “get” operations (receiving or source), as opposed to name patterns which are used for “put” operations (sending or destination).
The BatchFTP, Batch FTPOverSSL, BatchSFTP, BatchLocalFile, and BatchInbound OTDs allow you to use regular expressions, for example, if you want to access all files with a specific extension.
Regular expressions operate as follows:
The directory/file names can be defined as either:
Actual file names (everywhere)
Name patterns (all names for “put” operations and pre/post transfer names for get operations)
Regular expressions (target names for “get” operations)
The difference between the regular expressions and name patterns is:
Regular expressions are used to match existing names on the FTP server or the local file system.
Name patterns are used to create names by replacing the special characters in the pattern.
For more information on name patterns using special characters, see Using Name Patterns With the Batch Adapter.
You can specify an extension, for example, .*\.dat$. Then, each time the get() method is called, the adapter gets the next file with a .dat extension. The adapter then retrieves each file into the OTD’s Payload node and updates the working file-name attribute with the name of the file currently being accessed.
For another example, you can use the file-matching the pattern data\.00[1-9] to get the files data.001, then data.002, and so on. In each case the “.” is escaped, which is consistent with regular-expression syntax. It also matches to xyzdata.001 and xyz.data.001, because it does not exclude anything before “data”. To make “data” the exact start of the matching pattern you must use ^data\.00[1-9] or \A data\.00[1-9].
The use of regular expressions is an advanced feature and must be implemented carefully. An improperly formed regular expression can cause undesired data or even the loss of data. You must have a clear understanding of regular-expression syntax and construction before attempting to use this feature. It is recommended that you test such configurations thoroughly before moving them to production.
You can enter a regular expression for the FTP or local file name in a variety of ways, for example, .*\.dat$ or ^xyz.*\.dat$. The first case indicates all files with an extension of .dat. The second case indicates all file names with an extension of .dat whose names start with xyz.
Another example could be file[0-9]\.dat. This expression specifies file0.dat, file1.dat, file2.dat, and so on, through file9.dat. This will also match xyz.file0.dat, xyz.file1.dat, and so on. This type of expression will not exclude anything in front of “file”. To exclude any characters before “file” (to make “file” the exact beginning) use ^file[0-9].dat or \Afile[0-9].dat.
These types of regular expression patterns can be used for a get operation.
The adapter provides a File Name Is Pattern or Directory Name Is Pattern configuration parameter after every property that allows a regular expression as an option. This feature allows you to specify that the pattern entered is a regular expression or just a static text entry to be interpreted literally.
Regular expressions will resolve even with a partial match to the file name. The resolution process searches for the file name contents rather than the file name.
There are special considerations you must be aware of when you use regular expressions for directory names. This section describes these restrictions and provides some examples.
The following restrictions apply when using regular expressions as directory names:
The directory root, the drive name, and directory separators must be expressed exclusively. That is, do not express any of these elements as a regular expression. Only folder names are expected to appear as regular expressions.
A regular expression must not span over the directory separators. If you use a regular expression between two directory separators, it must be one whole expression.
Escape all directory separators in a directory pattern if the separator conflicts with a regular expression special character (that is, ” * [ ] ( ) | + { } : . ^ $ ? \"). The back slash (\) is the special character used to escape other special characters in regular expressions. For Windows platforms, the directory separator is the back slash, so it must be escaped as \\.
For the Windows Universal Naming Convention (UNC), the directory root (including the computer name and the shared root folder name) must be expressed exclusively. That is, do not express the computer name and shared root folder as a regular expression.
Directory separators are platform dependent, for example:
For Windows platforms, the directory path follows this pattern:
drive:\\regexp1\\regexp2\\regexp3 ... |
or for Windows UNC notation, the directory path follows this pattern:
\\\\machineName\\shared_folder\\regexp1\\regexp2\\regexp3 ... |
For UNIX platforms, including mounted directories, the directory path follows this pattern:
/regexp1/regexp2/regexp3 ... |
The following are several examples of regular expression directory name usage:
Windows:
c:\\$\\^client\\collab\D\\ ... |
The expression \D indicates any non-digit character.
d:\\a.b\\c.d\\e.f\\g.h\\[0-9]\\ ... |
The symbol “.” means any character
Windows UNC notation:
\\\\My_Machine\\public\\xyz$\\^abc |
The prefix for Windows UNC notation is \\. After escaping, it becomes \\\\.
UNIX:
/abc\d/def/ghi/ ... |
The expression \d means any digit character.
/^PRE[0-9]{5}\.dat$/ ... |
This expression means to begin with PRE followed by a five-digit number and use a .dat extension. The symbol \. means to interpret the real character (a period) instead of any character. Therefore, PRE12345.dat does match, but PRE123456dat does not.
The Batch adapter allows you to use a Name Pattern, that is, special characters that symbolize often-used information as short-hand. You can use these character combinations to specify place holders for this specific information. Using these characters, you can quickly convey date/time, number, and file-name information.
The BatchFTP, Batch FTPOverSSL, BatchSFTP, BatchLocalFile, and BatchInbound OTDs allow you to use special characters or specify a name pattern. A name pattern allows you to specify patterns for file names and directory names. Name patterns are used for “put” operations (sending or destination) , as opposed to regular expressions which are used for “get” operations (receiving or source).
Special characters are utilities the adapters use for file-name pattern. The general rules for their use are:
Use % to indicate the special character that needs to be expanded.
Use %% to indicate the escaped character %; for example, abc%%d means abc%d, and the %d is not expanded again.
For example, for a put operation, a pattern such as file%#.dat can be used. This pattern uses the sequence number setting in the configuration, and each put creates successive files named file1.dat, file2.dat, and so on.
For information on regular expressions, see Using Regular Expressions With the Batch Adapter.
The adapter provides the following types of name patterns:
Date/Time stamp: Uses the format %[GyMdhHmsSEDFwWakKz], for example, abc%y%y%y%y means abc2001 (see Table 1–2 for more information).
Sequence number: Uses the format %#, %5#, for example, abc%# means abc1, abc2, abc3, and so on; for another example, abc%5# (zero-padded) means abc00001, abc00002, abc00003, ..., abc00010, ..., abc00100, and so on.
Working-file name: Uses the format %f; normally, it is used for pre- or post-file-transfer commands, for example, %f.abc means working_filename.abc.
The sequence of expansion operates in the reverse order of the previous list, that is, first the file name is expanded, then the sequence number, and finally the time stamp.
Some additional examples of name pattern:
abc.%y%y%y%y%M%M%d%d.%h%h%m%m%s%s%S%S%S means abc.20011112.162532678
abc%#.def%# means abc2.def3
%f.%# means xxxxx.4, xxxxx.5, ... (Where xxxxx is the working-file name)
Typically, the pre/post names with name patterns or regular expressions are resolved during get() and put() method calls. But sometimes, in using Collaboration Rules, the adapter has to get the resolved names before the actual get() or put() call.
In such cases, you can get the resolved names in this way through the ResolvedNamesForGet and ResolvedNamesForPut nodes in the BatchFTP OTD, for example:
getResolvedNamesForPut().getTargetFileName()
The previous code yields file1 based on the pattern file%#. In this usage, the OTD nodes can be used to make the desired method call.
The adapter uses the Java simple default date and time format syntax (U.S. locale). To specify these formats for name pattern, you must use a time pattern string.
The adapter uses the Java standard for date/time stamps from the Java class java.text.SimpleDateFormat. Some of these formats can differ from the list given here, depending on the Java SDK version you are using.
In these patterns, all ASCII letters are reserved as pattern letters. See Table 1–2 for a complete list.
Table 1–2 Time Pattern Strings and Meanings
Symbol |
Meaning |
Presentation |
Example |
---|---|---|---|
%G |
Era designator |
Text |
AD |
%y |
Year |
Number |
1996 |
%M |
Month in year |
Text and number |
July & 07 |
%d |
Day in month |
Number |
10 |
%h |
Hour in a.m./p.m. (1 through 12) |
Number |
12 |
%H |
Hour in day (0 through 23) |
Number |
0 |
%m |
Minute in hour |
Number |
30 |
%s |
Second in minute |
Number |
55 |
%S |
Millisecond |
Number |
978 |
%E |
Day in week |
Text |
Tuesday |
%D |
Day in year |
Number |
189 |
%F |
Day of week in month |
Number |
2 (second Wednesday in July) |
%w |
Week in year |
Number |
27 |
%W |
Week in month |
Number |
2 |
%a |
Marker for a.m./p.m. |
Text |
PM |
%k |
Hour in day (1 through 24) |
Number |
24 |
%K |
Hour in a.m./p.m. (0 through 1) |
Number |
0 |
%z |
Time zone |
Text |
Pacific Standard Time |
The general rules for date/time formats are:
Text: The count of pattern letters determines the format as follows:
For four or more pattern letters, use the full form.
For fewer than four, use the short or abbreviated form if one exists.
Number: The minimum number of digits as follows:
Shorter numbers are zero-padded to this amount.
The year is handled differently; that is, if the count of “y” is two, the year is truncated to two digits.
Text and number: For three or more pattern letters, use text; otherwise use a number.
Quotes and Delimiters: Use these symbols as follows:
Enclose literal text you want rendered within single quotes.
Use double quotes to mean single quotes.
Use commas for delimiters.
Table 1–3 shows some examples using the U.S. locale.
Format Pattern |
Result |
---|---|
yyyy.MM.dd, G, ’at’ hh:mm:ss, z |
1996.07.10 AD at 15:08:56 PDT |
E, M, dd, ’’yy |
Wednesday, July 10, ’96 |
h:mm, a |
12:08 PM |
h, ’o’’clock’ a, z |
12 o’clock PM., Pacific Daylight Time |
K:mm a, z |
0:00 p.m., PST |
yyyyy.M.dd, G, hh:mm, a |
1996.July.10 AD 12:08 PM |