BatchLocalFile OTD
The BatchLocalFile OTD provides access to files on your local system. While file access is not always necessary in Java CAPS, it makes sense for the Batch Adapter to have this feature because file processing is one of its core functions.
Additional BatchLocalFile features include regular expressions for accessing files and a sequence-numbering scheme for creating files.
BatchLocalFile OTD Structure
The BatchLocalFile OTD contains four top-level nodes, Client, Configuration, PersistentState, and State Manager. Expand these nodes to reveal additional sub-nodes.
Figure 7 BatchLocalFile OTD Structure
Configuration Node
As it iw with each of the Batch OTDs, each field sub-node under the Configuration node in the BatchLocalFile OTD corresponds to one of the adapter’s configuration parameters for that OTD. See BatchLocalFile Connectivity Map Properties for details.
Client Node
This OTD includes an additional top-level node, the Client. This node implements its respective functionality interface in the adapter.
The client interface represents how the functionality of the OTD is actually used. This functionality includes the basic operations and features of the OTD. The client interface provides the OTD’s the file services those who want to use them.
BatchLocalFile OTD Node Functions
The following list explains the nodes in the BatchLocalFile OTD, including primary functions:
-
Configuration: The field sub-nodes within this node corresponds to an adapter configuration parameter and contains the corresponding settings information. See BatchLocalFile Connectivity Map Properties for details on these parameters and settings.
Note –This OTD has configuration parameters that can be regular expressions. See Using Regular Expressions for details.
-
Client: The following sub-nodes, contained in this node, implement the adapter’s client interface in the OTD (LocalFileClient):
-
ResolvedNamesToGet and ResolvedNamesToPut: Allow you to get the real file or directory name used during a transfer and perform an operation with it. For example, you could do a local file transfer, with get() or put(), using the real name. You are able to retrieve the real file or directory name, even if these names have been expressed using regular expressions or special characters.These nodes contain sub-nodes allowing you to resolve file and directory names for target destinations, as well as names for pre- and post-transfer commands (see Pre/Post File Transfer Commands for details).
See Using Regular Expressions and Using Name Patterns for more information on these features.
-
InputStreamAdapter and OutputStreamAdapter: Allow you to use and control the OTD’s data-streaming features; see Streaming Data Between Components for details.
-
Payload: An in-memory buffer that contains the payload or message data you want to transfer by local file, in the form of a byte array.
-
get(), put(), and reset(): See Essential BatchLocalFile OTD Methods.
-
ResumeReadingInProgress: This node allows you to resume a data-streaming file transfer operation that was interrupted for whatever reason. These transfers occur piece by piece and usually involve large files. This feature allows you to resume at the same point where the transfer left off when it stopped.
Note –You can transfer data using the Payload node or by using data streaming (InputStreamAdapter and OutputStreamAdapter nodes), but you cannot use both methods in the same OTD.
-
Using the BatchLocalFile OTD
This section explains how to use the BatchLocalFile OTD’s features.
There is no required particular order for the calls that can be made on the BatchLocalFile OTD. The only required call is reset() after a transfer, if it is used for more than one transfer per Collaboration Rule execution. An example of this would be a dynamic batch order with multiple files to be transferred.
If you are using a Java Collaboration to generate multiple files with sequence numbers, using the BatchLocalFile interface, you must call the reset() method to indicate the end of one file and the start of the next one.
BatchLocalFile Specific Features
Using the BatchLocalFile OTD to read records from a local file has the following advantages:
-
Data Streaming: Allows your application to stream data directly to and from a local file system when used together with the BatchFTP OTD or the record-processing OTD. This feature minimizes the required RAM when large files are read, because the entire file is never loaded in memory.
-
Resume Reading: Allows your application to read large files in a number of subsequent Business Rule executions, when you are using data streaming. This operation is achieved by persisting information about the current successful file read operation and resuming the next read operation from that last stored position.
For more information on the Resume Reading feature, see Resume Reading Feature.
Pre/Post File Transfer Commands
The adapter has features that allow you to execute desired actions directly before or after the actual file transfer. You can enter these settings at the adapter configuration parameters or in the Configuration node of the desired OTD.
These features are available with both the BatchLocalFile OTD and the BatchFTP OTD.
Pre Commands
For an inbound transfer, the file can be made unavailable to other clients polling the target system with the same directory and file pattern or name (Rename). For an outbound transfer, you can perform an automatic backup of the existing file (Copy).
Your pre-transfer options are:
-
Rename: Rename the target file for protection or recovery; you must provide a desired directory and file name. The directory is created if it does not already exist.
-
Copy: Copy the target file for backup or recovery; you must enter a desired directory and file name.
-
None: Do nothing.
Caution – Each FTP server can behave differently when you are using Rename and a destination file already exists. For example, for some UNIX FTP servers, the destination file is overwritten without question. That is, no error or warning message is given. On other FTP servers, a Windows XP server for example, the system generates an error that results in exceptions being thrown in the called OTD method.Be sure you are familiar with the native behavior of the corresponding FTP server. If you are in doubt, try the action at the command prompt. If the action displays an error message, it is likely to result in the throwing of an exception in the Collaboration.
To gain proper protection, backup, or recovery, you must choose the appropriate setting that serves your purpose. For example, to recover from failures on an outbound appending transfer, use the Copy setting. When specifying file and directory names you can use regular expressions, special characters, or both.
Post Commands
For an inbound transfer, you can mark the transferred file as “consumed” by making an automatic backup (Rename) or by destroying it permanently (Delete). For an outbound transfer, you can make the transferred file available to other clients by renaming it. When specifying file and directory names you can use regular expressions, special characters, or both.
Your post-transfer options are:
-
Rename: Rename the transferred file; you must provide a desired directory and file name.
-
Delete: Delete the transferred file (inbound transfers only).
-
None: Do nothing.
Note –For an outbound transfer (publishing), the directory is created if it does not already exist.
For more information on Pre and Post Transfer commands see the following:
BatchFTP OTD
BatchLocalFile OTD
Essential BatchLocalFile OTD Methods
In addition to the field elements, the BatchLocalFile OTD’s Client node contains methods that extend the client interface functionality of the adapter. These methods are essential to the proper use of the OTD. These methods include:
-
get(): Retrieves a local file then stores its contents as a data payload. The method retrieves the first matching file based on the Target Directory Name and Target File Name parameters and stores the contents as a data payload (a byte array). It then performs any Post Transfer Command.
After this method call, you can get the payload’s contents via the method getPayload().
-
put(): Stores the data payload (as a byte array) to a file. It then performs any Post Transfer Command.
Before using this method call, you must set the file contents using the method setPayload().
The method throws an exception (LocalFileException) if an error occurs.
-
reset(): Allows you to return the Client node to its state immediately after the previous initialization.
Note –The reset() method is available in both BatchFTP and BatchLocalFile OTDs. It must be called when the OTD has to be reused for another transfer during the same execution of executeBusinessRules() (for example, when you are using the Dynamic Configuration feature). The reset() method resets the content of the Client node without resetting the whole OTD.
Resume Reading Feature
The purpose of this feature is to allow an application to read large files in parts instead of processing the whole file at once. Resume Reading allows your system to read files in a number of subsequent Business Rule executions, when you are using data streaming.
General Operation
The Resume Reading feature’s operation is achieved by keeping persistent information about the current successful file read operation, breaking, then resuming the next read operation from that last stored break position. As a result, the current file is read in parts, and the beginning and end of each part is determined by a predefined break condition.
You determine the break condition through the definition of your Business Rules. Since the Resume Reading feature operates based on reading one part of a file at a time per Business Rule, these rules must determine the break. Each Business Rule executes reading a part of the file, breaks, then passes to the next rule, which reads the next part up to the break, and so on, until the entire file is read.
A break condition can be any type of stopping point you determine in your Collaboration Rules. For example, this condition could be a fixed number of records, a delimiter, or reaching a specific character string.
The Client node in the OTD has a read-only property (ResumeReadingInProgress node) indicating whether there is a resume-reading operation in progress. This node is for informational purposes only. Also, the Resume Reading feature is available in the data-streaming mode only.
The feature has no special operational requirements besides setting the adapter configuration option. The adapter configuration has an option to enable or disable this feature. This option is also accessible at run time.
Note –
If this feature is enabled, the adapter always checks first for a resume-reading operation in progress. If this feature is not in progress, the adapter determines the next file based on the adapter configuration settings.
Step-by-step Operation
The diagram below illustrates how the Resume Reading feature operates along with pre- and post-file-transfer commands. In this example, the Collaboration executes the same Business Rule four times. Each execution causes the Collaboration to read another section of the file. When the Collaboration reads the final records, it executes the Post-Transfer commands
Figure 8 Resume Reading Operation
In this example, the reading happens in the following steps:
-
The adapter starts reading the file then reaches a break condition after a partial data read (the end of Part 1), the adapter’s pre-transfer commands have already been executed. The resume-reading state is stored, and no post-transfer commands are executed. The adapter is waiting for the next execution of the Business Rule.
-
The resume-reading operation is in progress but still attains only partial data reads. The adapter reads from one break condition to the next (Part 2 and Part 3 in the figure) The resume-reading state is stored in each case, and the adapter executes the Business Rule once per each part.
-
The resume-reading operation is in progress and completes its data read during the final execution of the Business Rule (Part 4). The adapter reads from a break condition to the end of a file. No resume-reading state is stored, and any post-transfer commands are then executed.
In all of the previous steps, the Business Rule is executed repeatedly, and the current read position in the file changes on each execution. If the file is smaller than Part 1 in the figure, the adapter does not reach a break condition. The operation is normal, and no resume-reading state is stored. The pre- and post-transfer commands are executed.
Operation Without Resume Reading Enabled
If the Resume Reading feature is not enabled:
-
Data-read Stop Then Restart: Any unread data at the end of the file is ignored.
-
Resume Reading in Progress: If there is a resume-reading operation in progress from a previous execution, an error is generated, and an exception is thrown.
Note –If there is a resume-reading operation in progress it cannot be interrupted and must be completed. The executeBusinessRules() method must be called enough times to fully consume the file. In other words, do not discontinue processing the file before it has been completely consumed.
To Avoid Storing a Resume Reading State
Sometimes a partial data-stream read is necessary even when the Resume Reading feature is enabled. For example, there could be some application logic on top of the record parsers, which might abandon the rest of the file because of a corrupted record and close the file successfully after reading only part of the file’s content.
In this case, you must set the LocalFileOTD.Configuration.ResumeReading node to False before calling finish(). This setting tells the BatchLocalFile OTD to complete the operation without storing a resume-reading state. You can set up the Collaboration Rule to then send notifications or take other measures, as desired.
Data Stream-adapter Provider
You can use the BatchLocalFile OTD to implement the adapter’s data streaming feature. This feature is also available with the FTP and record-processing OTDs. However, the BatchLocalFile OTD is a data stream-adapter provider, while the other two OTDs are only consumers. See Streaming Data Between Components for details on how to use the OTD’s data streaming feature.
Sequence Numbering
Sequence numbering for the BatchLocalFile OTD operates similar to sequence numbering for the BatchFTP OTD. See Sequence Numbering for details.
Generating Multiple Files with Sequence Numbering
When using a Java Collaboration to generate multiple files that have sequence numbering using the BatchLocalFile interface, reset() must be called to signify the end of one file and the start of the next.
Handling Type Conversions
This feature in this OTD operates in the same way as type conversion for the BatchFTP OTD. See Handling Type Conversions for details.
Recommended Practice
To parse records or construct payloads in your Collaboration, we recommend that you use the record-processing BatchRecord OTD together with the BatchLocalFile OTD. Using the two OTD together provides a number of advantages over just using the BatchFTP OTD.
Example 1: Parsing a Large File
For example, you have set up a Collaboration Rule to parse a large file and submit the records to a database or a JMS IQ Manager. If something goes wrong during the parsing process, the whole file needs to be transmitted again from the FTP server.
In contrast, streaming from a local file system can avoid later FTP transfers of the same file in case of error. This approach has the advantage of allowing you to use data streaming and the Resume Reading feature with large files (see Streaming Data Between Components and Resume Reading Feature.
Example 2: Slow, Complex Query
Another scenario could be a case where a slow, complex SQL query is used to retrieve a number of records. The Collaboration Rule packs them into a Payload node using the record-processing OTD then sends them via FTP to an external system. If the FTP transfer fails, the SQL query must be executed again.
In contrast, if the data payload has been stored locally with the BatchLocalFile OTD, the FTP transfer can be repeated without the need to re-execute the SQL query. In such cases, you can also use data streaming and local-file appending.
In both cases, the use of a data-streaming link can significantly reduce the memory requirements compared to the in-memory data-payload transfer used with the BatchFTP OTD.
OTD Limitations
The BatchLocalFile OTD supports mapped drives and NFS mounted drives. It does not, however, support the mapping of the drives. That is, the drive must already be mapped or mounted. The adapter itself does not perform any mapping or mounting.
The OTD supports Universal Reference Identifiers (URIs) but the scheme must be left off, for example,\\drive\directory\file_name.
- © 2010, Oracle Corporation and/or its affiliates