Due to the diversified nature of FTP server directory listing styles, that is, the format in which the list of directory entries are returned to the client as the result of a LIST command, the directory listing parser requires FTP server information regarding its specific format, to parse the result. These server specific parameters that provide format information are called FTP heuristics. When these heuristic parameters are set for a specific platform, it is called a directory listing style.
This section provides a general explanation of how the FTP Heuristics feature of the adapter operates, as well as some basic information on how to use it. It also explains the FTP Heuristics configuration parameters for the adapter.
FTP heuristics are a set of parameters that the adapter uses to interact with external FTP daemons on a platform-specific level. The FTP heuristics create and parse both path and file names in the style required by the external system’s platform (operating system).
The BatchFTP OTD provides a number of built-in heuristics configuration styles. The style is selected from the BatchFTP Adapter Connectivity Map property, FTP ⇒ Directory Listing Style. To select a style for your specific target platform do the following:
From the Connectivity Map, double-click the BatchFTP Adapter. The FTPBatch Adapter Properties Editor appears.
From the Editor’s left pane, under the configuration tree, select FTP. The FTP parameters are now displayed in the Properties pane.
Click the Directory Listing Style field and select a system platform from the drop-down menu. The properties available from the drop-down menu, reflect the styles listed in the FtpHeuristics.cfg file.
The adapter’s FTP Heuristics support the following platform types:
UNIX
AS400
AS400-UNIX
HCLFTPD 6.0.1.3
HCLFTPD 5.1
HP NonStop/Tandem
MPE
MSFTPD 2.0
MSP PDS (Fujitsu)
MSP PS (Fujitsu)
NetWare 4.11
Windows NT 3.5
Windows NT 4.0
UNIX
UNIX (EUC-JP)
UNIX (SJIS)
User Defined
User Defined1-10 (See Creating User Defined Heuristic Directory Listing Styles)
VM/ESA
VMS
VOS3 PDS (Hitachi)
VOS3 PS (Hitachi)
VOSK (Hitachi)
User defined name and heuristic configuration (See Creating User Defined Heuristic Directory Listing Styles
When using FTP with an AS400 UNIX (UFS) system, some specific FTP configuration settings are required (see FTP Configuration Requirements for AS400 UNIX (UFS) ).
The following systems support Japanese mainframes:
MSP PDS (Partitioned Data Sets)
MSP PS (Physical Sequential)
VOS3 PDS (Partition Data Sets)
VOS3 PS (Physical Sequential)
VOSK
The FTP Heuristic methods used to interface with MVS Sequential, MVS GDG, and MVS PDS, are designed for FTP servers (at the mainframe) that use the IBM IP stack. Therefore, when you use FTP with an MVS Sequential, MVS GDG, or MVS PDS file system on a mainframe computer, you need to make sure that the FTP server is using an IBM IP stack. If any other IP stack is in place, the FTP heuristics feature will not work or may require modification.
You can create “user defined” heuristic configurations that allow you to interface with other platforms that are not listed in the Directory Listing Styles. The Batch Adapter includes a mechanism that allows you to configure a set of heuristic properties so that the underlying parser can parse the LIST command result correctly. These properties are described under FTP Heuristics Configuration Parameters.
There are two methods for creating custom user defined directory listing styles:
Create a Custom Heuristics Configuration File: You can create a custom user defined heuristics configuration file, listing the style names and parameters in the same format as the FtpHeuristics.cfg file. This file is then located on the logical host. The configuration file location and the style name are then specified in the BatchFTP configuration properties (see To Create a Custom Heuristics Configuration File).
Modify the FTP Heuristics Configuration File: You can open FtpHeuristics.cfg file, add your user-defined style, and repackage the file. This method requires you to unzip a JAR file, add your custom style, and repackage the files (see To Modify the FTP Heuristics Configuration File). In many cases, this method may be more intrusive and cumbersome than the method listed above.
Using a text editor, create a user defined configuration file containing the property settings required to interface with your target platform. You can do this by copying a section (style) from the FtpHeuristics.cfg file that is similar to the style (platform parameter settings) that you are creating, or you can copy the format provided under Heuristics Configuration File Format.
Save your user defined configuration, as a CFG file, to a safe location on the logical host.
From the BatchFTP Environment properties, select the FTP ⇒ User Defined Heuristics Configuration File property, and enter the location and name of your user defined heuristics configuration file (for example C:\USER_DEFINED_HEURISTICS\UDH.cfg).
From the BatchFTP Connectivity Map properties, select FTP ⇒ User Defined Directory Listing Style, and enter the name of your user-named style (for example MY AS400-UNIX). You are allowed to list one user-named style. This style is now the configured Directory Listing Style, superseding the value of the Directory Listing Style property.
You can use this method to create multiple user-named styles by adding the styles to your user defined configuration file, and entering the different user defined style names in the Connectivity Map properties for each of your various FTPBatch component adapters.
You can also create multiple user defined configuration files if necessary, but this requires the creation of additional BatchFTP External Systems in the Environment. If you chose this method, you must copy your Environment components (drag-and-drop) to the correct BatchFTP External System before applying Automap.
If you decide to use this method for creating custom user defined heuristic configurations, take note of the following:
The BatchFTP Connectivity Map property, User Defined Directory Listing Style, supersedes the Directory Listing Style property. When a User Defined Directory Listing Style is specified, it is used as the heuristic configuration for the corresponding BatchFTP Adapter (OTD). To use the Directory Listing Style property value as the applied heuristic style, the User Defined Listing Style property value must be left blank.
Setting the User Defined Directory Listing Style property value to blank (no value) makes the selected Directory Listing Style property value (built-in heuristic configuration) the current enabled style.
At runtime, the user defined heuristics configuration file must exist on the logical host, and possess appropriate permission settings to allow the heuristic configuration parameters to be accessed by the deployed application.
An error message is generated by the BatchFTP OTD when a User Defined Directory Listing Style is specified, but the User Defined Heuristics Configuration File property value is blank, or associated the user defined heuristics configuration file is not accessible or does not contain a corresponding style configuration.
Setting the value of the User Defined Directory Listing Style triggers the loading of the corresponding heuristics configuration file specified by the User Defined Heuristics Configuration File property. If you make changes to the heuristics configuration file, set the User Defined Heuristics Configuration File property before setting the User Defined Directory Listing Style.
To modify the FtpHeuristics.cfg file to include your user defined heuristic configuration styles, do the following:
The FtpHeuristics.cfg file is contained by the stcbatch.jar file, which is found in the following location:
<JavaCAPS51>\edesigner\usrdir\modules\ext\batcheway\ stcbatch.jar |
where JavaCAPS51 is the Sun Java Composite Application Platform Suite install directory.
Unzip stcbatch.jar and locate the FtpHeuristics.cfg file.
Open FtpHeuristics.cfg with a text editor and add your user defined heuristic configuration styles.
Copy the User Defined section (or any other section), and paste it to the bottom of FtpHeuristics.cfg.
Rename the section and each property name with your user-defined name or one of the available listings (User Defined1, User Defined2, and so forth). See the example provided under Heuristics Configuration File Format. In this example, the user defined name is MY AS400-UNIX). Only one style with a user-defined name can be specified, but 10 configuration styles can be named as User Defined1-10.
Modify the new section’s properties for your target platform. See FTP Heuristics Configuration Parameters for property descriptions.
Repeat steps 2-4 above to create additional User Defined configurations.
Zip the stcbatch.jar file (including the updated FtpHeuristics.cfg file) and copy stcbatch.jar back to it’s original location.
From the BatchFTP Configuration Map properties, select FTP ⇒ User Defined ⇒ Directory Listing Style, and enter the name of your user-named style (for example MY AS400-UNIX), or you can select any one of the 10 User Defined properties from the Directory Listing Style dropdown list (see Creating User Defined Heuristic Directory Listing Styles).
Your configuration changes will be applied to any Projects that are built and deployed with this Enterprise Designer.
This example includes two user-named styles (MY AS400-UNIX, and UDH NT 4.0).
# # -------------------------------------------------------------------------- # Section: MY AS400-UNIX # -------------------------------------------------------------------------- # MY AS400-UNIX!Commands Supported By FTP Server!value=APPE%CWD%DELE%LIST%MKD%NOOP%PASS%QUIT%RETR%RNFR%RNTO %SITE%STOR%TYPE%USER!set=APPE%CWD%DELE%LIST%MKD%NOOP%PASS%QUIT%RETR %RNFR%RNTO%SITE%STOR%TYPE%USER MY AS400-UNIX!Header Lines To Skip!value=0!set=0 MY AS400-UNIX!Header Indication Regex Expression!value=!set= MY AS400-UNIX!Trailer Lines To Skip!value=0!set=0 MY AS400-UNIX!Trailer Indication Regex Expression!value=!set= MY AS400-UNIX!Directory Indication Regex Expression!value=!set= MY AS400-UNIX!File Link Real Data Available!value=No!set=No%Yes MY AS400-UNIX!File Link Indication Regex Expression!value=!set= MY AS400-UNIX!File Link Symbol Regex Expression!value=!set= MY AS400-UNIX!List Line Format!value=Fixed!set=Blank Delimited%Fixed MY AS400-UNIX!Valid File Line Minimum Position!value=52!set=52 MY AS400-UNIX!File Name Is Last Entity!value=Yes!set=No%Yes MY AS400-UNIX!File Name Position!value=52!set=52 MY AS400-UNIX!File Name Length!value=0!set=0 MY AS400-UNIX!File Extension Position!value=0!set=0 MY AS400-UNIX!File Extension Length!value=0!set=0 MY AS400-UNIX!File Size Verifiable!value=No!set=No%Yes MY AS400-UNIX!File Size Position!value=0!set=0 MY AS400-UNIX!File Size Length!value=0!set=0 MY AS400-UNIX!Special Envelope For Absolute Pathname!value=!set=’’ MY AS400-UNIX!Listing Directory Yields Absolute Pathnames!value=No!set=No%Yes MY AS400-UNIX!Absolute Pathname Delimiter Set!value=///!set=/// MY AS400-UNIX!Change Directory Before Listing!value=Yes!set=No%Yes MY AS400-UNIX!Directory Name Requires Terminator!value=No!set=No%Yes # # # -------------------------------------------------------------------------- # Section: UDH NT 4.0 # -------------------------------------------------------------------------- # UDH NT 4.0!Commands Supported By FTP Server!value=APPE%CWD%DELE%LIST%MKD%NOOP%PASS%QUIT%RETR%RNFR%RNTO%SITE% STOR%TYPE%USER!set=APPE%CWD%DELE%LIST%MKD%NOOP%PASS%QUIT%RETR%RNFR%RNTO%SITE% STOR%TYPE%USER UDH NT 4.0!Header Lines To Skip!value=0!set=0 UDH NT 4.0!Header Indication Regex Expression!value=!set= UDH NT 4.0!Trailer Lines To Skip!value=0!set=0 UDH NT 4.0!Trailer Indication Regex Expression!value=!set= UDH NT 4.0!Directory Indication Regex Expression!value=<DIR>!set=<DIR> UDH NT 4.0!File Link Real Data Available!value=No!set=No%Yes UDH NT 4.0!File Link Indication Regex Expression!value=\.lnk$!set=\.lnk$ UDH NT 4.0!File Link Symbol Regex Expression!value=!set= UDH NT 4.0!List Line Format!value=Blank Delimited!set=Blank Delimited%Fixed UDH NT 4.0!Valid File Line Minimum Position!value=4!set=4 UDH NT 4.0!File Name Is Last Entity!value=Yes!set=No%Yes UDH NT 4.0!File Name Position!value=4!set=4 UDH NT 4.0!File Name Length!value=0!set=0 UDH NT 4.0!File Extension Position!value=0!set=0 UDH NT 4.0!File Extension Length!value=0!set=0 UDH NT 4.0!File Size Verifiable!value=Yes!set=No%Yes UDH NT 4.0!File Size Position!value=3!set=3 UDH NT 4.0!File Size Length!value=0!set=0 UDH NT 4.0!Special Envelope For Absolute Pathname!value=!set= UDH NT 4.0!Listing Directory Yields Absolute Pathnames!value=No!set=No%Yes UDH NT 4.0!Absolute Pathname Delimiter Set!value=\\\\\\!set=\\\\\\ UDH NT 4.0!Change Directory Before Listing!value=No!set=No%Yes UDH NT 4.0!Directory Name Requires Terminator!value=No!set=No%Yes |
This section describes the configuration parameters for the Batch FTP Heuristics located in the FtpHeuristics.cfg file. The Batch FTP Heuristics configuration file (FtpHeuristics.cfg) contains the full set of parameters for each of the platforms listed under Platform Selection.
The FTP Heuristics configuration parameters are as follows:
Description
Specifies the commands that the FTP server on the given host supports.
Required Values
One or more FTP commands as selected from the list.
Description
Specifies the number of beginning lines from a LIST command to be considered as a potential header (subject to the Header Indication Regex Expression configuration parameter, discussed below) and skipped.
Required Values
A non-negative integer. Enter zero if there are no headers.
Additional Information
In the example below, the line “total 6” comprises a one-line header.
total 6 -rw-r----- 1 ed usr 110 Apr 15 13:43 AAA -rw-r--r-- 1 ed usr 110 Apr 15 13:33 aaa |
Description
Specifies a regular expression used to help identify lines which comprise the header in the output of a LIST command. All the declared lines of the header (see Header Lines To Skip, above) must match the regular expression.
Required Values
A regular expression. The default varies based on the FTP server’s operating system. If there is no reliable way of identifying the header lines in the LIST command’s output, leave this parameter undefined.
Additional Information
The regular expression “^ *total” indicates that each line in the header starts with “total,” possibly preceded by blanks, for example:
total 6 -rw-r----- 1 ed usr 110 Apr 15 13:43 AAA -rw-r--r-- 1 ed usr 110 Apr 15 13:33 aaa |
If the regular expression is undefined, then the header is solely determined by the value of the configuration parameter Header Lines To Skip.
Definition
Specifies the number of ending lines from a LIST command that are to be considered as a potential Trailer (subject to the Trailer Indication Regex Expression) and skipped.
Required Values
A non-negative integer. Enter zero if there are no trailers.
Definition
Specifies the regular expression used to help identify lines which comprise the trailer in the output of a LIST command. All the declared lines of the trailer (see Trailer Lines To Skip) must match the regular expression.
Required Values
A regular expression. If there is no reliable way of identifying the trailer lines in the LIST output, then leave this parameter undefined.
Additional Information
If the regular expression is undefined, then the header is determined solely by the value of the Trailer Lines To Skip configuration parameter.
Definition
Specifies a regular expression used to identify external directories in the output of a LIST command. Directories cannot be retrieved and must be filtered out of the file list.
Required Values
A regular expression. If there is no reliable way of identifying the directory in the LIST output, then leave this parameter undefined.
Additional Information
The regular expression “^ *d” specifies that a directory is indicated by a line starting with the lowercase ”d,’ possibly preceded by blanks, for example:
drwxr-xr-x 2 ed usr 2048 Apr 17 17:43 public_html |
Definition
Specifies whether a file may be a file link (a pointer to a file) on those operating systems whereon an FTP server will return the data for the real file as opposed to the content of the link itself.
Required Values
Yes or No.
Definition
Specifies a regular expression that identifies external file links in the output of a LIST command. File links are pointers to the real file and usually have some visual symbol, such as- >, mixed in with the file name in the output of the LIST command. Only the link name is desired within the returned list.
Required Values
A regular expression. If there is no reliable way of identifying a file link within a LIST output, then leave this parameter undefined.
Additional Information
The regular expression “^ *l” specifies that a file link is indicated by a line starting with the lowercase “l,” preceded possibly by blanks, for example:
lrwxr-xr-x 2 ed usr 2048 Apr 17 17:43 p -> public_html |
Definition
Specifies a regular expression that parses the external file link name in the output of a LIST command. Only the link name is required for the file list to be returned.
Required Values
A regular expression. If there is no reliable way of identifying a file link within a LIST output, then leave this parameter undefined.
Additional Information
The regular expression “[ ] ->[ ]” defines that a file link symbol is represented by an arrow surrounded by spaces (“ -> “). When parsed, only the file name to the right of the symbol is used.
In the following example, only the public_html would be used, not the “p” character:
lrwxrwxrwx 2 ed usr 4 Apr 17 17:43 p -> public_html |
Definition
Specifies whether fields in each line are blank delimited or fixed, that is, whether information always appears at certain columns.
Required Values
Blank Delimited or Fixed.
Additional Information
Even though some lines appear to be blank delimited, be wary of certain fields continuing their maximum value when juxtaposed with the next field without any separating blank. In such a case, we recommend you declare the line as “Fixed,” for example:
-rw-r--r-- 1 ed usr 110 Apr 15 13:33 aaa ^^^^^^^^^^ ^ ^^ ^^^ ^^^ ^^^ ^^ ^^^^^ ^^^ 1 2 3 4 5 6 7 8 9 |
Definition
Specifies the minimum number of positions (inclusive) a listing line must have in order to be considered as a possible valid file name line.
Required Values
For a Fixed list line format, enter a value equal to the number of columns, counting the first column at the far left as column 1. For a Blank Delimited list line format, enter a value equal to the number of fields, counting the first field on the far left as field 1.
For either case, if no minimum can be determined, set this value to zero (0).
Additional Information
For example, in the Blank Delimited line below, the minimum number of fields is 9:
-rw-r--r-- 1 ed usr 110 Apr 15 13:33 aaa ^^^^^^^^^^ ^ ^^ ^^^ ^^^ ^^^ ^^ ^^^^^ ^^^ 1 2 3 4 5 6 7 8 9 File Name |
The URL FTP Proxy will fail on ascertaining file names that have leading blanks, trailing blanks, or both.
Definition
Specifies whether the file name is the last entity on each line. This allows the file name to have imbedded blanks (however, leading or trailing blanks are not supported).
Required Values
Yes or No.
Definition
Specifies the starting position (inclusive) of a file name.
Required Values
For Fixed list line format, enter the column number, counting the first column on the far left as column 1. For Blank Delimited list line format, enter the field number, counting the first field on the extreme left as field 1.
Additional Information
For Blank Delimited List Line Format only, if the file name has imbedded blanks, then it can span over several fields, for example:
-rw-r--r-- 1 ed usr 110 Apr 15 13:33 aaa ^^^^^^^^^^ ^ ^^ ^^^ ^^^ ^^^ ^^ ^^^^^ ^^^ 1 2 3 4 5 6 7 8 9 File Name |
Definition
Represents the maximum width of a file name; valid only for Fixed list line format.
Required Values
Enter one of the following:
An Integer: Positive lengths imply that the file name is right-justified within the maximum field width, and thus leading-blanks are discarded.
Negative Lengths: That is, compared to the absolute length, imply that the file name is left-justified and trailing-blanks are discarded.
Zero (0) Value Length: If the file name is at the end of a file listing line, this value implies that the file name field extends to the end of the line.
For Blank Delimited list line format, this value is usually zero (0). However, if the File Name Length parameter is supplied even though a Blank Delimited list line format is specified, this implies that if the file name field exceeds the given length, then the rest of the List Line data occurs on the following line.
Definition
Specifies the left-most position of the file extension for those operating systems that present the file name extension separated from the main file name.
Required Values
For Fixed list line format, enter the column number, counting the first column at the extreme left as column 1. For Blank Delimited list line format, enter the field number, counting the first field at the far left as field 1. If there is no file extension (as on UNIX systems) set the value to zero (0).
Definition
Specifies the maximum width of the file extension; valid only for Fixed list line format.
Required Values
Enter one of the following:
An Integer
Positive Lengths: Imply that the file extension is right-justified within the maximum field width and therefore leading-blanks are discarded.
Negative Lengths: Imply that the file extension is left-justified and trailing-blanks are discarded (the absolute length is used).
Value of Zero (0): Always for the Blank Delimited list line format.
Definition
Specifies whether the file size is verifiable, significant, and accurate within a directory listing.
Required Values
Yes or No. The File Size Stability Check configurable parameter must also be enabled.
Additional Information
Even if the file size field of a listing line is not significant (that is, it is there but only represents an approximate value), the value of this parameter must be No. However, the file size location must still be declared in the File Size Position parameter below to assist determining which line of listing represents a valid file name, for example:
-rw-r--r-- 1 ed usr 110 Apr 15 13:33 aaa ^^^ File Size |
Use of this parameter does not guarantee that the file is actually stable. As this feature is intended only for backward compatibility with previous FTP implementations, we do not recommend that you rely on this functionality for critical data.
Definition
Specifies the left-most position in the listing line that represents the size of the file. Even though for some operating systems the value shown might not truly reflect the file size, this position is still important in ascertaining that the line contains a valid file name.
Required Values
A non-negative integer. For Fixed list line format, the position value is the column number (starting with one (1) on the far left). For Blank Delimited, this value represents the field number (starting with one (1) on the far left). If the LIST line does not have a size field, set this parameter to zero (0).
Example
-rw-r--r-- 1 ed usr 110 Apr 15 13:33 aaa ^^^^^^^^^^ ^ ^^ ^^^ ^^^ ^^^ ^^ ^^^^^ ^^^ 1 2 3 4 5 6 7 8 9 File Size |
The following text represents valid number representations of file sizes:
1234 or 1,234,567 or -12345 or +12345 or ’ 1234 ’ or 12/34 or 1,234/56 |
The following text represents invalid number representations of file sizes (the ^ indicates where the error occurs):
’12 34’ or 123,45,678 or 123-456-789 or --123 or 123- ^ ^ ^ ^ ^ or 12345678901 or any number > 4294967295 or < -2147483647 ^ (too large) or 123.45 or 12AB34 or 0x45 or ,123,456 or 12//34 ^ ^ ^ ^ ^ or /123 or 123/ or 12,3/45 ^ ^ ^ |
Definition
Specifies the maximum width (number of columns) of the file size field, only valid for Fixed List Line Format.
Required Values
A non-negative integer. For Blank Delimited list line format, set this value to zero (0).
Definition
Specifies special enveloping characters required to surround an absolute path name (for example, single quotes are used in MVS). Only use a single quote at the start of the directory name.
Required Values
A pair of enveloping characters. Even if the leading and trailing character is identical, enter it twice.
If no enveloping characters are required for an operating system, leave this parameter undefined.
On UNIX, this parameter is always undefined.
Definition
Specifies whether, when the DIR command is used on a directory name, the resulting file names are absolute.
Required Values
Yes or No.
On UNIX, this character is always set to No.
Definition
Specifies any absolute path requiring certain delimiters to separate directory names (or their equivalent) from each other and from the file name.
Required Values
Enter the delimiters for the absolute path, starting from the left, for:
Initial (left-most) directory delimiter
Intermediate directory delimiters
Initial (left-most) file name delimiter
Optionally, the ending (right-most) file name delimiter
Wherever there is no specific delimiter, use “\0” (backslash zero) to act as a placeholder. Delimiters that are backslashes need to be escaped with another backslash (see Table 56).
Table 56 Delimiters and Path Naming by Platform
OS |
Path Name Format |
Delimiter Set | ||||
---|---|---|---|---|---|---|
1 |
2 |
3 |
4 |
Enter |
||
UNIX |
/dir1/dir2/file.ext |
/ |
/ |
/ |
/// |
|
Windows |
C:\dir1\dir2\file.ext |
\\ |
\\ |
\\ |
\\\\\\ |
|
VMS |
disk1:[dir1.dir2]file.ext;1 |
[ |
. |
] |
; |
[.]; |
MVS PDS |
dir1.dir2(member) |
\0 |
. |
( |
) |
\0.() |
MVS Sequential |
dir1.dir2.filename |
\0 |
. |
. |
\0.. |
|
MVS GDG |
dir1.dir2.file(version#) (see Note) |
\0 |
. |
. |
\0.. |
|
AS400 |
dir1/file.ext |
\0 |
/ |
. |
\0/. |
Above, version # = 0 for current, +1 for new, -1 (-2, -3, etc.) for previous generations.
Definition
Determines whether a change directory (cd) command needs to be done before issuing the DIR command to get a listing of files under the desired directory.
Required Values
Yes or No.
The current Batch Adapter implementation does not rely on this parameter.
Definition
Determines whether a directory name that is not followed immediately by a file name requires the ending directory delimiter as a terminator (for example, as on VMS).
Required Values