Sun Adapter for Batch User's Guide

Using FTP Heuristics

Due to the diversified nature of FTP server directory listing styles, that is, the format in which the list of directory entries are returned to the client as the result of a LIST command, the directory listing parser requires FTP server information regarding its specific format, to parse the result. These server specific parameters that provide format information are called FTP heuristics. When these heuristic parameters are set for a specific platform, it is called a directory listing style.

This section provides a general explanation of how the FTP Heuristics feature of the adapter operates, as well as some basic information on how to use it. It also explains the FTP Heuristics configuration parameters for the adapter.

FTP Heuristics

FTP heuristics are a set of parameters that the adapter uses to interact with external FTP daemons on a platform-specific level. The FTP heuristics create and parse both path and file names in the style required by the external system’s platform (operating system).

Platform Selection

The BatchFTP OTD provides a number of built-in heuristics configuration styles. The style is selected from the BatchFTP Adapter Connectivity Map property, FTP ⇒ Directory Listing Style. To select a style for your specific target platform do the following:

  1. From the Connectivity Map, double-click the BatchFTP Adapter. The FTPBatch Adapter Properties Editor appears.

  2. From the Editor’s left pane, under the configuration tree, select FTP. The FTP parameters are now displayed in the Properties pane.

  3. Click the Directory Listing Style field and select a system platform from the drop-down menu. The properties available from the drop-down menu, reflect the styles listed in the FtpHeuristics.cfg file.

The adapter’s FTP Heuristics support the following platform types:

The following systems support Japanese mainframes:

IBM IP stack required for MVS Sequential, MVS GDG, and MVS PD

The FTP Heuristic methods used to interface with MVS Sequential, MVS GDG, and MVS PDS, are designed for FTP servers (at the mainframe) that use the IBM IP stack. Therefore, when you use FTP with an MVS Sequential, MVS GDG, or MVS PDS file system on a mainframe computer, you need to make sure that the FTP server is using an IBM IP stack. If any other IP stack is in place, the FTP heuristics feature will not work or may require modification.

Creating User Defined Heuristic Directory Listing Styles

You can create “user defined” heuristic configurations that allow you to interface with other platforms that are not listed in the Directory Listing Styles. The Batch Adapter includes a mechanism that allows you to configure a set of heuristic properties so that the underlying parser can parse the LIST command result correctly. These properties are described under FTP Heuristics Configuration Parameters.

There are two methods for creating custom user defined directory listing styles:

ProcedureTo Create a Custom Heuristics Configuration File

  1. Using a text editor, create a user defined configuration file containing the property settings required to interface with your target platform. You can do this by copying a section (style) from the FtpHeuristics.cfg file that is similar to the style (platform parameter settings) that you are creating, or you can copy the format provided under Heuristics Configuration File Format.

  2. Save your user defined configuration, as a CFG file, to a safe location on the logical host.

  3. From the BatchFTP Environment properties, select the FTP ⇒ User Defined Heuristics Configuration File property, and enter the location and name of your user defined heuristics configuration file (for example C:\USER_DEFINED_HEURISTICS\UDH.cfg).

  4. From the BatchFTP Connectivity Map properties, select FTP ⇒ User Defined Directory Listing Style, and enter the name of your user-named style (for example MY AS400-UNIX). You are allowed to list one user-named style. This style is now the configured Directory Listing Style, superseding the value of the Directory Listing Style property.

    You can use this method to create multiple user-named styles by adding the styles to your user defined configuration file, and entering the different user defined style names in the Connectivity Map properties for each of your various FTPBatch component adapters.

    You can also create multiple user defined configuration files if necessary, but this requires the creation of additional BatchFTP External Systems in the Environment. If you chose this method, you must copy your Environment components (drag-and-drop) to the correct BatchFTP External System before applying Automap.

Considerations

If you decide to use this method for creating custom user defined heuristic configurations, take note of the following:

ProcedureTo Modify the FTP Heuristics Configuration File

To modify the FtpHeuristics.cfg file to include your user defined heuristic configuration styles, do the following:

  1. The FtpHeuristics.cfg file is contained by the stcbatch.jar file, which is found in the following location:


    <JavaCAPS51>\edesigner\usrdir\modules\ext\batcheway\
    stcbatch.jar

    where JavaCAPS51 is the Sun Java Composite Application Platform Suite install directory.

  2. Unzip stcbatch.jar and locate the FtpHeuristics.cfg file.

  3. Open FtpHeuristics.cfg with a text editor and add your user defined heuristic configuration styles.

ProcedureTo Add User Defined Heuristic Configuration Styles

  1. Copy the User Defined section (or any other section), and paste it to the bottom of FtpHeuristics.cfg.

  2. Rename the section and each property name with your user-defined name or one of the available listings (User Defined1, User Defined2, and so forth). See the example provided under Heuristics Configuration File Format. In this example, the user defined name is MY AS400-UNIX). Only one style with a user-defined name can be specified, but 10 configuration styles can be named as User Defined1-10.

  3. Modify the new section’s properties for your target platform. See FTP Heuristics Configuration Parameters for property descriptions.

  4. Repeat steps 2-4 above to create additional User Defined configurations.

ProcedureTo Repackage the FtpHeuristics.cfg File.

  1. Zip the stcbatch.jar file (including the updated FtpHeuristics.cfg file) and copy stcbatch.jar back to it’s original location.

  2. From the BatchFTP Configuration Map properties, select FTP ⇒ User Defined ⇒ Directory Listing Style, and enter the name of your user-named style (for example MY AS400-UNIX), or you can select any one of the 10 User Defined properties from the Directory Listing Style dropdown list (see Creating User Defined Heuristic Directory Listing Styles).

  3. Your configuration changes will be applied to any Projects that are built and deployed with this Enterprise Designer.

Heuristics Configuration File Format

This example includes two user-named styles (MY AS400-UNIX, and UDH NT 4.0).


#

# --------------------------------------------------------------------------

#   Section:    MY AS400-UNIX

# --------------------------------------------------------------------------

#

MY AS400-UNIX!Commands Supported By FTP 
Server!value=APPE%CWD%DELE%LIST%MKD%NOOP%PASS%QUIT%RETR%RNFR%RNTO
%SITE%STOR%TYPE%USER!set=APPE%CWD%DELE%LIST%MKD%NOOP%PASS%QUIT%RETR
%RNFR%RNTO%SITE%STOR%TYPE%USER

MY AS400-UNIX!Header Lines To Skip!value=0!set=0

MY AS400-UNIX!Header Indication Regex Expression!value=!set=

MY AS400-UNIX!Trailer Lines To Skip!value=0!set=0

MY AS400-UNIX!Trailer Indication Regex Expression!value=!set=

MY AS400-UNIX!Directory Indication Regex Expression!value=!set=

MY AS400-UNIX!File Link Real Data Available!value=No!set=No%Yes

MY AS400-UNIX!File Link Indication Regex Expression!value=!set=

MY AS400-UNIX!File Link Symbol Regex Expression!value=!set=

MY AS400-UNIX!List Line Format!value=Fixed!set=Blank Delimited%Fixed

MY AS400-UNIX!Valid File Line Minimum Position!value=52!set=52

MY AS400-UNIX!File Name Is Last Entity!value=Yes!set=No%Yes

MY AS400-UNIX!File Name Position!value=52!set=52

MY AS400-UNIX!File Name Length!value=0!set=0

MY AS400-UNIX!File Extension Position!value=0!set=0

MY AS400-UNIX!File Extension Length!value=0!set=0

MY AS400-UNIX!File Size Verifiable!value=No!set=No%Yes

MY AS400-UNIX!File Size Position!value=0!set=0

MY AS400-UNIX!File Size Length!value=0!set=0

MY AS400-UNIX!Special Envelope For Absolute Pathname!value=!set=’’

MY AS400-UNIX!Listing Directory Yields Absolute Pathnames!value=No!set=No%Yes

MY AS400-UNIX!Absolute Pathname Delimiter Set!value=///!set=///

MY AS400-UNIX!Change Directory Before Listing!value=Yes!set=No%Yes

MY AS400-UNIX!Directory Name Requires Terminator!value=No!set=No%Yes

#

#

# --------------------------------------------------------------------------

#   Section:   UDH NT 4.0

# --------------------------------------------------------------------------

#

UDH NT 4.0!Commands Supported By FTP
 Server!value=APPE%CWD%DELE%LIST%MKD%NOOP%PASS%QUIT%RETR%RNFR%RNTO%SITE%
STOR%TYPE%USER!set=APPE%CWD%DELE%LIST%MKD%NOOP%PASS%QUIT%RETR%RNFR%RNTO%SITE%
STOR%TYPE%USER

UDH NT 4.0!Header Lines To Skip!value=0!set=0

UDH NT 4.0!Header Indication Regex Expression!value=!set=

UDH NT 4.0!Trailer Lines To Skip!value=0!set=0

UDH NT 4.0!Trailer Indication Regex Expression!value=!set=

UDH NT 4.0!Directory Indication Regex Expression!value=<DIR>!set=<DIR>

UDH NT 4.0!File Link Real Data Available!value=No!set=No%Yes

UDH NT 4.0!File Link Indication Regex Expression!value=\.lnk$!set=\.lnk$

UDH NT 4.0!File Link Symbol Regex Expression!value=!set=

UDH NT 4.0!List Line Format!value=Blank Delimited!set=Blank Delimited%Fixed

UDH NT 4.0!Valid File Line Minimum Position!value=4!set=4

UDH NT 4.0!File Name Is Last Entity!value=Yes!set=No%Yes

UDH NT 4.0!File Name Position!value=4!set=4

UDH NT 4.0!File Name Length!value=0!set=0

UDH NT 4.0!File Extension Position!value=0!set=0

UDH NT 4.0!File Extension Length!value=0!set=0

UDH NT 4.0!File Size Verifiable!value=Yes!set=No%Yes

UDH NT 4.0!File Size Position!value=3!set=3

UDH NT 4.0!File Size Length!value=0!set=0

UDH NT 4.0!Special Envelope For Absolute Pathname!value=!set=

UDH NT 4.0!Listing Directory Yields Absolute Pathnames!value=No!set=No%Yes

UDH NT 4.0!Absolute Pathname Delimiter Set!value=\\\\\\!set=\\\\\\

UDH NT 4.0!Change Directory Before Listing!value=No!set=No%Yes

UDH NT 4.0!Directory Name Requires Terminator!value=No!set=No%Yes

FTP Heuristics Configuration Parameters

This section describes the configuration parameters for the Batch FTP Heuristics located in the FtpHeuristics.cfg file. The Batch FTP Heuristics configuration file (FtpHeuristics.cfg) contains the full set of parameters for each of the platforms listed under Platform Selection.

The FTP Heuristics configuration parameters are as follows:

Commands Supported by FTP Server

Description

Specifies the commands that the FTP server on the given host supports.

Required Values

One or more FTP commands as selected from the list.

Header Lines To Skip

Description

Specifies the number of beginning lines from a LIST command to be considered as a potential header (subject to the Header Indication Regex Expression configuration parameter, discussed below) and skipped.

Required Values

A non-negative integer. Enter zero if there are no headers.

Additional Information

In the example below, the line “total 6” comprises a one-line header.


      total 6
      -rw-r-----   1 ed        usr           110 Apr 15 13:43 AAA
      -rw-r--r--   1 ed        usr           110 Apr 15 13:33 aaa

Header Indication Regex Expression

Description

Specifies a regular expression used to help identify lines which comprise the header in the output of a LIST command. All the declared lines of the header (see Header Lines To Skip, above) must match the regular expression.

Required Values

A regular expression. The default varies based on the FTP server’s operating system. If there is no reliable way of identifying the header lines in the LIST command’s output, leave this parameter undefined.

Additional Information

The regular expression “^ *total” indicates that each line in the header starts with “total,” possibly preceded by blanks, for example:


      total 6
      -rw-r-----   1 ed        usr           110 Apr 15 13:43 AAA
      -rw-r--r--   1 ed        usr           110 Apr 15 13:33 aaa

If the regular expression is undefined, then the header is solely determined by the value of the configuration parameter Header Lines To Skip.

Trailer Lines To Skip

Definition

Specifies the number of ending lines from a LIST command that are to be considered as a potential Trailer (subject to the Trailer Indication Regex Expression) and skipped.

Required Values

A non-negative integer. Enter zero if there are no trailers.

Trailer Indication Regex Expression

Definition

Specifies the regular expression used to help identify lines which comprise the trailer in the output of a LIST command. All the declared lines of the trailer (see Trailer Lines To Skip) must match the regular expression.

Required Values

A regular expression. If there is no reliable way of identifying the trailer lines in the LIST output, then leave this parameter undefined.

Additional Information

If the regular expression is undefined, then the header is determined solely by the value of the Trailer Lines To Skip configuration parameter.

Directory Indication Regex Expression

Definition

Specifies a regular expression used to identify external directories in the output of a LIST command. Directories cannot be retrieved and must be filtered out of the file list.

Required Values

A regular expression. If there is no reliable way of identifying the directory in the LIST output, then leave this parameter undefined.

Additional Information

The regular expression “^ *d” specifies that a directory is indicated by a line starting with the lowercase ”d,’ possibly preceded by blanks, for example:


      drwxr-xr-x   2 ed    usr     2048 Apr 17 17:43 public_html

File Link Real Data Available

Definition

Specifies whether a file may be a file link (a pointer to a file) on those operating systems whereon an FTP server will return the data for the real file as opposed to the content of the link itself.

Required Values

Yes or No.

File Link Indication Regex Expression

Definition

Specifies a regular expression that identifies external file links in the output of a LIST command. File links are pointers to the real file and usually have some visual symbol, such as- >, mixed in with the file name in the output of the LIST command. Only the link name is desired within the returned list.

Required Values

A regular expression. If there is no reliable way of identifying a file link within a LIST output, then leave this parameter undefined.

Additional Information

The regular expression “^ *l” specifies that a file link is indicated by a line starting with the lowercase “l,” preceded possibly by blanks, for example:


      lrwxr-xr-x   2 ed        usr   2048 Apr 17 17:43 p ->       public_html

File Link Symbol Regex Expression

Definition

Specifies a regular expression that parses the external file link name in the output of a LIST command. Only the link name is required for the file list to be returned.

Required Values

A regular expression. If there is no reliable way of identifying a file link within a LIST output, then leave this parameter undefined.

Additional Information

The regular expression “[ ] ->[ ]” defines that a file link symbol is represented by an arrow surrounded by spaces (“ -> “). When parsed, only the file name to the right of the symbol is used.

In the following example, only the public_html would be used, not the “p” character:


     lrwxrwxrwx   2 ed        usr  4 Apr 17 17:43 p -> public_html

List Line Format

Definition

Specifies whether fields in each line are blank delimited or fixed, that is, whether information always appears at certain columns.

Required Values

Blank Delimited or Fixed.

Additional Information

Even though some lines appear to be blank delimited, be wary of certain fields continuing their maximum value when juxtaposed with the next field without any separating blank. In such a case, we recommend you declare the line as “Fixed,” for example:


      -rw-r--r--   1 ed        usr           110 Apr 15 13:33 aaa
      ^^^^^^^^^^   ^ ^^        ^^^           ^^^ ^^^ ^^ ^^^^^ ^^^
           1       2  3         4             5   6  7    8    9

Valid File Line Minimum Position

Definition

Specifies the minimum number of positions (inclusive) a listing line must have in order to be considered as a possible valid file name line.

Required Values

For a Fixed list line format, enter a value equal to the number of columns, counting the first column at the far left as column 1. For a Blank Delimited list line format, enter a value equal to the number of fields, counting the first field on the far left as field 1.

For either case, if no minimum can be determined, set this value to zero (0).

Additional Information

For example, in the Blank Delimited line below, the minimum number of fields is 9:


      -rw-r--r--   1 ed        usr           110 Apr 15 13:33 aaa
      ^^^^^^^^^^   ^ ^^        ^^^           ^^^ ^^^ ^^ ^^^^^ ^^^
           1       2  3         4             5   6  7    8    9
                                                         File Name

Note –

The URL FTP Proxy will fail on ascertaining file names that have leading blanks, trailing blanks, or both.


File Name Is Last Entity

Definition

Specifies whether the file name is the last entity on each line. This allows the file name to have imbedded blanks (however, leading or trailing blanks are not supported).

Required Values

Yes or No.

File Name Position

Definition

Specifies the starting position (inclusive) of a file name.

Required Values

For Fixed list line format, enter the column number, counting the first column on the far left as column 1. For Blank Delimited list line format, enter the field number, counting the first field on the extreme left as field 1.

Additional Information

For Blank Delimited List Line Format only, if the file name has imbedded blanks, then it can span over several fields, for example:


      -rw-r--r--   1 ed        usr           110 Apr 15 13:33 aaa
      ^^^^^^^^^^   ^ ^^        ^^^           ^^^ ^^^ ^^ ^^^^^ ^^^
          1        2  3         4             5   6  7    8    9
                                                         File Name

File Name Length

Definition

Represents the maximum width of a file name; valid only for Fixed list line format.

Required Values

Enter one of the following:

File Extension Position

Definition

Specifies the left-most position of the file extension for those operating systems that present the file name extension separated from the main file name.

Required Values

For Fixed list line format, enter the column number, counting the first column at the extreme left as column 1. For Blank Delimited list line format, enter the field number, counting the first field at the far left as field 1. If there is no file extension (as on UNIX systems) set the value to zero (0).

File Extension Length

Definition

Specifies the maximum width of the file extension; valid only for Fixed list line format.

Required Values

Enter one of the following:

File Size Verifiable

Definition

Specifies whether the file size is verifiable, significant, and accurate within a directory listing.

Required Values

Yes or No. The File Size Stability Check configurable parameter must also be enabled.

Additional Information

Even if the file size field of a listing line is not significant (that is, it is there but only represents an approximate value), the value of this parameter must be No. However, the file size location must still be declared in the File Size Position parameter below to assist determining which line of listing represents a valid file name, for example:


       -rw-r--r--   1 ed        usr           110 Apr 15 13:33 aaa
                                              ^^^
                                              File Size

Note –

Use of this parameter does not guarantee that the file is actually stable. As this feature is intended only for backward compatibility with previous FTP implementations, we do not recommend that you rely on this functionality for critical data.


File Size Position

Definition

Specifies the left-most position in the listing line that represents the size of the file. Even though for some operating systems the value shown might not truly reflect the file size, this position is still important in ascertaining that the line contains a valid file name.

Required Values

A non-negative integer. For Fixed list line format, the position value is the column number (starting with one (1) on the far left). For Blank Delimited, this value represents the field number (starting with one (1) on the far left). If the LIST line does not have a size field, set this parameter to zero (0).

Example


       -rw-r--r--   1 ed        usr           110 Apr 15 13:33 aaa
       ^^^^^^^^^^   ^ ^^        ^^^           ^^^ ^^^ ^^ ^^^^^ ^^^
            1        2  3         4             5   6  7    8    9
                                           File
                                              Size

The following text represents valid number representations of file sizes:


      1234 or 1,234,567 or -12345 or +12345 or ’  1234  ’ or 12/34       or 1,234/56

The following text represents invalid number representations of file sizes (the ^ indicates where the error occurs):


      ’12 34’ or 123,45,678 or 123-456-789 or --123 or 123-
         ^          ^                 ^        ^          ^
      or 12345678901 or any number > 4294967295 or < -2147483647
         ^ (too large)
      or 123.45 or 12AB34 or 0x45 or ,123,456 or 12//34
            ^        ^        ^      ^              ^
      or /123 or 123/ or 12,3/45
         ^          ^        ^

File Size Length

Definition

Specifies the maximum width (number of columns) of the file size field, only valid for Fixed List Line Format.

Required Values

A non-negative integer. For Blank Delimited list line format, set this value to zero (0).

Special Envelope For Absolute Path Name

Definition

Specifies special enveloping characters required to surround an absolute path name (for example, single quotes are used in MVS). Only use a single quote at the start of the directory name.

Required Values

A pair of enveloping characters. Even if the leading and trailing character is identical, enter it twice.

If no enveloping characters are required for an operating system, leave this parameter undefined.


Note –

On UNIX, this parameter is always undefined.


Listing Directory Yields Absolute Path Names

Definition

Specifies whether, when the DIR command is used on a directory name, the resulting file names are absolute.

Required Values

Yes or No.


Note –

On UNIX, this character is always set to No.


Absolute Path Name Delimiter Set

Definition

Specifies any absolute path requiring certain delimiters to separate directory names (or their equivalent) from each other and from the file name.

Required Values

Enter the delimiters for the absolute path, starting from the left, for:

Wherever there is no specific delimiter, use “\0” (backslash zero) to act as a placeholder. Delimiters that are backslashes need to be escaped with another backslash (see Table 56).

Table 56 Delimiters and Path Naming by Platform

OS

Path Name Format

Delimiter Set

       
   

Enter 

UNIX 

/dir1/dir2/file.ext

 

/// 

Windows 

C:\dir1\dir2\file.ext

\\ 

\\ 

\\ 

 

\\\\\\ 

VMS 

disk1:[dir1.dir2]file.ext;1

[.]; 

MVS PDS 

dir1.dir2(member)

\0 

\0.() 

MVS Sequential 

dir1.dir2.filename

\0 

 

\0.. 

MVS GDG 

dir1.dir2.file(version#) (see Note)

\0 

 

\0.. 

AS400 

dir1/file.ext

\0 

 

\0/. 

Above, version # = 0 for current, +1 for new, -1 (-2, -3, etc.) for previous generations.

Change Directory Before Listing

Definition

Determines whether a change directory (cd) command needs to be done before issuing the DIR command to get a listing of files under the desired directory.

Required Values

Yes or No.


Note –

The current Batch Adapter implementation does not rely on this parameter.


Directory Name Requires Terminator

Definition

Determines whether a directory name that is not followed immediately by a file name requires the ending directory delimiter as a terminator (for example, as on VMS).

Required Values

Yes or No.