Process Files in Schedule Integrations with a Stage File Action

You can use the stage file action to process files or file references in schedule integrations. The stage file action can process each file downloaded by the FTP Adapter. The stage file action can read (and remove any trailer), write, zip, unzip, encrypt, decrypt, and list files in a staged location known to Oracle Integration.

The stage file action can also read (and remove any trailer) and unzip referenced files in a staged location. The stage file action is similar in functionality to adapters. However, unlike adapters, you do not need to create a connection to use the stage file action. The stage file action has no credential properties and security policies. The stage file action also differs from the File Adapter and FTP Adapter in that it provides the ability to define a file format for read and write operations. For the stage file action to process or act upon files and attachments, they must be available in Oracle Integration. Make the files available in Oracle Integration either using the download operation in the FTP Adapter or consuming the SOAP/REST APIs that return multipart or MTOM attachments. Oracle Integration flows exposed as REST endpoints using the REST Adapter and exposing an interface to accept multipart attachments also automatically stage files in Oracle Integration when the requests are posted to the endpoint.

For example, you may include a stage file action in an integration as follows:

  • Configure an FTP Adapter with the following settings:

    • Download File operation

    • Unzip the File option

    • Input directory and download directory path

  • Because the ZIP file may contain multiple files, configure a for-each action after the FTP Adapter to iterate over repeated elements.

  • To read each file from the input directory, configure a stage file action after the for-each action to read each file from the input directory as follows:

    • In the Expression Builder, specify the file name and directory from which to read the file.

    • Specify the schema file by loading a comma-separated value (CSV) file that contains the data structure.

Note:

  • Do not use special characters in schema names (such as #). Otherwise, integration activation fails.

  • The stage file action only supports the .zip file format. For example, if the input file is .gz format, Oracle Integration cannot unzip or read the contents of the file.

Process File References

You can specify a file reference to process when you select to read the entire file, read a file in segments, or unzip a file.

This feature provides the following benefits:

  • Processes upstream operations that provide a file reference. For example, a REST Adapter connection enables you to download an attachment into an attachment folder. The REST Adapter provides a file reference, but does not provide a file name or directory. The stage file action provides both these options.

    The following operations provide file references:

    • Attachment reference (REST Adapter attachments)

    • Stream reference (REST Adapter invoke response)

    • MTOM (for a SOAP Adapter invoke connection response)

    • FileReference (for an FTP Adapter)

  • Provides a prerequisite to process encrypted content. This means that other adapters do not need to duplicate the decryption operation.

  • Encrypts content as a post-processing operation. Therefore, other adapters do not need to duplicate the encryption operation.

Create Local Files

Note the following details about local file behavior:

  • Local Oracle Integration folders can only be created using a stage file write operation from within the integration.

  • Within the integration scope, the file is available for further processing.

  • Using stage file operations such as read, write, and others enables you to read the contents in the scope in which the file is available.

  • The file is not visible outside the scope in which it was created.

  • You can use stage file write-related variables in mapping operations to point to this virtual file.

  • Once the integration moves outside the scope of file visibility, the local file is deleted.

Configure a Stage File Action

You can configure a stage file action in the Stage File panel.

Restrictions on Using Stage File Action Operations with the File/Attachment Features of the Connectivity Agent

When configuring the stage file action in the Stage File panel, note that there are restrictions on using some operations with the connectivity agent.
  • List File operation: Files uploaded through the connectivity agent are not available with the List File operation.
  • Read Entire File and Read File in Segments operations: Files uploaded through the connectivity agent can only be read with a file reference.
  • Zip File operation: Files uploaded through the connectivity agent are not available with the Zip File operation.
  • Unzip File operation: Files uploaded through the connectivity agent can only be unzipped using a file reference.

Open the Stage File Panel

Note:

Stage file operations cannot share a schema root element that defines the record structure if they are working with different file formats and record structures. Either the target namespace or the root element name must be unique while parsing different file formats and records structures.
  1. Add a stage file action to an integration in either of the following ways:
    • On the side of the canvas, click Actions Integration actions icon and drag the Stage File action to the appropriate location.
    • Click Add icon at the location where you want to add the stage file action, then select Stage File.

  2. Complete the fields on the Basic Info page, and click >.

    The Configure Operation page is displayed.

Configure Stage File Operations

The Configure Operation page enables you to define the file operations to perform. You can select to read an entire file, read files in chunked sizes, unzip a file, write a file, zip a file, or list a file, You use the Expression Builder to build the specific details of your operation. The Expression Builder shows all upstream sources (including assignments, for-each actions, invoke connections, and so on) for you to specify these details. You can also select to encrypt and decrypt files with a Pretty Good Privacy (PGP) authentication key that you configured on the Upload Certificate page. See Upload a Certificate to Connect with External Services.

  1. From the drop-down list, select the stage file operation to perform. The page refreshes to display fields appropriate to your selection. There are restrictions when using the connectivity agent with some stage file action operations. See Restrictions on Using Stage File Action Operations with the File/Attachment Features of the Connectivity Agent.

    Read Entire File

    Table 6-1 - Read Entire File

    Property Description

    Configure File Reference

    • Yes: Select to process an upstream operation that provides a file reference. Once selected, you specify the file reference.

    • No: Select to process a file name.

    Specify the File Reference

    (Appears if you select Yes for the Configure File Reference.)

    Click the field and specify a file reference in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the File Name

    (Appears if you select No for the Configure File Reference.)

    Click the field and specify a file name (for example, /compress:schedule/compress:start Time) in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.

    Note: The file size must be less than 10 MB. For files greater than 10 MB, use the Read File in Segments operation.

    Specify the Directory to read from
    Click the field and specify a directory from which to read files (for example, /compress:schedule/compress:start Time) in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Remove Trailer

    Select to not remove the trailer, to remove the last row, or to remove the last n rows.

    Decrypt Content

    Click the check box, then select the private key of the target location to use to decrypt the file.
    Verify

    If you select Decrypt Content, the Verify check box is enabled and the name changes to Verify Content.

    Select the key to verify the content during decryption.

    1. Select Verify Content.
    2. Select the key to use from the list.

    You upload the PGP keys to use on the Certificates page. See Upload a Certificate to Connect with External Services.

    Read File in Segments

    This option enables you to read files in segments (chunks). Chunking files enables large files to be processed, one logical chunk at a time. A logical chunk (segment) is retrieved from a huge file, enabling the file to stay within memory constraints.

    You can also read large XML files containing repeating elements and multiple namespaces. A use case is provided.

    See Read Large XML Files Containing Multiple Namespaces.

    Note:

    • If you select the Read File in Segments operation, you cannot specify an opaque or JSON schema. If you do, you receive the following runtime error:
      NXSD has infinite loop.
Flow has bad NXSD or bad input file which is causing infinite loop.
Either NXSD is not designed properly or input file is not compatible with
NXSD. Processing of file at stage read was not advancing and looping at same
location in input file.
Please download ICS flow and review NXSD file or inspect input file to ensure
there are no infinite loop.
Stage Read Failed

    Table 6-2 - Read File in Segments

    Property Description

    Configure File Reference

    • Yes: Select to process an upstream operation that provides a file reference.

    • No: Select to process a file name.

    Specify the File Reference

    (Appears if you select Yes for the Configure File Reference.)

    Click the field and specify the file reference in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.

    Specify the File Name

    (Appears if you select No for the Configure File Reference.)

    Click the field and specify a file name in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.

    The Read File in Segments option creates a stage file action that includes a scope part. This enables you to drag actions inside the scope (such as for-each actions, additional stage file actions, and others) for more complex scenarios.

    Specify the Directory to read from
    Click the field and specify a directory from which to read files in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Segment Size

    Enter the segment size. The segment size defaults to 200 records per segment, but can be changed. The minimum value is 200 records and the maximum value is 2000 records. Chunking files enables large files to be processed, one logical chunk at a time. A logical chunk (segment) is retrieved from a huge file, enabling the file to stay within memory constraints.

    Sequential Processing

    Select to read the segments sequentially.
    Remove Trailer

    Select to not remove the trailer, to remove the last row, or to remove the last n rows.

    Write File

    Note:

    • When writing to a stage file, there is no built-in support for headers. You must create headers by explicitly writing them to the file.
    • Multiple stage file action write operations appending data to the same stage file within a loop results in the data being overwritten. This results in data loss.

    Table 6-3 - Write File

    Property Description
    Specify the File Name
    Click the field and specify a file name in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the Output Directory
    Click the field and specify an output directory in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Append

    Optionally select to append records to the existing file.

    Note: If you append JSON or XML content, the final file is invalid XML or JSON.

    Get file attributes value on mapper

    Select this checkbox only if you want to calculate the size and checksum attribute values for your file in the mapper response. If selected, both attributes are enabled in the response mapper underneath ICSFile. By default, the size and checksum attributes are disabled in the mapper response because calculating the values for both attributes is performance-intensive.

    Encrypt Click the check box, then select the public key of the target location to use to encrypt the file.
    Sign

    If you select Encrypt, the Sign check box is enabled and the name changes to Sign Content.

    Select the private PGP key to sign the file.

    1. Select Sign Content.
    2. Select the PGP key to use from the list.

    You upload the PGP keys to use on the Certificates page. See Upload a Certificate to Connect with External Services.

    Zip Files

    Table 6-4 - Zip Files

    Property Description
    Specify the File Name
    Click the field and specify a file name in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the Directory to zip
    Click the field and specify a directory to zip up in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the Output Directory
    Click the field and specify an output directory in which for the ZIP file in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Get file attributes value on mapper

    Select this checkbox only if you want to calculate the size and checksum attribute values for your file in the mapper response. If selected, both attributes are enabled in the response mapper underneath ICSFile. By default, the size and checksum attributes are disabled in the mapper response because calculating the values for both attributes is performance-intensive.

    Unzip File

    Table 6-5 - Unzip File

    Property Description

    Configure File Reference

    • Yes: Select to process an upstream operation that provides a ZIP file reference. Once selected, you specify the file reference and the directory in which to unzip the file.

    • No: Select to process a ZIP file.

    Specify the File Reference

    (Appears if you select Yes for the Configure File Reference.)

    Click the field and specify a file reference in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.

    Specify the Zip File Name

    (Appears if you select No for the Configure File Reference.)

    Click the field and specify a ZIP file name to read in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the Zip File Directory
    Click the field and specify a directory in which to unzip the file in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the Directory to Unzip
    Click the field and specify the directory to unzip in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Get file attributes value on mapper

    Select this checkbox only if you want to calculate the size and checksum attribute values for your file in the mapper response. If selected, both attributes are enabled in the response mapper underneath ICSFile. By default, the size and checksum attributes are disabled in the mapper response because calculating the values for both attributes is performance-intensive.

    Encrypt File

    Table 6-6 - Encrypt File

    Property Description
    Specify the File Reference
    Click the field and specify a file reference in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the File Name
    Click the field and specify a file name in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the Output Directory
    Click the field and specify an output directory in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Get file attributes value on mapper

    Select this checkbox only if you want to calculate the size and checksum attribute values for your file in the mapper response. If selected, both attributes are enabled in the response mapper underneath ICSFile. By default, the size and checksum attributes are disabled in the mapper response because calculating the values for both attributes is performance-intensive.

    Specify PGP Key to encrypt file Select the private key of the target location to use to encrypt the file. You can encrypt a file up to 1 GB in size.

    When using the mapper, note that encrypt is visible as an element for mapping.
    Sign Content Select the public PGP key to sign the file.
    1. Select the check box.
    2. Select the PGP key to use to sign the file during encryption.

    You upload the PGP keys to use on the Certificates page. See Upload a Certificate to Connect with External Services.

    Decrypt File

    Table 6-7 - Decrypt File

    Property Description
    Specify the File Reference
    Click the field and specify a file reference in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the File Name
    Click the field and specify a file name in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the Output Directory
    Click the field and specify an output directory in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Get file attributes value on mapper

    Select this checkbox only if you want to calculate the size and checksum attribute values for your file in the mapper response. If selected, both attributes are enabled in the response mapper underneath ICSFile. By default, the size and checksum attributes are disabled in the mapper response because calculating the values for both attributes is performance-intensive.

    Specify PGP Key to decrypt file Select the private key of the target location to use to decrypt the file. You can decrypt a file up to 1 GB in size.

    When using the mapper, note that decrypt is visible as an element for mapping.
    Verify Content Select the PGP key to verify the file.
    1. Select the check box.
    2. Select the PGP key to use to verify the file during decryption.

    You upload the PGP keys to use on the Certificates page. See Upload a Certificate to Connect with External Services.

    List File

    Table 6-8 - List File

    Property Description
    Specify the Directory to List Files from
    Click the field and specify a directory from which to list files in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.
    Specify the File Pattern to use

    Specify the pattern of the file name to transfer to the output directory. Click the ? icon for the list of supported patterns.

    List Files Recursively

    Select to list the files recursively.
    Get file attributes value on mapper

    Select this checkbox only if you want to calculate the size and checksum attribute values for your file in the mapper response. If selected, both attributes are enabled in the response mapper underneath ICSFile. By default, the size and checksum attributes are disabled in the mapper response because calculating the values for both attributes is performance-intensive.

  2. When complete, click > .

    The Schema Options page is displayed if you selected a read or write stage file operation.

Define the Schema File

The Schema Options page enables you to define a schema. This page is displayed if you selected a read or write stage file operation.

  1. Provide the following details.

    Property Description
    Do you want to specify the structure for the contents of the file?

    Select Yes to define a schema format to use for the files to transfer. Select No if a schema is not required and you want to send opaque files (for example, a GIF or PNG file).

    Which one the following choices would be used to describe the structure of the file contents?
    Select an option:

    • Sample delimited document (erg. CSV): Select to create a new schema from a CSV file. On a subsequent page of this panel, you are prompted to select the CSV file from which to create the schema.

    • XML schema (XSD) document: Select to create a new schema from an XML schema (XSD) file. You can upload a ZIP file with nested XML schema files.

    • Sample XML document (Single or No Name Space): Select to create a new schema from a sample XML file with a single or no name space.

    • Sample JSON document: Select to create a new schema from a JSON file. This option only appears if you selected the Read Entire File or Write File operation.

  2. When complete, click >.

Define the Schema Format

Based on your selection on the Schema Options page, the Format Definition page enables you to select the file from which to create a schema. This page is displayed if you selected a read or write stage file operation.

  1. Follow the instructions based on the schema option you selected:

    Note:

    Oracle does not certify schemas that were manually updated with undocumented annotations. Use discretion if specifying these schemas. For example, runtime errors can occur if you use Oracle SOA Suite schemas that are annotated with constructs that are undocumented and not certified in Oracle Integration 3.

    Table 6-9 - Sample Delimited Document (e.g. CSV)

    Element Description

    • Select the Delimited Data File

    Drag and drop the delimited comma-separated value (CSV) file from which to create the schema file. The content of the file is then displayed at the bottom of the page.

    • Enter the Record Name

    Enter the record name. This becomes the parent element in the created schema file for the record names selected as column headers from the CSV file.

    • Enter the Recordset Name

    Enter the recordset name. This becomes the root element of the created schema file.

    • Select the Field Delimiter

    Select one of the following supported file delimiter options:

    • Single space

    • Comma

    • Semicolon

    • Tab

    • Pipe (for example, Name|City|Country)

    • Character Set

    Select a character set. The selected value is used as the encoding format while reading the sample data file.

    • Optionally Enclosed By

    This value causes occurrences of the selected delimiter to be ignored during processing. For example, when processing the following record:

    Fred,"2 Old Street, Old Town,Manchester",20-08-1954,0161-499-1718

    If the selected Field Delimiter is “,” and the Optionally Enclosed By value is quot; (“), then the value 2 Old Street, Old Town,Manchester is treated as a single record column.

    • Use the First Row as Column Headers

    Displays by default the first row of the selected CSV file as the column headers.

    • Detach

    Select to edit the CSV file in a separate window.

    • Mark All As Optional

    Select to mark elements as optional in the schema file. By default, all elements are mandatory. You can also select the data type (for example, string, byte, integer, and so on) to use for each column in the table and mark specific elements as optional. While this option enables you to select all elements as optional, you must have at least one mandatory element to validate this page. This check box provides a convenient method to select the majority of elements as optional.

    Table 6-10 - XML Schema (XSD) Document

    Element Description

    • Select a New File

    Drag and drop an existing XML schema file or schema archive file from the file system.

    • Selected File Name

    Displays the selected schema file name.

    • Select the Element Name

    Select the schema element. This field is displayed after the XML schema file is selected. The element name is treated as the root element in the uploaded schema file.

    • Select Repeating Batch Element
    Click the field and specify the repeating node in the schema to support payload chunking in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.

    Table 6-11 - Sample XML Document (Single or No NameSpace)

    Element Description

    • Select a New File

    Drag and drop a sample XML document from the file system. The file must contain no namespace or a single namespace.

    • Selected File Name

    Displays the selected schema file name.

    • Select the Schema Element

    Select the schema element. This field is displayed after the sample XML file is selected. The element name is treated as the root element in the uploaded schema file.

    • Select Repeating Batch Element
    Click the field and specify the repeating node in the schema to support payload chunking in any of the following ways:
    • Entering manually.
    • Selecting from the list.
    • Clicking Switch to Developer View Switch view icon to open the Sources tree.

    Table 6-12 - Sample JSON document

    Element Description

    • Select a New File

    Drag and drop a sample JSON document from the file system.

    • Selected File Name

    Displays the selected schema file name.

    • Select the Schema Element

    Select the schema element. This field is displayed after the JSON file is selected. The element name is treated as the root element in the uploaded schema file.

  2. Complete the fields, and click >.

  3. Review your selections on the Summary page, and click Done.

Review the Stage File Action in the Integration Canvas

Once design is complete, the stage file action is displayed in the integration canvas.

  • If you designed a stage file action with a Read File in Segments operation, a scope portion is created. Place your cursor over the icon to display a message indicating that segmentation (chunking) is enabled.

    You can drag additional actions into the scope part of the stage file action to perform tasks. For example, you can add a for-each action to the stage file action to process the segmented chunks of large files one record at a time. You can also add child stage file actions to perform further processing on each of the chunks. However, you cannot configure additional chunking on the child stage file action.


    The

    Note:

    If a variable is declared outside of a loop of a stage file action that was configured with the Read File in Segments operation and updated within the stage file action loop, the last updated value for that variable is available outside of the loop. In such a case, the DOP (degree of parallelism) is set to 1 to avoid concurrency issues.

  • If you designed a stage file action with an operation other than a Read File in Segments operation, a scope portion is not included.

Complete your integration design and configure any business identifiers for tracking fields in messages (including file storage-related parameters).

Related Documentation
The following sections provide examples of using a stage file action in an integration: