Large File Upload Improvements

Previously, uploading a file using the plugin-driven upload batch process assumed that the content of the entire file was processed as a single unit of work. With this approach, an upload of a large file has often hit various resource limitations and timed out.

As of this release, the following features were introduced to better support large file uploads:

• A new batch control is provided for splitting a large file into multiple smaller files. Refer to batch control F1FSPLIT for more information.
• The existing File Upload batch algorithm entity was enhanced to support algorithms designed to process records in smaller units of work than the entire file. Note that the algorithm can still be designed to process all the records in a file as a single unit of work as before.

The ability to upload large files in batch is required to meet various business requirements.

Steps to Enable

If you have large files to upload, choose one of the new features to better process the file.

To use the batch job that takes a large file and splits it to smaller files, complete these steps

1. Create a batch control using the batch template File Split Template (F1FSPLIT)
2. Include this batch in the scheduler prior to the existing upload process you have for uploading the file.  The file name parameter for this should be configured with an appropriate glob syntax to handle multiple files.

No coding changes are required for this option.

To use the feature where the algorithm can handle chunks of work, coding changes are needed.  Refer to the section "The File Upload Algorithm" in the Uploading Records topic in the Background Processes chapter of the Framework Administrative User Guide. With this option the additional batch process to split the file is not needed.