Upload and Transforming a File

To upload a file in the ORMB application, you should use the File Transformation and Upload (C1-FTRAN) batch which uploads either an XML file that contains record(s) in compliance with the ORMB service schema or simply transforms and uploads the file in customer format, either in XML or CSV or PSV or JSON format. The C1-FTRAN batch uses the Record Transformation Algorithm to transform the file records into ORMB conform service schema and upload the same in the ORMB system. The uploaded file details are logged with the FILE_​ID field in the CI_​FILE_​REQUEST table with either Complete or Pending status.

Note:

  • File Record Transformation is only eligible for XML or CSV or PSV or JSON or Fixed Formats.

  • The file will only be uploaded with Pending status when the all of the records in the file are not uploaded due to failure, else the file will be uploaded with Complete status.

  • Every file will contain records for only a single File Request Type.

The C1-FTRAN batch is required to read the file(s) on the SFTP server at the given file path (passed as the soft parameter). If a File Validation Algorithm is mapped to the corresponding file request type then header, footer, and checksum validation can be performed. This batch reads one file at a time and adds a token on all of the records of the file. Once all the file records are read with a token, you must then create a ThreadWorkUnit for every single token, i.e. Number of records equal to the Number of Work-units. Before logging the file details and creating a ThreadWorkUnit, the file transformation batch validates the following conditions,

Condition Result
If the File Path is invalid Batch is terminated with ERROR status without logging any file details in the CI_​FILE_​REQUEST table.
If the File Path of Error Log is invalid Batch is terminated with an ERROR status.
If the File does not exists in the path The C1-FTRAN batch is terminated with an ERROR status without logging any file details within the CI_​FILE_​REQUEST table.
If the File name Extension does not matches with the one provided in Request Type Configuration Batch will be terminated with ERROR status without logging any file details in the CI_​FILE_​REQUEST table.
If the file size exceeds the limit of 2GB Batch is terminated with COMPLETE status. The file details are logged with an ERROR status and in the CI_​FILE_​REQUEST table.
If the File Request Type is incorrect Batch is terminated with an Error status.
If encrypted files are uploaded, File Decryption process gets executed on those files. If File Decryption process fails Batch is terminated with a COMPLETE status. The file and error details are logged with an ERROR status and in theCI_​FILE_​REQUEST table.
If the File Header (first line in the file) does not exist Batch is terminated with a COMPLETE status. The file and error details are logged with an ERROR status and in theCI_​FILE_​REQUEST table.
If the File Footer (last line in the file) does not exist Batch is terminated with a COMPLETE status. The file and error details are logged with an ERROR status and in theCI_​FILE_​REQUEST table.
If the File Header and File Footer are excluded, and the number of records do not match with number of lines in the file Batch is terminated with a COMPLETE status. The file and error details are logged with an ERROR status and in theCI_​FILE_​REQUEST table.
If the uploaded file already exists in the ORMB system Batch will be terminated with an ERROR status. The file and error details are logged with a REJECT status and in theCI_​FILE_​REQUEST table.
If the checksum validation (for example, search for <FILENAME>.checksum parameter to retrieve the checksum value) fails Batch will be terminated with an ERROR status. The file and error details are logged with a REJECT status and in theCI_​FILE_​REQUEST table.

A list of ThreadWorkUnit is created by tokenizing the number of records in the corresponding file. Every single token is a single file record with unique REQUEST_​ID identifier in CI_​FILE_​REQUEST_​DETAIL table with PENDING status. All the files on SFTP server having same file extension corresponding to that configured for the given file record type will be read and uploaded in ORMB system. There are two fields to capture its corresponding XML (Payload) record data. Either of this field will be populated.

  • REQUEST with CLOB DataType - This field is populated when the record with File Request Type has the configuration of File Size Greater than 32KBas TRUE.

  • BO_​DATA_​AREA with VARCHAR2(32000) DataType - This will be populated when, request with File Request Type having configuration of Record Size Greater than 32KB as FALSE.

This handling is for better performance while uploading and processing file request data.

Note:

  • File that already exists in ORMB system will be ignored.

  • Java exceptions/Unknown errors will not be handled.

This batch supports both single-thread and multi-threaded execution using File Atomicity flag on File Request Type. Using the errorLogFilePath (Execute batch using File Chunks) soft-parameter this batch supports two ways for creating ThreadWorkUnits that are required for batch execution.

  • Fetch and create ThreadWorkUnits for all these number of requests in PENDING or RETRY status for the given File Request Type from the CI_​FILE_​REQUEST_​DETAIL table.

    Field Name Field Description
    File ID Used to specify a unique identifier for a file.
    File Name Used to specify the request object that is to be processed.
    Request Type Used to specify the file request type of the request object to be processed.
    Request Object Used to specify the payload / dataline in the application.
    File Retry Used to specify the value for the file retry field to skip the processed record in case the batch has been aborted.
    Duplicate Used to specify the value for the duplicate field to skip processing the request, if there are multiple records with the same identifier in a same file.
    ID Used to specify the value for the ID, specific to payment stage upload and transaction business services.
    XPath Used to specify the value for the XPath, specific to payment stage upload and transaction business services file request transformation.
    Field Name Used to specify the field name specific to payment stage upload and transaction business services file request transformation.

  • Fetch and create ThreadWorkUnits for all these number of request(s) in PENDING or RETRY status for the given File Request Type from CI_​FILE_​REQUEST_​DETAIL table.

    • Number of files are created in proportion to the number of threads, for example, if the number of threads are 100, then 100 files are created.

    • These files will be created at the same file path on the SFTP server from where the uploaded files are being read.

    • All these newly created files will have approximately equal number of requests i.e. if there are 1000 requests, then, each file will have 10 requests or records.

    • ThreadWorkUnit will have supplementary data only for,

      Field Name Field Description
      ID Used to specify the value for the ID, specific to payment stage upload and transaction business services.
      XPath Used to specify the value for the XPath, specific to payment stage upload and transaction business services file request transformation.
      Field Name Used to specify the field name specific to payment stage upload and transaction business services file request transformation.
      Note:

      These new temporary files are deleted after all the requests in the file are uploaded in the ORMB system.

File Upload Interface Master Configuration
Attribute Name Attribute Description
Validate Checksum Checksum validation is performed before reading and staging the file details in the ORMB system.
Audit Log Required Individual file request status (PENDING / PROCESSED / ERROR / SKIPPED / INPROGRESS / RETRY / IGNORE / RETRYLIMITEXCEED) transition will be logged in ORMB system
Archive File In-case there is no file validation failure, file is moved to Archive File Location else file is moved to Archive Error File location.
File Encryption In-case if the file encryption returns true, then the batch retrieves the encrypted file on SFTP server and decrypts the file using the File Decryption algorithm.
File Request Type Configuration
File Request Type (Attribute Name) Attribute Description
File Transformation Required

  • If the file transformation required attribute is true, the file request transformation is done using Request Transformation Algorithm i.e. (request in either CSV or PSV or XML or JSON to ORMB conform service schema).

  • If the attribute file transformation required is false, the payload that is provided within a file is used to invoke the corresponding service.
File Transformation Algorithm

  • The file transformation algorithm is used for transforming the file request into an XML request, which is in compliance with the ORMB service schema.

  • A sample transformation algorithm C1-FRTA is provided. If you want to use this algorithm, then provide the Map Field XPath for every configured field, with File Segment Type as Field Detail in the Transformation Details section on the File Request Type configuration UI.
File Atomicity

  • If the file atomicity attribute is true, then the C1-FTRAN batch is executed in a single thread mode, i.e. for All or None Transactions.

  • If the file atomicity attribute is false, then the C1-FTRAN batch is executed in multi-threaded mode. Multi-threading is based on number of requests within the file(s). For the multi-threaded batch we use the standard commit strategy.
File Upload and Process

  • If the file upload and process attribute is true in the single-threaded mode, then for a single record transaction failure every other record transaction performed will be roll-backed. The file status will be marked as Error with its failure updates placed in CI_​FILE_​REQUEST table. All the corresponding file requests will be updated with an Error status in the CI_​FILE_​REQUEST_​DETAIL table.

  • If the file upload and process attribute is true in multi-threaded batch, then for a single record transaction failure only that record transaction will be roll-backed. File status will be marked as Complete with its failure updates placed in CI_​FILE_​REQUEST table. Only particular file requests are updated with Errorstatus in the CI_​FILE_​REQUEST_​DETAIL table.

  • If the file upload and process attribute is false, then the requests in the file are uploaded in the application with the PENDING status.
Skip Duplicates

  • If the skip duplicates attribute is true, then all the duplicate records within a corresponding file will be skipped while uploading in the ORMB staging.

  • If the skip duplicates attribute is false, then all the available records within a file will be uploaded in ORMB staging.
Service Logs Required

  • If the service logs required attribute is true, then service execution details corresponding to the individual request will persist in CI_​FILE_​REQUEST_​DTL_​SERVICE table.

  • For every individual request detail in this service table we can capture its identifier details only if we configure a FK Reference for the corresponding Service in a File Request Type.

  • In the file request detail dashboard these identifier details are used to navigate to the entity specific user-interface.
File Header Required

  • If the file header required attribute is true, then the first row data is passed as a header string to File Validation Algorithm.

  • If the file header required attribute is false, then the first row data will be considered as another request data to be persisted in the ORMB system.
Header XML Tag

  • The header xml tag is required, if the File Format is XML and the File Transformation Required attribute is set to true along with the File Header Required attribute.

  • The header xml tag contains the name of header tag that is used in XML.
File Footer Required

  • If the file footer required attribute is true, the last row data is passed as a header string to File Validation Algorithm.

  • If the file footer required attribute is false, the last row data is considered as another request data which is present in the ORMB system.
Footer XML Tag

  • The footer xml tag is required, if the File Format is XML , File Transform attribute is set to true and the File Footer attribute is set to true.

  • The footer xml tag contains the name of footer tag that is used in XML.
Root XMl Tag

  • The root xml tag is required, if the File Format is XML, and the File Transform attribute is set to true.

  • The root xml tag will have the name of root tag being used in XML.

You can also specify the following soft parameters while executing this batch:

Parameter Name Description Mandatory (Yes or No)
File Path Used to specify the relative path where the file gets uploaded. This path will always begin with either @SHARED_​DIRor @INSTALL_​DIR

  • @SHARED_​DIR is the configured path of the shared directory.

  • @INSTALL_​DIR is the configured path of the installation directory with a property named spl.runtime.environ.SPLEBASE within the spl.properties file.

 Yes
File Name Used to specify the name of the file (for example CustomerOnboard). If this value is left blank then all the files with the file request type having a corresponding file extension are picked from the given file path. The File Name parameter also supports inclusion of the partial value,

  • If the filename isCUST% then the filename that ends with CUST must be considered.

  • If the filename is%CUST then the filename that starts with CUST must be considered.

  • If the filename is%CUST% then the filename that contains CUST must be considered.

 Yes
File Request Type Used to specify the file request type which is uploaded in the system. Yes
Error Log File Path Used to specify the CSV file (along with the error logs) that are stored at a provided relative path. This path will always start with either @SHARED_​DIR or @INSTALL_​DIR.

  • @SHARED_​DIR is the configured path of the shared directory.

  • @INSTALL_​DIR is the configured path of the installation directory with a property named spl.runtime.environ.SPLEBASE within the spl.properties file.
Execute Batch using File Chunks Used to specify the execution behavior of the batch. If the parameter value is Y then the batch is executed using file chunks. No
Override Maximum Errors Used to specify the value to override the maximum number of errors that are allowed before the execution is terminated. No
Thread Pool Name Used to specify the thread pool name on which you want to execute the batch. No
Note: If the File Transformation and Upload (C1- FTRAN) batch fails or aborts due to some reason, then you can restart the batch over and over again with the same set of parameters.

Post Execution Check / Clean Up

Once the batch is successfully completed, the logged file details are updated with the COMPLETE status. If the File Upload and Process attribute value for the corresponding File Request Type is set as true, then all of those requests in that file are processed and uploaded with the execution status as either PROCESSED or ERROR or SKIPPED or INPROGRESS or RETRY or IGNORE or RETRYLIMITEXCEED