Batch Processing

Overview

The implementation process involves loading data files for dimensions and fact data into the database. For new implementations, the best practice is to test the interfaces in a logical sequence, in small test cycles..

Once all required data has been loaded and all interfaces have been tested, the scheduled batch cycles that perform different tasks can be used, depending on the frequency involved. The application has INTRADAY processes that are used for ASO, as well as DAILY, WEEKLY, and QUARTERLY batch cycles, each of which performs different tasks, depending on which applications are being used.

Incremental Exports

As described in Supported PROCESS_QUEUE Trigger Values, all incremental export files are controlled by a set of dates that define the beginning and ending range of data to be exported. This data is stored in a configuration table called RSE_EXP_GRP and can be seen in the Manage Configuration screen. Each incremental export has a date associated with the data to be exported. Only data that has a date/time value between the FROM_DT and TO_DT columns of the RSE_EXP_GRP that it is associated with, will be exported when the export file is created.

When testing an application, it is important to realize that if a test export of data is required, you must make sure that data is available to be exported and that the data is associated with a date that is in the range of the export group. If an export runs and does not produce any data in the file, you should check the values of the Export Group to ensure the dates were not set incorrectly.

When you create or process data in the application user interface, want to test the export of that data, you must advance Export Group's date range by running the appropriate export preparation step as described in this chapter. This causes the date range to advance and enables the exporting of the data that is available for exporting. Note that if the Export Group date range is advanced too many times, the data that you want to export may no longer be in the current range for exporting.

You may encounter such issues when using this custom batch process to trigger the execution of exports; however, these issues will not occur once the application is running the batch routines in an automated manner, because the batch processes are only executed once per batch cycle.

Batch Process Flow

Figure 19-1 illustrates the batch process flow.

Figure 19-1 Batch Process Flow

Description of "Figure 19-1 Batch Process Flow"

Here is the process.

The on-premise batch shell script extracts data to files initiated by the customer scheduler.
Merch batch script creates the zip file named RI_RMS_DATA.zip. Additionally, zip files named RI_CE_DATA.zip and RI_MFP_DATA.zip should be created.
You should upload the three zip files to Object Storage using File Transfer Service (FTS).
Once the file is in Object Storage, the cloud service batch processing will retrieve the files from Object Storage after they have passed an anti-virus and malware scan.
The batch load process begins with tasks that
1. Archive the files that have been received in a date/time stamped directory.
2. Perform the presence validation exercise that verifies that all expected files for the customer's subscribed applications in the zipped files. This terminates if any expected files are missing.
3. Clear the previous day's files from the $MMHOME/data/staging directory.
4. Unzip the zip file into the $MMHOME/data/staging directory.

Table 19-1 lists the zip files.

Table 19-1 Supported Zip Files

Zip File Name	Frequency	File Type	Notes
RI_RMS_DATA.zip	Daily	Inbound	All files which start with W_* can be placed in any combination of the RI*zip files.
RI_CE_DATA.zip	Daily	Inbound	All files which start with W_* can be placed in any combination of the RI*zip files.
RI_MFP_DATA.zip	Daily	Inbound	All files which start with W_* can be placed in any combination of the RI*zip files.
ORASE_WEEKLY.zip	Weekly	Inbound	Any inbound file that does not start with W_* and has a weekly frequency can be placed in here.
ORASE_INTRADAY.zip	Intraday	Inbound	Any inbound file that has an intraday frequency can be placed in here.
ORASE_WEEKLY_ADHOC.zip	Weekly	Inbound	Used for instances where AdHoc jobs are run. Any inbound file for that AdHoc job that does not start with W_* and has a weekly frequency can be placed in here.
ORASE_WEEKLY_extract.zip	Weekly	Outbound	Any outbound file that has a weekly frequency will be placed in here.
ORASE_INTRADAY_extract.zip	Intraday	Outbound	Any outbound file that has an intraday frequency will be placed in here.

Configuring Additional Data Files

It may be necessary to configure support for additional data files into AIF, beyond the ORASE_WEEKLY.zip and ORASE_INTRADAY.zip. If this is required, it is possible to configure additional files to go along with those. This section describes how to add support for this.

It is possible to receive additional zip files, with additional suffixes to the existing zip. In this example, the assumption is that a new zip file named ORASE_WEEKLY_IP.zip must be processed when ORASE_WEEKLY.zip is processed. The IP portion of this is what can be configured, as explained below.

Using the Retail AI Foundation Platform's UI, select the Data Management / Manage Configuration menu options. Then, select RSE_CONFIG_CODE as the table to configure. If you want to search for existing configurations, you can enter a search value of "%ZIP" in the PARAM_NAME Search field and select Search. There are no default extensions defined here, but once one has been created, this will display them.

To add a new entry to allow processing this additional file, select the Create Param icon to add a new configuration. You can enter values such as those shown in Table 19-2 into the dialog box.

Table 19-2 Additional Data Files

Field	Value	Notes
APPL_CODE	RSE
PARAM_NAME	ORASE_WEEKLY_ZIP	The format of this is important. If adding an additional file for the Intraday batch, the value would be ORASE_INTRADAY_ZIP instead.
PARAM_CODE	IP	This is the suffix that the additional zip will use.
PARAM_VALUE	Y	Y to enable this, or N to disable this file.
CONFIGURABLE_FLG	Y	Fixed value.
UPDATEABLE_FLG	N	Fixed value.
DESCR	Additional zip for IP files.	Adjust this description so it describes the zip contents/source.

Based on the example values in Table 19-2, a new zip file named ORASE_WEEKLY_IP.zip will now be expected when the ORASE_WEEKLY.zip is expected.

If a file configured as defined above must be temporarily disabled, you can edit the PARAM_VALUE via the UI,so that it has a value of "N" instead of a value of "Y".

File Transmissions

Files required for processing such as the ORASE_WEEKLY.zip file and the ORASE_WEEKLY_IW.zip file must be uploaded to Object Storage using the File Transfer Service (FTS). For uploading to Object Storage, it must include the incoming prefix and file name. For more information on FTS, refer to the “Implementation Tools File Transfer Services” chapter in the Oracle Retail Analytics and Planning Implementation Guide.