5 Batch Orchestration
This chapter describes the tools, processes, and implementation considerations for configuring and maintaining the batch schedules used by the Retail Analytics and Planning. This includes nightly, weekly, and ad hoc batch cycles added in the Process Orchestration and Monitoring (POM) tool for each of the RAP applications.
Overview
All applications on the Retail Analytics and Planning have either a nightly or weekly batch schedule. Periodic batches allow the applications to move large amounts of data during off-peak hours. They can perform long-running calculations and analytical processes that cannot be completed while users are in the system, and close out the prior business day in preparation for the next one.
To ensure consistency across the platform, all batch schedules have some level of interdependencies established, where jobs of one application require processes from another schedule to complete successfully before they can begin. The flow diagram below provides a high-level view of schedule dependencies and process flows across RAP modules.
Figure 5-1 Batch Process High-Level Flow
![Batch Process High-Level Flow Batch Process High-Level Flow](img/batch-flow-swimlane.png)
The frequency of batches will vary by application. However, the core data flow through the platform must execute nightly. This includes data extraction from RMS by way of RDE (if used) and data loads into Retail Insights and AI Foundation.
Downstream applications from Retail Insights, such as Merchandise Financial Planning, may only execute the bulk of their jobs on a weekly basis. This does not mean the schedule itself can run weekly (as MFP batch has been run in previous versions); those end-of-week processes now rely on consumption and transformations of data happening in nightly batches. For example, Retail Insights consumes sales and inventory data on a daily basis. However, the exports to Planning (and subsequent imports in those applications) are only run at the end of the week, and are cumulative for all the days of data up to that point in the week.
For this reason, assume that most of the data flow and processing that is happening within the platform will happen every day and plan your file uploads and integrations with non-Oracle systems accordingly.
While much of the batch process has been automated and pre-configured in POM, there are still several activities that need to be performed which are specific to each implementation. TheTable 5-1 table summarizes these activities and the reasons for doing them. Additional details will be provided in subsequent sections of this document
Table 5-1 Common Batch Orchestration Activities
Activity | Description |
---|---|
By default, most batch processes are enabled for all of the applications. It is the implementer’s responsibility to disable batches that will not be used by leveraging the Customer Modules Management screen in Retail Home. |
|
The POM application supports external integration methods including external dependencies and process callbacks. Customers that leverage non-Oracle schedulers or batch processing tools may want to integrate POM with their existing processes. |
|
Schedules in POM must be given a start time to run automatically. Once started, you have a fixed window of time to provide all the necessary file uploads, after which time the batch will fail due to missing data files. |
|
It is possible to export the batch schedules from POM to review process/job mappings, job dependencies, inter-schedule dependencies, and other details. This can be very useful when deciding which processes to enable in a flow or when debugging fails at specific steps in the process, and how that impacts downstream processing. |
Initial Batch Setup
As discussed in Implementation Tools, setting up the initial batches requires access to two tools: POM and Retail Home. Refer to that chapter if you have not yet configured your user accounts to enable batch management activities.
When using POM and Customer Modules Management, it is required to enable or disable the application components based on implementation needs. The sections below describe which modules are needed to leverage the core integrations between applications on the platform. If a module is not present in the below lists, then it is only useful if you are implementing that specific application and require that specific functionality. For example, most modules within Retail Insights are not required for the platform as a whole and can be disabled if you do not plan to implement RI.
After you make changes to the modules, make sure to synchronize your batch schedule in POM following the steps in Implementation Tools. If you are unsure whether a module should be enabled or not, you can initially leave it enabled and then disable jobs individually from POM as needed.
Common Modules
The following table lists the modules that are used across all platform implementations. These modules process core foundation data, generate important internal datasets, and move data downstream for other applications to use. Verify in your environment that these are visible in Customer Modules Management and are enabled.
Table 5-2 RAP Common Batch Modules
Application | Module | Usage Notes |
---|---|---|
RAP |
RAP_COMMON > BATCH |
Contains the minimum set of batch processes needed to load data into the platform and prepare it for one or more downstream applications (such as MFP or AI Foundation) when RI is not being implemented. This allows implementers to disable RI modules entirely to minimize the number of active batch jobs. |
RAP |
RAP_COMMON > RDXBATCH |
Contains the batch processes for extracting data from RI to send to one or more Planning applications. |
RAP |
RAP_COMMON > ZIP_FILES |
Choose which ZIP packages must be present before the nightly batch process begins. The batch only looks for the enabled files when starting and fails if any are not present. |
RAP |
RAP_COMMON > SIBATCH |
Choose which input interfaces will be executed for loading flat files into RAP. Disabled interfaces will not look for or load the associated file even if it is provided. |
RAP |
RAP_COMMON > SICONTROLFILES |
Choose which input files must be present before the nightly batch process begins. Active control files mark a file as required, and the batch fails if it is not provided. Inactive control files are treated as optional and are loaded if their associated job is enabled, but the batch will not fail if they are not provided. |
RAP |
HISTORY > BATCH |
Choose which functional areas
require history data or data corrections, either now or after you start batches. Refer to the Module Name column for the matching
CSV file names and enable the modules where you plan to provide data. This affects the |
RAP | RAP_COMMON > EGRESS | These modules must be disabled for most implementations, unless you are an existing v19 or earlier customer that is working with Oracle to migrate your environments (also known as a Core Update for Planning applications). |
Figure 5-2 Example of RAP Common Modules
![Example of RAP Common Modules Example of RAP Common Modules](img/exmpl-rap-common-module.png)
If you are implementing some or all of AI Foundation, then there are some additional modules to review. These modules may or may not be required, as they are based on which interface files you plan to load as part of the nightly batch process.
Application | Module | Usage Notes |
RAP | AIF>CONTROLFILES |
Enable only the files you plan to send based on your AI Foundation implementation plan. Refer to the Interfaces Guidefor details. This is only used for non-foundation or legacy DAT files. This directly affects what files are required to be in the zip files, when an RI Daily Batch is executed. Required files that are missing will fail the batch. |
RAP | AIF > BATCH | Enable RSP_REQUIRED if you are implementing any AI
Foundation module, plus others based on your application needs. For
example, if you are loading sales then enable
RSP_SALES . If you are loading inventory, then enable
RSP_INVPOS , and so on.
|
After setting up the common modules and syncing with POM, ensure that certain critical batch processes in the AIF DATA schedule (which is used by all of RAP) are enabled in Batch Monitoring. This can be used as a check to validate the POM sync occurred:
-
RESET_ETL_THREAD_VAL_STG_JOB
-
TRUNCATE_STAGE_TABLES_JOB (unless you are using RDE with RMFCS v22+, then this must be disabled instead)
-
DELETE_STATS_JOB
-
RI_UPDATE_TENANT_JOB
Some of these jobs begin in a disabled state in POM (depending on the product version) so the POM sync should ensure they are enabled. If they are not enabled after the POM sync, be sure to enable them before attempting any batch runs.
Additionally, there are certain jobs that must remain disabled unless advised to enable them by Oracle. Make sure the following jobs are disabled in the AIF DATA schedule after syncing with POM:
-
OBIEE_CACHE_CLEAR_JOB
-
ODI_LOG_EXTRACTOR_JOB
-
ODI_LOG_LOADER_JOB
-
If you are not providing a file named
RA_SRC_CURR_PARAM_G.dat
in all ZIP uploads, disableBATCH_VALIDATION_JOB
,RA_SRC_CURR_PARAM_G_COPY_JOB
, andRA_SRC_CURR_PARAM_G_STG_JOB
-
If you are using stock ledger in RI, only one of the following can be used and the other must be disabled:
W_RTL_STCKLDGR_SC_LC_MH_F_GREG_JOB
,W_RTL_STCKLDGR_SC_LC_MH_F_JOB
RI Modules
The following table lists modules within the Retail Insights offer codes which may be used if you are implementing any part of Retail Insights.
Table 5-3 RI Batch Modules
Application | Module | Usage Notes |
---|---|---|
RI | RMI > BATCH | Enable some or all of the modules in this section to use Merchandising data in Retail Insights. |
RI | RCI > BATCH | Enable some or all of the modules in this section to use Customer and Consumer data in Retail Insights. |
RI |
RMI > CONTROLFILES RCI > CONTROLFILES |
Enable control files based on which interface files will be coming on the nightly batch uploads. Enabling control files marks files as required and the batch will fail without them. Leaving these disabled will still process the files, they will just be considered optional, and batch will not fail if they are missing. |
AI Foundation Modules
The following table lists modules within the AI Foundation applications which may be used by one or more other RAP applications, in addition to the common modules from the prior section. This primarily covers forecasting and integration needs for Planning application usage. It is important to note that the underlying process for generating forecasts leverages jobs from the Lifecycle Pricing Optimization (LPO) application, so you will see references to that product throughout the POM batch flows and in Retail Home modules.
Table 5-4 AI Foundation Shared Batch Modules
Application | Module | Usage Notes |
---|---|---|
AI Foundation |
FCST > Batch |
Collection of batch jobs associated with foundation loads, demand estimation, and forecast generation/export. The modules under this structure are needed to get forecasts for Planning. |
To initialize data for the forecasting program, use the ad hoc POM process RSE_MASTER_ADHOC_JOB
described
in Sending Data to AI Foundation. After the platform is initialized, you may use the Forecast Configuration user interface to set up and run your
initial forecasts. For complete details on the requirements and implementation process for forecasting, refer to the Retail
AI Foundation Cloud Services Implementation Guide.
For reference, the set of batch programs that should be enabled for forecasting are listed below (not including foundation data loads common to all of AI Foundation). Enable these by syncing POM with MDF modules, though it is best to validate that the expected programs are enabled after the sync.
Note:
Jobs with PMO in the name are also used for Lifecycle Pricing Optimization and are shared with the forecasting module.Job Name |
---|
PMO_ACTIVITY_LOAD_START_JOB |
PMO_ACTIVITY_STG_JOB |
PMO_ACTIVITY_LOAD_JOB |
PMO_ACTIVITY_LOAD_END_JOB |
PMO_CREATE_BATCH_RUN_JOB |
PMO_RUN_EXEC_SETUP_JOB |
PMO_RUN_EXEC_START_JOB |
PMO_RUN_EXEC_PROCESS_JOB |
PMO_RUN_EXEC_END_JOB |
RSE_CREATE_FCST_BATCH_RUN_JOB |
RSE_FCST_BATCH_PROCESS_JOB |
RSE_FCST_BATCH_RUN_END_JOB |
RSE_CREATE_FCST_BATCH_RUN_ADHOC_JOB |
RSE_FCST_BATCH_PROCESS_ADHOC_JOB |
There are also two processes involved in forecast exports to MFP, one as part of weekly batch and the other as an ad hoc job which you can run during implementation.
Job Name |
---|
RSE_MFP_FCST_EXPORT_JOB |
RSE_MFP_FCST_EXPORT_ADHOC_JOB |
Batch Setup Example
This section will guide you through an example of setting modules in Retail Home and syncing with POM. A basic implementation of MFP with forecasting in AIF might send all the following foundation data files (and their CTX files):
-
CALENDAR.csv
-
PRODUCT.csv
-
ORGANIZATION.csv
-
EXCH_RATE.csv
-
SALES.csv
-
INVENTORY.csv
-
RECEIPT.csv
-
TRANSFER.csv
-
MARKDOWN.csv
-
ADJUSTMENT.csv
-
ORDER_HEAD.csv
-
ORDER_DETAIL.csv
-
RTV.csv
-
RA_SRC_CURR_PARAM_G.dat
These files will be bundled into a single ZIP file named RAP_DATA.zip
. To configure the Customer Modules
for this batch implementation, perform the following steps:
-
At the top level, you may disable the RI module entirely (if it’s visible), because you are not implementing that solution
-
Enable and expand the RAP module. Within the
RAP_COMMON
sub-module, enable only the following components and disable the rest:-
RAP>RAP_COMMON>ZIP_FILES: RAP_DATA_ZIP
-
RAP>RAP_COMMON>SICONTROLFILES: RAP_SI_DIM_CALENDAR, RAP_SI_DIM_EXCHANGE_RATES, RAP_SI_DIM_ONORDER, RAP_SI_DIM_ORGANIZATION, RAP_SI_DIM_PRODUCT, RAP_SI_FACT_ADJUSTMENT, RAP_SI_FACT_INVENTORY, RAP_SI_FACT_MARKDOWN, RAP_SI_FACT_ORDER_DETAIL, RAP_SI_FACT_RECEIPT, RAP_SI_FACT_RTV, RAP_SI_FACT_SALES, RAP_SI_FACT_TRANSFER
-
RAP>RAP_COMMON>SIBATCH: RAP_SI_INVADJ, RAP_SI_INVPOS, RAP_SI_INVRECEIPT, RAP_SI_INVRTV, RAP_SI_INVTRANSFER, RAP_SI_MARKDOWN, RAP_SI_ONORDER, RAP_SI_PO, RAP_SI_REQUIRED, RAP_SI_SALES
-
RAP>RAP_COMMON>RDXBATCH: RAP_RDX_INVADJ, RAP_RDX_INVPOS, RAP_RDX_INVECEIPT, RAP_RDX_INVRTV, RAP_RDX_INVTRANSFER, RAP_RDX_MARKDOWN, RAP_RDX_PO, RAP_RDX_REQUIRED, RAP_RDX_SALES
-
RAP>RAP_COMMON>BATCH: RAP_INVADJ, RAP_INVPOS, RAP_INVRECEIPT, RAP_INVRTV, RAP_INVTRANSFER, RAP_MARKDOWN, RAP_PO, RAP_REQUIRED, RAP_SALES
-
-
To use AIF for forecasting, you will also need certain parts of the AI Foundation modules (named RSP in Retail Home). Enable the RSP root module, expand it, and enable the FCST module. All options under FCST should also be enabled. Disable all other RSP modules here.
-
If you are integrating with Merchandising using the RDE batch jobs, then you are likely not providing flat files for most things, and the setup process will be different. Refer to Integration with Merchandising to understand the batch setup process in more detail.
Once all changes are made, make sure to Save the updates using the button below the table. Only after saving the changes will they be available to sync with POM.
The next set of steps will be performed from POM:
-
Go to the Batch Administration screen for each schedule (Nightly tabs of AIF DATA and AIF APPS schedules) and disable ALL jobs in the Nightly schedule. By disabling all jobs, the next step will selectively enable just what is needed.
-
Click Sync with MDF in each schedule. Wait for the sync to complete (it will take some time and the screen will prevent any updates while syncing).
-
Review the list of jobs that are now enabled to verify that it is correct (for example, that the jobs relating to PRODUCT file loads are all enabled similarly to when you ran historical data loads).
-
Go to the Batch Monitoring screen. If you have a schedule already open for a past business date, click Close Schedule. If you are already on the current business date then just click Restart Schedule and skip the next step.
-
Change the POM business date to the date you will be loading nightly batch data for (for example if your data files will have data for 2/25/2023 then that should also be the business date). Open a new schedule using the provided button.
-
Navigate to each nightly batch schedule and click the Inter-Schedule Dependencies link. Enable all the dependencies between applications you are using (RI, RSP, and RPASCE all have dependencies that must be turned on).
-
Enable and schedule the batches to run from Schedule Administration. The inter-schedule dependencies will ensure downstream jobs are not run until the necessary dependencies are complete (for example, RSP jobs will wait for RI loads to complete).
-
Make sure the business date in the data warehouse is one day prior to the first day of data you plan to load through the nightly batch. If necessary, use the standalone POM process
LOAD_CURRENT_BUSINESS_DATE_ADHOC
to advance the business date. For example, if your first nightly batch is loading data for2023-04-30
then you must ensure the data warehouse is currently on business date2023-04-29
. The nightly batch will automatically advance the date starting with2023-04-30
. -
If you chose not to provide the
RA_SRC_CURR_PARAM_G.dat
file for ensuring proper business date validation, then go back to Batch Administration in the AIF DATA nightly schedule and disableBATCH_VALIDATION_JOB
,RA_SRC_CURR_PARAM_G_COPY_JOB
, andRA_SRC_CURR_PARAM_G_STG_JOB
.
The next set of steps may or may not be needed depending on your business calendar configuration:
-
From the Batch Administration Nightly schedules, locate the column for Days of the Week. By default, weekly jobs may run on either Saturday or Sunday (varies by process). You must align the weekly jobs to your week-ending dates for anything involved in integration or calculations, such as RDX and PDS batch jobs.
-
For example, if you wish to set your week-ending date as Saturday for all jobs involved in RI > Planning integrations, filter the AIF DATA Schedule job names using each of the following codes: PDS, RDX
-
For each set of PDS or RDX jobs, look at the Days of the Week value. Anything that is not set to run Saturday should be modified by editing the job record and changing the day to Saturday. You might also want some jobs to run daily, in which case you select all days of the week here.
-
Repeat this process in the RPASCE schedule, moving any jobs to the desired Day of the Week value.
-
When all changes are done, be sure to Restart Schedule from the Batch Monitoring screen.
Lastly, ensure that the following jobs are disabled in the AIF DATA schedule after all other setup is done, as they should not be used in the current version:
-
OBIEE_CACHE_CLEAR_JOB
-
ODI_LOG_EXTRACTOR_JOB
-
ODI_LOG_LOADER_JOB
You are now ready to begin running your nightly batches. As soon as you run the first batch cycle, you must continue to run AIF DATA and AIF APPS batch cycles every night in sequence. You must not skip any days, because some jobs have internal processing based on the day of week that they execute and skipping days will prevent proper operation of these jobs. If you are not providing daily fact data, then you must still run the daily batches with a full set of dimension files, such as products and locations; you can never run a batch without dimension files present and populated with data. The recommended approach is for the source system providing data files to always package and upload a nightly ZIP file every day even when no data is changing. The uploaded ZIP should contain the dimension data plus empty fact files where daily data is not being generated.
Adjustments in POM
While the bulk of your batch setup should be done in Retail Home, it may be necessary to fine-tune your schedule in POM after the initial configuration is complete. You may need to disable specific jobs in the nightly schedules (usually at Oracle’s recommendation) or reconfigure the ad hoc processes to use different programs. The general steps to perform this activity are:
-
From Retail Home, click the link to navigate to POM or go to the POM URL directly if known. Log in as a batch administrator user.
-
Navigate to the Batch Administration screen.
-
Select the desired application tile, and then select the schedule type from the nightly, recurring, or standalone options.
-
Search for specific job names, and then use the Enabled option to turn the program on or off.
-
From the Batch Monitoring screen, click Restart Schedule to apply the changes.
Note:
If you sync with MDF again in the future, it may re-enable jobs that you turned off inside a module that is turned on in Retail Home. For that reason, the module configuration is typically used only during implementation, then POM is used once you are live in production.Configure POM Integrations
Retailers often have external applications that manage batch execution and automation of periodic processes. They might also require other downstream applications to be triggered automatically based on the status of programs in the Retail Analytics and Planning. POM supports the following integrations that should be considered as part of your implementation plan.
Table 5-5 POM Integrations
Activity | References |
---|---|
Trigger RAP batches from an external program |
POM Implementation Guide > Integration > Invoking Cycles in POM |
Trigger external processes based on RAP batch statuses |
POM Implementation Guide > Integration > External Status Update |
Add external dependencies into the RAP batch to pause execution at specific points |
POM Implementation Guide > Integration > External Dependency |
Schedule the Batches
Once you are ready to begin regularly scheduled batch processing, you must log in to POM to enable each batch cycle and provide a start time for the earliest one in the batch sequence. The general steps are:
-
Log into POM as an administrator user.
-
Access the Scheduler Administration screen.
-
Select each available tile representing a batch schedule and select the Nightly batch option. AIF DATA and AIF APPS schedules must run every night once they are enabled; you cannot run them weekly or skip any days.
-
Edit the rows in the table to enable each batch and enter a start time. You can start all of the batches with similar times or stagger them apart based on how long you think they need to wait before their dependencies will be fulfilled by the linked applications. After any change to the schedule times, you must also Restart the Schedule in Batch Monitoring for it to take effect.
-
For each batch after AIF DATA that must execute, you must verify the inter-schedule dependencies are enabled for those batches. You can find the dependencies and enable them by going to the Batch Monitoring screen and selecting each batch’s Nightly set of jobs. Click the number in front of Inter-Schedule Dependencies and click Enable to the right of each displayed row if the current status is Disabled. This should change the status of the row to Pending.
Refer to the POM User Guide for additional information about the Scheduler screens and functionality. Scheduling the batch to run automatically is not required if you are using an external process or scheduler to trigger the batch instead.
For the AIF DATA nightly batch (which consumes the foundation input files for platform data), the batch will first look for all the required input ZIP packages configured through Customer Modules Management. The batch will wait for 4 hours for all required files, after which it will fail with an error in POM. When choosing a start time for the batch, it is best to start the 4-hour window an hour before you expect to have all the files uploaded for it to use. The batch performs reclassification of aggregate tables as its first step, which can take the full hour to complete in the case of very large reclassifications. This still provides 3+ hours during which you can schedule all file uploads and non-Oracle integration processes to occur.
The other application batches, such as those for the AI Foundation modules, begin automatically once the required Retail Insights batch jobs are complete (assuming you have enabled those batches). Batch dependencies have been pre-configured to ensure each downstream process begins as soon as all required data is present.
Batch Flow Details
POM allows you to export the full batch schedule configuration to a spreadsheet for review. This is an easier way to see all of the jobs, processes, and dependencies that exist across all batch schedules that you are working with on the platform. Perform the following steps to access batch configuration details:
-
From Retail Home, click the link to navigate to POM, or go to the POM URL directly if known. Log in as a batch administrator user.
-
Navigate to the Batch Administration screen.
-
Select the desired application tile, and then select the schedule type from the nightly, recurring, or standalone options.
-
Click the Export Config button to download the schedule data.
-
Choose your preferred format (XLS or JSON).
-
XLS is meant for reviewing it visually.
-
JSON is used for importing the schedule to another environment.
-
-
Save the file to disk and open it to review the contents.
For more details about the tabs in the resulting XLS file, refer to the POM User Guide, “Export/Import Schedule Configuration”.
Planning Applications Job Details
Planning applications such as Merchandise Financial Planning schedule jobs through POM, and run ad hoc tasks directly using Online Administration Tasks (OAT) within the application. POM is the preferred approach for scheduling jobs in RAP for managing interdependencies with jobs from RI and AI Foundation. Refer to the application Administration Guides for more details on scheduling tasks through OAT.
Planning applications also use a special Batch Framework that controls jobs using batch control files. You may configure these batch control files for various supported changes within the application. The POM schedule for Planning internally calls an OAT task controlled by the batch control entries. A standard set of nightly and weekly jobs for Planning are defined to schedule them in POM. You also have the option to disable or enable the jobs either directly through POM, or by controlling the entries in batch control files. Refer to the Planning application-specific Implementation Guides for details about the list of jobs and how jobs can be controlled by making changes to batch control files.
Reprocessing Nightly Batch Files
When a nightly batch fails due to a corrupt or improperly formed data file, you may want to re-send just the fixed data
file and not the entire nightly ZIP upload. An ad hoc POM process is available for uploading individual data files and appending
them to the current nightly batch run. The process is called REPROCESS_ZIP_FILE_PROCESS_ADHOC
. It expects
a file RI_REPROCESS_DATA.zip
to be uploaded using File Transfer Services (FTS). The ad hoc program will import
and unpack the ZIP file and directly move any contents into the staging area with all existing data files sent from a previous
upload. No existing file will be deleted; it will overwrite only the files provided in RI_REPROCESS_DATA.zip
.
Once a new file has been placed, you will still need to re-run the jobs to import that file. Or, if a COPY
job for that file is what failed, re-run that job and the batch will resume from there.