Batch Concepts

2 Batch Concepts

This chapter describes concepts in Process Orchestration and Monitoring (POM) that are key to configuring and implementing the product successfully.

Batch Schedule

POM logically groups batch jobs in a batch schedule into a hierarchy of Processes, Flows and Cycles based on functionality and expected execution frequency.

Figure 2-1 Batch Schedule Diagram

Batch Job

A Job is representative of a program that is meant to run and is an atomic unit of work, that needs to be accomplished.

The following Job Types are supported by POM out-of-the-box.

Note:

Other custom job type can be used that are built using the Job Type feature of the System Configuration screen. See Generic ReST Jobs for more information.

Job Type	Description
EXEC	Refers to shell-script based Jobs. POM is expected to start a shell script and track its execution. This is the default type of Job. If a Job type has not been specified on the batch schedule spreadsheet, it is assumed to be of this type.
BDI (Bulk Data Integration)	Indicative of BDI Jobs. POM invokes a BDI Process and tracks its execution for this type of Jobs.
RI (Retail Insights)	Refers to DIS (Data Intelligence Services) Jobs for the RI Schedule. POM invokes RI endpoints and tracks their execution.
RASE (Retail Advanced Science Engine)	Refers to DIS (Data Intelligence Services) Jobs for the RSP Schedule. The wrapper endpoints invoked here are the same as the ones used by the RI job type.
RPAS (Retail Predictive Application Server)	Indicative of RPAS-WebService Jobs for the RPAS schedule. POM invokes RPAS endpoints to start and track RPAS batches.
OB (Order Broker)	Indicative of OB-WebService Jobs for the OB schedule. POM invokes OB endpoints to start and track OB batches.
OMS (Order Management System)	Indicative of OMS-WebService Jobs for the OMS schedule. POM invokes OMS endpoints to start and track OMS batches.
RDS (Retail Data Services)	Indicative of RDS Jobs. POM invokes RDS endpoints to start and track these Jobs.

Batch Process

A Batch Process is a collection of related Batch Jobs that will always run in sequential order. The dependencies between the Jobs within a Process ensure that these Jobs cannot be run in parallel.

Batch Processes on the Adhoc Cycle can be run from the POM UI or invoked using the ReST API. See Invoking the POM Service in the Integration chapter of this guide.

Batch Flow

A Batch Flow is a collection of related Batch Processes that must be run as a single unit. Within a Batch Flow one Process must be designated as the "first" Process, and another as the "last" Process. Other Processes can be run between them in sequential or parallel fashion, based on the dependencies setup.

Batch Flows can be defined on the Hourly Cycles and the Adhoc Cycle. The Flows on the Hourly Cycle can only be run once, while the Flows on the Adhoc Cycle can be run any number of times. These Flows can be invoked from the POM UI and can also be invoked through the ReST API. See Invoking the POM Service in the Integration chapter of this guide.

Batch Cycle

A Batch Cycle is the container in which Flows, Processes and Jobs are setup. There are three types of Batch Cycles.

Adhoc / Standalone Cycle

The Flows and Processes defined in this Cycle can run multiple times a day on an as-needed basis. Such invocations can run independent of other Processes and Flows on other Cycles.

Example: A Flow can be defined whose main responsibility is to purge data in database tables to boost performance. This Flow can then be run every 15 minutes without impacting Flows or Processes on other Cycles.

Recurring / Hourly Cycles

Recurring or Hourly Cycles are specialized Batch Cycles with the following properties

A Batch Schedule can have a maximum of 24 Hourly Cycles.
The Flows and Processes defined on an Hourly Cycle are replicated across all the defined Hourly Cycles. Dependencies are built between these batch entities, to ensure that unless it is completed in a prior cycle, it will not run in the next cycle.
All the Hourly Cycles are meant to be run prior to running the Nightly Flow. Once the Nightly Flow starts, all the Jobs on the Hourly Cycles are marked completed.

Example: The Sales processing Jobs in RMS support trickle processing by running every 30 minutes during the store trading hours. The schedule is pre-loaded with 24 Hourly Cycles. The Hourly Cycles are time triggered from the Scheduler or through ReST APIs externally. Based on the client's business operations, the individual flows of each cycle need to be scheduled.

Note:

If there are any errors on the Hourly Cycles, then the Nightly Cycle will not start until they are resolved.

Nightly Cycle

This Batch Cycle contains the set of Jobs that are executed at end of the business day. All of these Processes and Jobs belong to the default Nightly Flow of this Cycle. Once the Nightly Flow has started, all the loaded Flows and Processes on the Hourly Cycles are marked complete.

The Nightly Flow can be time-triggered or triggered using Schedule Links. It can be time-triggered through the POM Scheduler or externally through the ReST API.

Batch Dependencies

A dependency is a construct that prevents the execution of a Job, until a condition is satisfied.

Type	Description
Internal Dependency	Internal dependencies are the dependencies between Jobs of the same Batch Schedule.
Inter-schedule Dependency	Inter-Schedule dependencies are the dependencies between Jobs of different schedules running on the same POM instance.
External Dependency	External dependencies are dependencies between processes running on external systems, such as a Customer's system and Jobs running on the POM schedule.

Batch Schedule Links

Schedule Links or Execution Links, allow a Job from one Schedule, to trigger the Nightly Cycle of a different Schedule.

Batch Metadata

All information from the Batch Schedule spreadsheet is referred to as the Batch metadata. This information is modifiable from the POM UI on the Batch Administration Screen.

Batch Scheduler Day

Simply stated, this is a single copy of the Batch metadata or an instance of the Batch Schedule that is associated with a business date. All executions of batches for that business date are recorded on that Scheduler Day or Schedule Instance. Without a Scheduler Day, no Flows or Processes can be run.

A Batch Scheduler Day is marked Closed at the completion of all the Batch Jobs of the Nightly Cycle. The business date is then incremented by one, and a new Scheduler Day is created for the next day.

If however, there are Adhoc Flows or Processes executing when the Nightly Cycle completes, the closure of the Scheduler Day is then deferred until the completion of the running batches.