Batch Execution Mechanism

The recommendation for Batch Execution with an External Scheduler is as follows:

During the definition of a batch using the Batch Definition window of Operations module, the Batch is called as EXTBATCH and the Information Domain in which this Batch is defined is called as INFODOM. Hence INFODOM_EXTBATCH becomes the Batch ID.

Consider a scenario, to run the following tasks in this Batch.

The first task 'Task1' loads data in a warehouse table FCT_CUSTOMER.
The second task 'Task2' loads data in a warehouse table DIM_GEOGRAPHY.
The third task 'Task3' is a Data Transformation, uses both the Tables mentioned above. Hence this can run only if both the above tasks, Task1 and Task2 are complete.
If either Task1 or Task2 fails, a new task namely Task 4 can be executed with the Data Transformation which uses the data of the previous load.
The final task is a task namely Task5 which is a Cube building task. This takes several hours as it builds a Cube with many dimensions and hierarchies and holds large number of combinations.

The parameters for the Tasks are chosen from the drop-down choices provided. OFSAAI provides the choices through its Data Model Management.

Since, the Task 3 or Task 5 is executed based on conditional success / failure of previous tasks, the conditionality needs to be simulated in the ES. If the External Scheduler wants to control the order/conditionality for tasks then it needs to be defined in such a way that they have the same precedence. Here it would be ideal to define it as follows. The arrows in the following figure, shows the precedence involved.

Figure 17-4 Illustration of Batch Execution

This illustration shows the Illustration of Batch Execution. It shows how the Task are split and executed.

The export of such a Batch from OFSAAI would look like the following. For more information, see OFSAAI Standard XML.

<BATCH BATCHID="INFODOM_EXTBATCH" NOOFTASKS="5" SYSTEMLOCALE="+5:30 GMT"
          INFODOMAIN="INFODOM" REVUSER="OPERADMIN" DEFTYPE="DEF"><RUNINFO REVUID="" EXTUID="" BATCHSTATUS="" INFODATE=""
          LAG=""/><TASK TASKID="Task1" COMPONENTID="LOAD DATA" TASKSTATUS="N"
          FILTER="N"><PRECEDENCE><ONSUCCESSOF><TASKID/></ONSUCCESSOF><ONFAILUREOF><TASKID/></ONFAILUREOF></PRECEDENCE></TASK><TASK TASKID="Task2" COMPONENTID="CUBE CREATE" TASKSTATUS="N"
          FILTER="N"><PRECEDENCE><ONSUCCESSOF><TASKID/></ONSUCCESSOF><ONFAILUREOF><TASKID/></ONFAILUREOF></PRECEDENCE></TASK><TASK TASKID="Task3" COMPONENTID="RUN EXECUTABLE" TASKSTATUS="N"
          FILTER="N"><PRECEDENCE><ONSUCCESSOF><TASKID/></ONSUCCESSOF><ONFAILUREOF><TASKID/></ONFAILUREOF></PRECEDENCE></TASK><TASK TASKID="Task4" COMPONENTID="EXTRACT DATA" TASKSTATUS="N"
          FILTER="N"><PRECEDENCE><ONSUCCESSOF><TASKID/></ONSUCCESSOF><ONFAILUREOF><TASKID/></ONFAILUREOF></PRECEDENCE></TASK><TASK TASKID="Task5" COMPONENTID=" TRANSFORM DATA" TASKSTATUS="N"
          FILTER="N"><PRECEDENCE><ONSUCCESSOF><TASKID/></ONSUCCESSOF><ONFAILUREOF><TASKID/></ONFAILUREOF></PRECEDENCE></TASK>
    </BATCH>

Valid Values for Task Status are as follows:

Table 17-1 Details of the Task Status and their Value

Task Status	Value
N	Not Started
O	On Going
F	Failure
S	Success

Valid Values for Batch Status are as follows:

Table 17-2 Details of the Batch Status and their Value

Batch Status	Value
N	Not Started
O	On Going
R	For Restart
C	Complete

Valid values for FILTER are as follows:

Table 17-3 Details of the Filter Status and their Value

Filter Status	Value
H	Hold
K	Exclude/Skip
N	No Filter

When the definition of a Batch is exported and imported in ES, the Task Status, the Batch Status, and the Filter become irrelevant. This happens if you export a specific run of a Batch, which is not currently supported by OFSAAI. This should be included as a part of the XML for completeness.

After importing it in the ES, the Administrators can decide the order in which the tasks must be executed and alter the order of execution without violating the precedence set in OFSAAI. For example, the Administrator might configure it as in the following figure.

Figure 17-5 Illustration of Batch Execution

This illustration shows the Batch Execution representation, on how the batch initializes and the tasks are split, and then De-initialize.

The invocation of ESIC by the ES and the command line parameters passed for each task for the above configuration is as follows. For more information about command line parameters see ESIC Command Line Parameters and Job Types.

The ES needs to provide the 'Ext Unique ID'. In this case it is MAESTRO_INFODOM_EXTBATCH _20031001_1.

To Initialize the Batch Run:

esic -JI -Urevuser –Ppassword -RMAESTRO_INFODOM_EXTBATCH _20031001_1 -IINFODOM –BEXTBATCH -D20031001 -F/tmp/INFODOM

Task 1:

esic -JXT -Urevuser –Ppassword -RMAESTRO_ INFODOM_EXTBATCH_20031001_1 -IINFODOM -WC –TTask1

Task 2:

esic -JXT -Urevuser –Ppassword -RMAESTRO_ INFODOM_EXTBATCH_20031001_1 -IINFODOM -WC –TTask2

Task 3:

esic -JXT -Urevuser –Ppassword -RMAESTRO_ INFODOM_EXTBATCH_20031001_1 -IINFODOM -WC –TTask3

Task 4:

esic -JXT -Urevuser –Ppassword -RMAESTRO_ INFODOM_EXTBATCH_20031001_1 -IINFODOM -WC –TTask4

Task 5:

esic -JXT -Urevuser –Ppassword -RMAESTRO_ INFODOM_EXTBATCH_20031001_1 -IINFODOM -WC –TTask5

De-initialize:

esic -JD -Urevuser –Ppassword -RMAESTRO_ INFODOM_EXTBATCH_20031001_1 -IINFODOM -BINFODOM_EXTBATCH -D20031001

Ensure the following scenarios while executing an ES Batch:

Every Task executed in ES must have an equivalent task defined in a Batch within the Operations module, except for specific tasks such as Initialization, De-initialization, and Status Query / Alter Tasks.
If ES requests to alter the status of a task that has already been requested for execution, an error value is returned specific to such a case. The same hold good for Batch Run as well.
Task Execution must follow the precedence as defined in OFSAAI. Else, the task execution would result in failure.
Re executing a task of a Batch run, which was successfully executed will result in failure.
Execution of a Batch whose definition does not exist or deleted will result in failure. An error value is returned specific to such a case.
Execution of a task before the initialization of Batch will result in failure.
Simultaneous execution of the same Task of a Batch Run will result in failure. The same holds good for a Batch Run as well.