Generic ReST Jobs

6 Generic ReST Jobs

This chapter describes how a user can set up ReST Jobs that can be invoked and tracked by POM.

Introduction

As a Job Scheduler, POM provides the ability for applications to invoke and track different types of Jobs. The table below lists the types of Jobs that are supported in POM by default.

Job Type	Description
EXEC	Refers to shell-script based Jobs. POM is expected to start a shell script and track its execution. Note: This is the default type of Job. If a Job type has not been specified on the batch schedule spreadsheet, it is assumed to be of this type.
BDI	Indicative of BDI (bulk data) Jobs. POM will invoke a BDI Process and track its execution for these types of Jobs.
RI	Refers to Data Intelligence Services (DIS) Jobs for the RI Schedule. POM invokes custom DIS wrapper endpoints and tracks their execution.
RASE	Refers to Data Intelligence Services (DIS) Jobs for the RSP Schedule. The wrapper endpoints invoked here are the same as used by the RI job type.
RPAS	Indicative of RPAS-WebService Jobs for the RPAS schedule. POM invokes RPAS endpoints to start and track RPAS batches.

While this list of types meets the needs of Retail Applications that already have their Batch Schedules on POM, it is fixed and lacks extensibility.

To remedy this, POM provides generic interfaces to define and run ReST-based Jobs on external systems that meet certain requirements.

Features of Generic ReST Jobs

To have POM integrate with an external system and to invoke and track Jobs run on that external system, the external system must provide the following capabilities.

#	Ability / ReST endpoints	Description	Mandatory / Optional
1	Start requested Job	Ability to start a Job that was requested and return a unique ID to track its execution. If the Job can’t be started or does not exist, throw an exception	Mandatory
2	Check status of Job	Ability to accept a unique execution ID and return a status. In case the execution ID is unknown, throw an exception	Mandatory
3	Restart Job	Ability to restart a Job, given an execution ID of a failed job. Once restarted, a unique execution ID to track the new execution must be returned.	Optional. In cases where the Restart API is not defined, POM uses the Start API (#1)
4	Kill Job	Ability to stop/abort/kill a running Job. This is not meant to be a graceful stop. It should be a hard-stop and not just a request to the Job to stop when possible.	Optional. If a KILL is requested when it’s not defined, then POM shows an exception popup.
5	Job Log	Ability to provide runtime logs for the given execution ID.	Optional.
6	Check system status	Ability for POM to check whether the system is up and available. This will show up on the POM Health-check screen.	Optional. In case this is not defined, this will show up in the Health check popup with an error.

Execution Sequence

At runtime, POM invokes generic ReST jobs as depicted below.

Initially the status of the Job in POM is LOADED.
The Execution Engine sends the request to the Job Agent to run the Job. The Agent Poller picks the Job for execution.
The Agent sets the status of the Job to RUNNING.
The Start endpoint is invoked on the External System, with the details of the Job to run. The endpoint starts the Job and returns a unique ID to track the execution (100 returned, as shown above).
The Agent then goes to sleep for 5 seconds (configurable interval).
It then invokes the Status endpoint on the External System with the execution ID returned previously (in this case 100)
If a status of ERROR or COMPLETED is returned, then the Job is assumed to have finished and the status is set on the POM job accordingly.
In case the status is still RUNNING, the Agent then goes back to sleep for 5 seconds and thereafter polls the status again. It does so, until a terminal status of ERROR or COMPLETED is received.

Handling Restarts

A restart is handled in almost the same sequence as shown above. Except, instead of invoking the Start endpoint, the Restart endpoint will be used with the previously failed execution ID.

In case the Restart endpoint has not been defined, then the Start endpoint will be used.

Handling Kills

In case of killing/aborting a RUNNING Job, the Kill endpoint is invoked. A successful response from this endpoint will let POM perform its cleanup and also mark the Job with a status of ERROR.

In case the Kill endpoint has not been defined for a Custom Job Type, then invoking the Kill from the POM UI for a running Job of that type will simply show a popup stating the Kill is not defined for the selected Job type.

Defining Custom Job Types

Generic ReST Jobs are defined in POM as Custom Job Types. A class of Generic ReST Jobs is represented in POM as a single Custom Job Type.

All Jobs in POM today have a Job Type attribute. For Generic ReST Jobs, this Job Type will be a Custom Job Type that will encapsulate all the necessary information needed about the destination system.

Batch Schedule Spreadsheet

Custom Job Types can be defined through the Job Types tab of the Batch Schedule spreadsheet. Custom Job Types that are specified on the spreadsheet get seeded into POM when the Batch Schedule is loaded. See Batch Schedule Spreadsheet Explained for more information.

POM UI

The Administrator can also maintain Custom Job Types in the Job Types section of the System Configuration screen. Refer to the “Edit Job Type” subsection of the “System Configuration” section of the POM User Guide.

New Custom Job Types can be added there as well.

Environment-specific Information

The only aspects of a Custom Job Type that will change based on its deployment are its URL and its OAuth Scope.

Today teams use an endpoint to configure a new Schedule in POM. This endpoint has been enhanced to accept the URL and the OAuth scope needed for a Custom Job Type. Refer to the “Configure New Schedule” subsection of the “System Configuration” section in the POM User Guide for further information.

Internal Representation

A JSON representation of a Custom Job Type stored within POM looks like the following:

In this case the Custom Job Type is named MOCKTYPE. All Jobs of this type would invoke the endpoints defined above.

Job Parameter Restrictions

The definition of parameters to Generic ReST Jobs in POM is restricted to a double-pipe-delimited string of key value pairs.

For example: key=value||key1=value1||key2=value2

Custom Job Type Endpoint Specifications

Authentication

POM only supports the OAuth System Credential Grant mode of authentication. No other types of authentication, such as Basic Auth, are supported.

The OAuth scope stored for each Custom Job Type informs POM about which scopes are to be used when invoking the Custom Job Type APIs.

Job Start API

HTTP Method

POST

Request Payload (JSON)

Attribute	Description
cycleName	POM Batch Cycle
flowName	POM Batch Flow
processName	POM Batch process name
jobName	POM Batch job name
parameters	Job Parameters (Double-pipe delimited key-value pairs) as defined in POM
agentExecutionId	A unique ID assigned by POM for every job run

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of successful invocations, the response will contain the following:

Attribute	Description
executionId	Unique ID returned by the target app to POM for status tracking
status	Any status
executionInfo (optional)	Any additional info the target app would like to share with POM

In case of any issues while submitting the Job, a 500 exception response must be thrown.

Job Status API

HTTP Method

GET

Query Parameters (JSON)

Attribute	Description
processName	POM Batch process name
jobName	POM Batch job name
executionId	Unique Id returned by the target app to POM for status tracking

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of a successful invocation, the response will contain the following:

Attribute	Description
executionId	Unique Id returned by the target app to POM for status tracking
status	RUNNING / ERROR / COMPLETED.
executionInfo (optional)	Any additional info the target app would like to share with POM.

In case of any issues while submitting the Job, a 500 exception response must be thrown.

Job Restart API

HTTP Method

POST

Request Payload (JSON)

Attribute	Description
cycleName	POM Batch Cycle
flowName	POM Batch Flow.
processName	POM Batch process name
jobName	POM Batch job name
parameters	Job Parameters (Double-pipe delimited key-value pairs) as defined in POM
agentExecutionId	A unique ID assigned by POM for every job run
executionId	Unique Id of the previously failed execution returned by the target app to POM for status tracking

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of a successful invocation, the response will contain the following:

Attribute	Description
executionId	Unique Id returned by the target app to POM for status tracking
status	Any status
executionInfo (optional)	Any additional info the target app would like to share with POM.

In case of any issues while restarting the Job, a 500 exception response must be thrown.

Job Kill API

HTTP Method

POST

Request Payload (JSON)

Attribute	Description
processName	POM Batch process name
jobName	POM Batch job name
executionId	Unique ID of the previously failed execution returned by the target app to POM for status tracking

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of successful invocations, the response would contain the following

Attribute	Description
executionId (optional)	Unique ID returned by the target app to POM for status tracking
status (optional)	Any status
executionInfo (optional)	Any additional info the target app would like to share with POM.

In case of Job cannot be killed, a 500 exception response must be thrown.

Job Log API

HTTP Method

GET

Query Parameters (JSON)

Attribute	Description
processName	POM Batch process name
jobName	POM Batch job name
executionId	Unique ID returned by the target app to POM for status tracking

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of a successful invocation, the response will contain the log data for the execution provided. This log will be viewable in the log downloaded from the POM UI.

In case of any issues while fetching the Job log, a 500 exception response must be thrown.

Job System Check API

HTTP Method

GET

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

A success response of 200 or 204 indicates that the system is up and running.

Any exception while calling this endpoint, should be interpreted as the system being down.