6 Generic ReST Jobs
This chapter describes how a user can set up ReST Jobs that can be invoked and tracked by POM.
Introduction
As a Job Scheduler, POM provides the ability for applications to invoke and track different types of Jobs. The table below lists the types of Jobs that are supported in POM by default.
Job Type | Description |
---|---|
EXEC |
Refers to shell-script based Jobs. POM is expected to start a shell script and track its execution. Note: This is the default type of Job. If a Job type has not been specified on the batch schedule spreadsheet, it is assumed to be of this type. |
BDI |
Indicative of BDI (bulk data) Jobs. POM will invoke a BDI Process and track its execution for these types of Jobs. |
RI |
Refers to Data Intelligence Services (DIS) Jobs for the RI Schedule. POM invokes custom DIS wrapper endpoints and tracks their execution. |
RASE |
Refers to Data Intelligence Services (DIS) Jobs for the RSP Schedule. The wrapper endpoints invoked here are the same as used by the RI job type. |
RPAS |
Indicative of RPAS-WebService Jobs for the RPAS schedule. POM invokes RPAS endpoints to start and track RPAS batches. |
While this list of types meets the needs of Retail Applications that already have their Batch Schedules on POM, it is fixed and lacks extensibility.
To remedy this, POM provides generic interfaces to define and run ReST-based Jobs on external systems that meet certain requirements.
Features of Generic ReST Jobs
To have POM integrate with an external system and to invoke and track Jobs run on that external system, the external system must provide the following capabilities.
# | Ability / ReST endpoints | Description | Mandatory / Optional |
---|---|---|---|
1 |
Start requested Job |
Ability to start a Job that was requested and return a unique ID to track its execution. If the Job can’t be started or does not exist, throw an exception |
Mandatory |
2 |
Check status of Job |
Ability to accept a unique execution ID and return a status. In case the execution ID is unknown, throw an exception |
Mandatory |
3 |
Restart Job |
Ability to restart a Job, given an execution ID of a failed job. Once restarted, a unique execution ID to track the new execution must be returned. |
Optional. In cases where the Restart API is not defined, POM uses the Start API (#1) |
4 |
Kill Job |
Ability to stop/abort/kill a running Job. This is not meant to be a graceful stop. It should be a hard-stop and not just a request to the Job to stop when possible. |
Optional. If a KILL is requested when it’s not defined, then POM shows an exception popup. |
5 |
Job Log |
Ability to provide runtime logs for the given execution ID. |
Optional. |
6 |
Check system status |
Ability for POM to check whether the system is up and available. This will show up on the POM Health-check screen. |
Optional. In case this is not defined, this will show up in the Health check popup with an error. |
Execution Sequence
At runtime, POM invokes generic ReST jobs as depicted below.
-
Initially the status of the Job in POM is
LOADED
. -
The Execution Engine sends the request to the Job Agent to run the Job. The Agent Poller picks the Job for execution.
-
The Agent sets the status of the Job to
RUNNING
. -
The Start endpoint is invoked on the External System, with the details of the Job to run. The endpoint starts the Job and returns a unique ID to track the execution (
100
returned, as shown above). -
The Agent then goes to sleep for 5 seconds (configurable interval).
-
It then invokes the Status endpoint on the External System with the execution ID returned previously (in this case
100
) -
If a status of
ERROR
orCOMPLETED
is returned, then the Job is assumed to have finished and the status is set on the POM job accordingly. -
In case the status is still
RUNNING
, the Agent then goes back to sleep for 5 seconds and thereafter polls the status again. It does so, until a terminal status ofERROR
orCOMPLETED
is received.
Handling Restarts
A restart is handled in almost the same sequence as shown above. Except, instead of invoking the Start endpoint, the Restart endpoint will be used with the previously failed execution ID.
In case the Restart endpoint has not been defined, then the Start endpoint will be used.
Handling Kills
In case of killing/aborting a RUNNING
Job, the Kill endpoint is invoked. A successful response from this
endpoint will let POM perform its cleanup and also mark the Job with a status of ERROR
.
In case the Kill endpoint has not been defined for a Custom Job Type, then invoking the Kill from the POM UI for a running Job of that type will simply show a popup stating the Kill is not defined for the selected Job type.
Defining Custom Job Types
Generic ReST Jobs are defined in POM as Custom Job Types. A class of Generic ReST Jobs is represented in POM as a single Custom Job Type.
All Jobs in POM today have a Job Type attribute. For Generic ReST Jobs, this Job Type will be a Custom Job Type that will encapsulate all the necessary information needed about the destination system.
Batch Schedule Spreadsheet
Custom Job Types can be defined through the Job Types tab of the Batch Schedule spreadsheet. Custom Job Types that are specified on the spreadsheet get seeded into POM when the Batch Schedule is loaded. See Batch Schedule Spreadsheet Explained for more information.
POM UI
The Administrator can also maintain Custom Job Types in the Job Types section of the System Configuration screen. Refer to the “Edit Job Type” subsection of the “System Configuration” section of the POM User Guide.
New Custom Job Types can be added there as well.
Environment-specific Information
The only aspects of a Custom Job Type that will change based on its deployment are its URL and its OAuth Scope.
Today teams use an endpoint to configure a new Schedule in POM. This endpoint has been enhanced to accept the URL and the OAuth scope needed for a Custom Job Type. Refer to the “Configure New Schedule” subsection of the “System Configuration” section in the POM User Guide for further information.
Custom Job Type Endpoint Specifications
Authentication
POM only supports the OAuth System Credential Grant mode of authentication. No other types of authentication, such as Basic Auth, are supported.
The OAuth scope stored for each Custom Job Type informs POM about which scopes are to be used when invoking the Custom Job Type APIs.
Job Start API
HTTP Method |
POST |
||||||||||||||
Request Payload (JSON) |
|
||||||||||||||
Authentication |
OAuth System Credentials Grant |
||||||||||||||
Response Payload (JSON) |
In case of successful invocations, the response will contain the following:
In case of any issues while submitting the Job, a 500 exception response must be thrown. |
Job Status API
HTTP Method |
GET |
||||||||
Query Parameters (JSON) |
|
||||||||
Authentication |
OAuth System Credentials Grant |
||||||||
Response Payload (JSON) |
In case of a successful invocation, the response will contain the following:
In case of any issues while submitting the Job, a 500 exception response must be thrown. |
Job Restart API
HTTP Method |
POST |
||||||||||||||||
Request Payload (JSON) |
|
||||||||||||||||
Authentication |
OAuth System Credentials Grant |
||||||||||||||||
Response Payload (JSON) |
In case of a successful invocation, the response will contain the following:
In case of any issues while restarting the Job, a 500 exception response must be thrown. |
Job Kill API
HTTP Method |
POST |
||||||||
Request Payload (JSON) |
|
||||||||
Authentication |
OAuth System Credentials Grant |
||||||||
Response Payload (JSON) |
In case of successful invocations, the response would contain the following
In case of Job cannot be killed, a 500 exception response must be thrown. |
Job Log API
HTTP Method |
GET |
||||||||
Query Parameters (JSON) |
|
||||||||
Authentication |
OAuth System Credentials Grant |
||||||||
Response Payload (JSON) |
In case of a successful invocation, the response will contain the log data for the execution provided. This log will be viewable in the log downloaded from the POM UI. In case of any issues while fetching the Job log, a 500 exception response must be thrown. |