6 Generic ReST Jobs
This chapter describes how a user can set up ReST Jobs that can be invoked and tracked by POM.
Introduction
As a Job Scheduler, POM provides the ability for applications to invoke and track different types of Jobs. The list of Job Types that are supported in POM by default are defined in the Batch Job section of the Batch Concepts chapter of this guide.
While this list of types meets the needs of Retail Applications that already have their Batch Schedules on POM, it is fixed and lacks extensibility.
To remedy this, POM provides generic interfaces to define and run ReST-based Jobs on external systems that meet certain requirements.
Features of Generic ReST Jobs
To have POM integrate with an external system and to invoke and track Jobs run on that external system, the external system must provide the following capabilities.
# | Ability / ReST endpoints | Description | Mandatory / Optional |
---|---|---|---|
1 |
Start requested Job |
Ability to start a Job that was requested and return a unique ID to track its execution. If the Job can’t be started or does not exist, throw an exception |
Mandatory |
2 |
Check status of Job |
Ability to accept a unique execution ID and return a status. In case the execution ID is unknown, throw an exception |
Mandatory |
3 |
Restart Job |
Ability to restart a Job, given an execution ID of a failed job. Once restarted, a unique execution ID to track the new execution must be returned. |
Optional. If the Restart API is not defined, POM uses the Start API (#1) |
4 |
Kill Job |
Ability to stop/abort/kill a running Job. This is not meant to be a graceful stop. It should be a hard-stop and not just a request to the Job to stop when possible. |
Optional. If a KILL is requested when it’s not defined, then POM shows an exception popup. |
5 |
Job Log |
Ability to provide runtime logs for the given execution ID. |
Optional. |
6 |
Check system status |
Ability for POM to check whether the system is up and available. This will show up on the POM Health-check screen. |
Mandatory. If this is not defined, this will show up in the Health check popup with an error and affects the automated patching process. |
Execution Sequence
At runtime, POM invokes generic ReST jobs as depicted below.
-
Initially the status of the Job in POM is
LOADED
. -
The Execution Engine sends the request to the Job Agent to run the Job. The Agent Poller picks the Job for execution.
-
The Agent sets the status of the Job to
RUNNING
. -
The Start endpoint is invoked on the External System, with the details of the Job to run. The endpoint starts the Job and returns a unique ID to track the execution (
100
returned, as shown above). -
The Agent then goes to sleep for 5 seconds (configurable interval).
-
It then invokes the Status endpoint on the External System with the execution ID returned previously (in this case
100
) -
If a status of
ERROR
orCOMPLETED
is returned, then the Job is assumed to have finished and the status is set on the POM job accordingly. -
In case the status is still
RUNNING
, the Agent then goes back to sleep for 5 seconds and thereafter polls the status again. It does so, until a terminal status ofERROR
orCOMPLETED
is received.
Handling Restarts
A restart is handled in almost the same sequence as shown above. Except, instead of invoking the Start endpoint, the Restart endpoint will be used with the previously failed execution ID.
In case the Restart endpoint has not been defined, then the Start endpoint will be used.
Handling Kills
In case of killing/aborting a RUNNING
Job, the Kill endpoint is invoked. A successful response from this
endpoint will let POM perform its cleanup and also mark the Job with a status of ERROR
.
In case the Kill endpoint has not been defined for a Custom Job Type, then invoking the Kill from the POM UI for a running Job of that type will simply show a popup stating the Kill is not defined for the selected Job type.
Applications Notifications
Applications that run their batches through POM are capable of having POM generate a contextual notification upon the completion of a job. The completion can be a success or failure. This is accomplished by the application optionally specifying notification details in the return payload of the Job Status endpoint. For example, an application can upon processing over a million records have the job return a notification object containing the message: ‘Job abcd processed over a million records today’.
See the Job Status API specification in the Generic ReST Jobs section later in this chapter for details.
Defining Custom Job Types
Generic ReST Jobs are defined in POM as Custom Job Types. A class of Generic ReST Jobs is represented in POM as a single Custom Job Type.
All Jobs in POM today have a Job Type attribute. For Generic ReST Jobs, this Job Type will be a Custom Job Type that will encapsulate all the necessary information needed about the destination system.
Batch Schedule Spreadsheet
Custom Job Types can be defined through the Job Types tab of the Batch Schedule spreadsheet. Custom Job Types that are specified on the spreadsheet get seeded into POM when the Batch Schedule is loaded. See Batch Schedule Spreadsheet Template for more information.
POM UI
The Administrator can also maintain Custom Job Types in the Job Types section of the System Configuration screen. Refer to the “Edit Job Type” subsection of the “System Configuration” section of the POM User Guide.
New Custom Job Types can be added there as well.
Environment-specific Information
The only aspects of a Custom Job Type that will change based on its deployment are its URL and its OAuth Scope.
Today teams use an endpoint to configure a new Schedule in POM. This endpoint has been enhanced to accept the URL and the OAuth scope needed for a Custom Job Type. Refer to the “Configure New Schedule” subsection of the “System Configuration” section in the POM User Guide for further information.
Custom Job Type Endpoint Specifications
Credentials & Scopes
POM only supports the OAuth System Credential Grant mode of authentication. No other types of authentication are supported.
The OAuth scope stored for each Custom Job Type informs POM about which scopes are to be used when invoking the Custom Job Type APIs.
POM makes calls to these Job Type endpoints by using this scope and the default IDCS credentials it is configured with. However, these endpoints are generally hosted on an application that has its own IDCS application. In such cases, POM cannot invoke those endpoints using its default IDCS credentials. Therefore, POM provides an option to configure a custom IDCS slice and custom credentials to be used for this type of authentication. These credentials are the Client Id & Secret of the respective application/ OAuth client, as shown below. POM calls the custom job type endpoints that can be authenticated using an IDCS instance different that POM’s. Custom credentials are also published to agents during new scheduler day creation.
It is expected that the IDCS Client App that these custom credentials point to will contain the necessary scopes to call these endpoints.
Job Start API
HTTP Method |
POST |
||||||||||||||
Request Payload (JSON) |
The attributes of the JSON payload are as follows :
|
||||||||||||||
Authentication |
OAuth System Credentials Grant |
||||||||||||||
Response Payload (JSON) |
In case of a successful invocation, a success HTTP code (
In case of an issue, an HTTP code other than the success code (that is, not |
Job Status API
HTTP Method |
GET |
||||||||||||||||
Query Parameters (JSON) |
|
||||||||||||||||
Authentication |
OAuth System Credentials Grant |
||||||||||||||||
Response Payload (JSON) |
In case of a successful invocation, a success HTTP code (
In case of an issue, an HTTP code other than the success code (that is, not |
Job Restart API
HTTP Method |
POST |
||||||||||||||||
Request Payload (JSON) |
The attributes of the request payload are as follows:
|
||||||||||||||||
Authentication |
OAuth System Credentials Grant |
||||||||||||||||
Response Payload (JSON) |
In case of a successful invocation, a success HTTP code (
In case of an issue, an HTTP code other than the success code (that is, not |
Job Kill API
HTTP Method |
POST |
||||||||
Request Payload (JSON) |
The attributes of the request payload are as follows:
|
||||||||
Authentication |
OAuth System Credentials Grant |
||||||||
Response Payload (JSON) |
In case of a successful invocation, a success HTTP code (
In case of an issue, an HTTP code other than the success code (that is, not |
Job Log API
HTTP Method |
GET |
||||||||
Query Parameters (JSON) |
|
||||||||
Authentication |
OAuth System Credentials Grant |
||||||||
Response Payload (JSON) |
In case of a successful invocation, the response will contain the log data for the execution provided. This log will be viewable in the log downloaded from the POM UI. The logs from the response are also sent in the job log. Any other response while calling this endpoint would be captured as an error while downloading logs. |