Generic ReST Jobs

6 Generic ReST Jobs

This chapter describes how a user can set up ReST Jobs that can be invoked and tracked by POM.

Introduction

As a Job Scheduler, POM provides the ability for applications to invoke and track different types of Jobs. The list of Job Types that are supported in POM by default are defined in the Batch Job section of the Batch Concepts chapter of this guide.

While this list of types meets the needs of Retail Applications that already have their Batch Schedules on POM, it is fixed and lacks extensibility.

To remedy this, POM provides generic interfaces to define and run ReST-based Jobs on external systems that meet certain requirements.

Features of Generic ReST Jobs

To have POM integrate with an external system and to invoke and track Jobs run on that external system, the external system must provide the following capabilities.

#	Ability / ReST endpoints	Description	Mandatory / Optional
1	Start requested Job	Ability to start a Job that was requested and return a unique ID to track its execution. If the Job can’t be started or does not exist, throw an exception	Mandatory
2	Check status of Job	Ability to accept a unique execution ID and return a status. In case the execution ID is unknown, throw an exception	Mandatory
3	Restart Job	Ability to restart a Job, given an execution ID of a failed job. Once restarted, a unique execution ID to track the new execution must be returned.	Optional. If the Restart API is not defined, POM uses the Start API (#1)
4	Kill Job	Ability to stop/abort/kill a running Job. This is not meant to be a graceful stop. It should be a hard-stop and not just a request to the Job to stop when possible.	Optional. If a KILL is requested when it’s not defined, then POM shows an exception popup.
5	Job Log	Ability to provide runtime logs for the given execution ID.	Optional.
6	Check system status	Ability for POM to check whether the system is up and available. This will show up on the POM Health-check screen.	Mandatory. If this is not defined, this will show up in the Health check popup with an error and affects the automated patching process.

Execution Sequence

At runtime, POM invokes generic ReST jobs as depicted below.

Initially the status of the Job in POM is LOADED.
The Execution Engine sends the request to the Job Agent to run the Job. The Agent Poller picks the Job for execution.
The Agent sets the status of the Job to RUNNING.
The Start endpoint is invoked on the External System, with the details of the Job to run. The endpoint starts the Job and returns a unique ID to track the execution (100 returned, as shown above).
The Agent then goes to sleep for 5 seconds (configurable interval).
It then invokes the Status endpoint on the External System with the execution ID returned previously (in this case 100)
If a status of ERROR or COMPLETED is returned, then the Job is assumed to have finished and the status is set on the POM job accordingly.
In case the status is still RUNNING, the Agent then goes back to sleep for 5 seconds and thereafter polls the status again. It does so, until a terminal status of ERROR or COMPLETED is received.

Handling Restarts

A restart is handled in almost the same sequence as shown above. Except, instead of invoking the Start endpoint, the Restart endpoint will be used with the previously failed execution ID.

In case the Restart endpoint has not been defined, then the Start endpoint will be used.

Handling Kills

In case of killing/aborting a RUNNING Job, the Kill endpoint is invoked. A successful response from this endpoint will let POM perform its cleanup and also mark the Job with a status of ERROR.

In case the Kill endpoint has not been defined for a Custom Job Type, then invoking the Kill from the POM UI for a running Job of that type will simply show a popup stating the Kill is not defined for the selected Job type.

Applications Notifications

Applications that run their batches through POM are capable of having POM generate a contextual notification upon the completion of a job. The completion can be a success or failure. This is accomplished by the application optionally specifying notification details in the return payload of the Job Status endpoint. For example, an application can upon processing over a million records have the job return a notification object containing the message: ‘Job abcd processed over a million records today’.

See the Job Status API specification in the Generic ReST Jobs section later in this chapter for details.

Defining Custom Job Types

Generic ReST Jobs are defined in POM as Custom Job Types. A class of Generic ReST Jobs is represented in POM as a single Custom Job Type.

All Jobs in POM today have a Job Type attribute. For Generic ReST Jobs, this Job Type will be a Custom Job Type that will encapsulate all the necessary information needed about the destination system.

Batch Schedule Spreadsheet

Custom Job Types can be defined through the Job Types tab of the Batch Schedule spreadsheet. Custom Job Types that are specified on the spreadsheet get seeded into POM when the Batch Schedule is loaded. See Batch Schedule Spreadsheet Template for more information.

POM UI

The Administrator can also maintain Custom Job Types in the Job Types section of the System Configuration screen. Refer to the “Edit Job Type” subsection of the “System Configuration” section of the POM User Guide.

New Custom Job Types can be added there as well.

Environment-specific Information

The only aspects of a Custom Job Type that will change based on its deployment are its URL and its OAuth Scope.

Today teams use an endpoint to configure a new Schedule in POM. This endpoint has been enhanced to accept the URL and the OAuth scope needed for a Custom Job Type. Refer to the “Configure New Schedule” subsection of the “System Configuration” section in the POM User Guide for further information.

Internal Representation

A JSON representation of a Custom Job Type stored within POM looks like the following:

Job Parameter Restrictions

The definition of parameters to Generic ReST Jobs in POM is restricted to a double-pipe-delimited string of key value pairs.

For example: key=value||key1=value1||key2=value2

Custom Job Type Endpoint Specifications

Credentials & Scopes

POM only supports the OAuth System Credential Grant mode of authentication. No other types of authentication are supported.

The OAuth scope stored for each Custom Job Type informs POM about which scopes are to be used when invoking the Custom Job Type APIs.

POM makes calls to these Job Type endpoints by using this scope and the default IDCS credentials it is configured with. However, these endpoints are generally hosted on an application that has its own IDCS application. In such cases, POM cannot invoke those endpoints using its default IDCS credentials. Therefore, POM provides an option to configure a custom IDCS slice and custom credentials to be used for this type of authentication. These credentials are the Client Id & Secret of the respective application/ OAuth client, as shown below. POM calls the custom job type endpoints that can be authenticated using an IDCS instance different that POM’s. Custom credentials are also published to agents during new scheduler day creation.

It is expected that the IDCS Client App that these custom credentials point to will contain the necessary scopes to call these endpoints.

Job Start API

HTTP Method

POST

Request Payload (JSON)

The attributes of the JSON payload are as follows :

Attribute	Description
cycleName	POM Batch Cycle
flowName	POM Batch Flow.
processName	POM Batch process name
jobName	POM Batch job name
parameters	Job Parameters (Double-pipe delimited key-value pairs) as defined in POM
agentExecutionId	A unique ID assigned by POM for every job run

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of a successful invocation, a success HTTP code (2xx) is returned with a payload like the following:

{
   "executionId": "112",
   "executionInfo": Job started,
   "status": "STARTED"
}

Attribute	Description
executionId	Unique Id returned by the target app to POM for status tracking
status	Any status
executionInfo (optional)	Any additional info the target app would like to share with POM.

In case of an issue, an HTTP code other than the success code (that is, not 2xx) is sent along with an error payload of any format which will be captured as part of the Job failure logs.

Job Status API

HTTP Method

GET

Query Parameters (JSON)

Attribute	Description
processName	POM Batch process name
jobName	POM Batch job name
executionId	Unique Id returned by the target app to POM for status tracking

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of a successful invocation, a success HTTP code (2xx) is returned with a payload like the following:

{
   "status": "COMPLETED",
   "executionInfo": null,
   "notification": {
      "info": "Number of stock counts: 231",
      "type": "StockCountErrorNotification",
      "severity": 1
   }
}

Attribute	Description
executionId	Unique Id returned by the target app to POM for status tracking
status	RUNNING / ERROR / COMPLETED.
executionInfo (optional)	Any additional info the target app would like to share with POM.
notification	An optional Notification Object containing details regarding an Application’s Notification that needs to be generated through POM.
notification.info	Message to be used for the App Notification
notification.type	Notification Type to be used for the App Notification. This must be an existing Notification Type on the Consuming Application.
notification.severity	Severity of the App Notification generated. Valid values are 1-Critical, 2-Important, 3-Normal

In case of an issue, an HTTP code other than the success code (that is, not 2xx) is sent along with an error payload of any format which will be captured as part of the Job failure logs.

Job Restart API

HTTP Method

POST

Request Payload (JSON)

The attributes of the request payload are as follows:

Attribute	Description
cycleName	POM Batch Cycle
flowName	POM Batch Flow.
processName	POM Batch process name
jobName	POM Batch job name
parameters	Job Parameters (Double-pipe delimited key-value pairs) as defined in POM
agentExecutionId	A unique ID assigned by POM for every job run
executionId	Unique Id of the previously failed execution returned by the target app to POM for status tracking

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of a successful invocation, a success HTTP code (2xx) is returned with a payload like the following:

{
   "executionId": "112",
   "executionInfo": Job re-started,
   "status": "STARTED"
}

Attribute	Description
executionId	Unique Id returned by the target app to POM for status tracking
status	Any status
executionInfo (optional)	Any additional info the target app would like to share with POM.

In case of an issue, an HTTP code other than the success code (that is, not 2xx) is sent along with an error payload of any format which will be captured as part of the Job failure logs.

Job Kill API

HTTP Method

POST

Request Payload (JSON)

The attributes of the request payload are as follows:

Attribute	Description
processName	POM Batch process name
jobName	POM Batch job name
executionId	Unique Id of the previously failed execution returned by the target app to POM for status tracking

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of a successful invocation, a success HTTP code (2xx) is returned with a payload like the following:

{
   "executionId": "112",
   "executionInfo": Job Killed,
   "status": "CANCELLED"
}

Attribute	Description
executionId (optional)	Unique Id returned by the target app to POM for status tracking
status (optional)	Any status
executionInfo (optional)	Any additional info the target app would like to share with POM.

In case of an issue, an HTTP code other than the success code (that is, not 2xx) is sent along with an error payload of any format which will be captured as part of the Job failure logs.

Job Log API

HTTP Method

GET

Query Parameters (JSON)

Attribute	Description
processName	POM Batch process name
jobName	POM Batch job name
executionId	Unique Id returned by the target app to POM for status tracking

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

In case of a successful invocation, the response will contain the log data for the execution provided. This log will be viewable in the log downloaded from the POM UI. The logs from the response are also sent in the job log.

Any other response while calling this endpoint would be captured as an error while downloading logs.

Job System Check API

HTTP Method

GET

Authentication

OAuth System Credentials Grant

Response Payload (JSON)

A success response of 200 or 204 indicates that the system is up and running.

Any other response while calling this endpoint, should be interpreted as the system being down.