6 Configuring Scheduler Resilience (Release 9.2.2.4)

This chapter contains the following topics:

6.1 Understanding Scheduler Resilience

The scheduler is used to run notifications and orchestrations at designated intervals defined by the schedule attached to each. The scheduler runs as a process on one or more AIS Servers. As notifications and orchestrations are incorporated into critical business processes, the scheduling and execution of those notifications and orchestrations becomes equally critical. The scheduler must be able to tolerate failures, restart with minimum human intervention, and load balance in that if one scheduler instance begins to fall behind, another can pick up the overflow.

Starting a notification or orchestration on a scheduler to run at some interval creates a scheduled job with a set of properties. To make the scheduler resilient, you can store the scheduled job properties for notifications and orchestrations centrally in the database. This means that if you are running a single instance of the scheduler, the scheduler can restart from a failure and continue processing from the queue of scheduled jobs without an administrator having to resubmit all jobs manually. Or if you are running multiple instances of the scheduler, the schedulers can each take scheduled jobs from the queue. If any single instance of a scheduler fails, the other schedulers can proceed independently for continuous operations.

Description of resilientschedulerflow.png follows
Description of the illustration ''resilientschedulerflow.png''

In this graphic, the following steps occur:

  1. The user requests notifications and/or orchestrations to run on their schedule using the scheduler service.

  2. The scheduler service creates a job for each of the notifications and/or orchestrations.

  3. The job definitions are stored in a database.

  4. The scheduler on one of the AIS Servers finds a job to run when that job's scheduled interval is reached.

  5. The job calls a notification or orchestration service.

6.2 Creating Tables in the Database

In order to store scheduled jobs in the database, you need to create the Quartz tables and indexes where the jobs will be stored. First, obtain the appropriate create script for Quartz-2.2.3 and the database you are using. Modify the script as necessary to create the tables in a database accessible from your AIS Server. Although these are not standard JD Edwards tables, they can be added to the JD Edwards EnterpriseOne database.

For more information on obtaining the scripts and creating these tables, see the "Setting up Quartz Scheduler Resilience to be used with Notifications and Orchestrations" document on My Oracle Support at: https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=555331132725298&id=2368608.1

6.3 Installing the JDBC Driver

The AIS Servers containing the scheduler will access the Quartz runtime database through JDBC. If your AIS Server is deployed to WebLogic and your database is Oracle, the JDBC driver will already have been installed and you can skip this step.

For more information on obtaining, uploading, and installing JDBC drivers, see the Manage JDBC Drivers chapter in the JD Edwards EnterpriseOne Tools Server Manager Guide.

6.4 Configuring AIS Server Manager Settings

You can configure the following settings for the Scheduler. You can access and change these in Server Manager under Configuration, Advanced, Miscellaneous, Scheduler Configuration. The settings are:

  • This is a Scheduler Server. Select this option to designate this AIS server as a Scheduler server. The Scheduler server will automatically start when the associated AIS server is started. As a result, the jobs that are designated as auto-start jobs will be automatically started when the Scheduler server is started. See, Working with Scheduler (Release 9.2.4)

    The auto-start process requires the following information to ping the server until it is up before continuing. Selecting the "This is a Scheduler Server" option required you to enter the following details:

    • Scheduler Bootstrap User. User name used when retrieving the Scheduler auto-start jobs.

    • Scheduler Bootstrap Password. Password used when retrieving the Scheduler auto-start jobs. The password is encrypted in the Server Manager.

    • Scheduler Bootstrap Environment. Environment used when retrieving the Scheduler auto-start jobs.

    • AIS Scheduler Instance URL. Enter the URL for this AIS Scheduler server instance used for Scheduler auto-start processing. (Protocol://Server:Port) Protocol is http or https.

  • Scheduler Resilience. Enable this option in order to store the runtime data in a database.

  • Number of Threads. The number of threads allocated to the scheduler, from 1 through 100. The default value is 10.

  • Thread Priority. The priority of scheduler threads as compared to the priority of other threaded processes on the server. Valid values are between 1 and 10. The default value is 5.

  • JDBC Max Connections. The maximum number of connections to the scheduler database.

  • JDBC Driver Class Name. The java class name of the database driver for the database that contains scheduler runtime data.

    Examples of driver class names are:

    • MS SQL Server: com.microsoft.sqlserver.jdbc.SQLServerDriver

    • Oracle: oracle.jdbc.driver.OracleDriver

    • IBMi: com.ibm.as400.access.AS400JDBCDriver

  • JDBC URL. The URL where the database storing the Quartz tables is deployed.

    Examples of URLs are:

    • MS SQL Server: jdbc:sqlserver://<HOST>:<PORT>;databaseName=<DATABASE_NAME>

    • Oracle: jdbc:oracle:thin:@<HOST>:<PORT>:<SID> OR Jdbc:oracle:thin:@<HOST>:<PORT>/<SERVICENAME>

    • IBMi: jdbc:as400://<HOST>/<SCHEMA>;libraries=<*LIBL>

  • JDBC UserName. The user name used when creating the Quartz tables.

  • JDBC Password. The password used when creating the Quartz tables.

The Server Manager interface provides more details on the usage of each setting. Click the "i" (Information) icon next to each in Server Manager for more information.

6.5 Scheduler Administration

Starting with Tools 9.2.4, you can start and stop the scheduled notification and orchestration jobs using the Scheduler user interface page in the Orchestrator Studio 9.2.4.0. For information on starting and stopping the notification and orchestration jobs using the Orchestrator Studio, see "Working with Scheduler" in the JD Edwards EnterpriseOne Orchestrator Guide.

Alternatively, you can also use a set of REST APIs for starting, stopping, and managing scheduled jobs.

Schedule REST APIs respect UDO security and feature security. When you start a notification or orchestration job, the user starting the job is authenticated to determine what is started. Only notifications and orchestrations to which that user has access are started.

There is also a REST API you can use for troubleshooting. The List API is similar to a log file in that it contains information such as number of errors, percentage of errors, exception messages, and time to execute for jobs that are currently executing. It does not contain all history like an actual log file would. It only shows the last exception message that occurred.

These are the services you can use to administer the scheduler:

Endpoint Behavior
<URL>/jderest/v2/scheduler/start Starts all notifications and orchestrations for which the service caller has permissions.
<URL>/jderest/v2/scheduler/stop Stops all notifications and orchestrations. They are stopped permanently and will need to be restarted.
<URL>/jderest/v2/scheduler/startjob/ORC_1707250005TOOLS Starts a single notification. URL ends with /<OMW NAME> (short or long).
<URL>/jderest/v2/scheduler/stopjob/ORC_1707250005TOOLS Stops a single notification. URL ends with /<OMW NAME> (short or long).
<URL>/jderest/v2/scheduler/list Use to view runtime information about what is currently running.
<URL>/jderest/v2/scheduler/listJobs (Release 9.2.3.4) Use to view information about the collection of Notifications and Orchestrations that are scheduled to run, are currently running, or have no schedule attached.
<URL>/jderest/v2/scheduler/listExecuting (Release 9.2.3.4) Use to view the attributes of all the Notifications and Orchestrations that are currently executing on the server.
<URL>/jderest/v2/scheduler/view (Release 9.2.3.4) Use to view information about all the Notifications and Orchestrations. You can add an optional parameter of filterScheduled=true which will limit the returned data to Notifications or Orchestrations that have an attached schedule.
<URL>/jderest/v2/scheduler/startJobs (Release 9.2.3.4) Starts a collection of Notifications and Orchestrations in the scheduler.
<URL>/jderest/v2/scheduler/stopJobs (Release 9.2.3.4) Stops a collection of Notifications and Orchestrations in the scheduler.

Starting with Tools 9.2.4, the format of "jobname" for an orchestration and a notification is unique with the combination of:

  • Notification ID or Orchestration ID

  • User Name

  • Environment

  • Role

This enables you to execute multiple instances of a job based on the combination of Notification ID or Orchestration ID, User Name, Environment, and Role.

As a result, the services like start, startjob, startjobs, stop, stopjob, stopjobs, view, list, and listjobs will return specific jobs for each unique combination of Notification ID or Orchestration ID, User Name, Environment, and Role.

Example: For the startJob service, endpoint of the URL can be the Notification or Orchestration ID (long or short). However, the job that the service starts is unique to the Notification or Orchestration ID along with the authenticated user, environment, and role.

When you enter the endpoint of the URL as the Notification or Orchestration ID for the stopJob service, all the instances of the jobs with the specified Notification or Orchestration ID are stopped. A job name can be passed as NTF/ORCH_ID.USER.ENV.ROLE (such as: NTF_1903210002JDE.ABC.JDV920.*ALL) to stop a specific job.

An error message is returned when you start a job that is already running. You have to manually stop the job before the job with the same Notification or Orchestration ID, user, environment, and role can be started.

A resilient scheduler can be started and stopped while not affecting the jobs that have previously been scheduled to run.

Starting and stopping scheduler instances are done with these services:

Endpoint Behavior
<URL>/jderest/v2/scheduler/startSchedulers Start a collection of schedulers, one per AIS host.
<URL>/jderest/v2/scheduler/startScheduler Start a scheduler - the AIS server where the service is running is the scheduler host.
<URL>/jderest/v2/scheduler/stopSchedulers Stop a collection of schedulers, one per AIS host.
<URL>/jderest/v2/scheduler/stopScheduler Stops a scheduler. The AIS server where the service is running is the scheduler host.
<URL>/jderest/v2/scheduler/pingSchedulers Check whether a collection of schedulers are up, one per AIS host.
<URL>/jderest/v2/scheduler/pingScheduler Checks whether the scheduler is up. The AIS server where the service is running is the scheduler host.

All of these services use the same service body. A single schedulerHost is valid.

Service Body

{
                "username": "<user>",
                "password": "<password>",
                "environment": "<environment>",
                "role": "<role>",
                "scheduleIntervalMinutes": 1,           
                "schedulerHosts": [{
                       "protocol": "<protocol>",
                       "host": "<AIS host>",
                       "port": "<port>"
                                },
                   {
                       "protocol": "<protocol>",
                       "host": "<AIS host 2>",
                       "port": "<port>"
                                }
 
                        ]
       }   

Note:

If you have not configured scheduler resilience, jobs are run on a single instance and are volatile. Stopping the AIS Server or the scheduler causes the jobs to be cleared. Without resilience, multiple schedulers should not be started because they will duplicate the notification and orchestration executions from the other scheduler instances.

For more information on REST APIs used for starting, stopping, and managing the scheduler, see "Scheduler Service" in the JD Edwards EnterpriseOne Tools REST API for the Application Interface Services Server Guide.

6.6 Troubleshooting Misfires

The AIS log displays if notifications and orchestrations are backing up. In other words, some may be running long and causing others to be placed in an exception queue where the scheduler will try to pick them up later.

Note:

A job misfire is logged only if it is five minutes overdue.

Here is an example of what the message might look like:

13 Feb 2018 17:53:18,007 [SEVERE] - [AIS]             Scheduler Trigger for Name: NTF_1712060007JDE, and Group SCH_1802080001JDE (Notification/Orchestration:  description , Schedule: description) MISFIRED. This condition is caused when notifications or orchestrations are not starting close to their scheduled start time. This may occur if other notifications or orchestrations have not yet completed, indicating that more jobs are scheduled than the system can complete within the schedule time. This can be fixed by scheduling the jobs to run less frequently, staggering the schedules, or by increasing the number of threads in the Scheduler, which may require additional system resources.

Some possible solutions for this issue include:

  • Decrease Frequency. There may be jobs scheduled to run every 5 minutes that can run every 30 minutes.

  • Staggering. Do not schedule too many jobs to run at the same time, or in multiples of the same number. Schedules that are set to run every 5, 10, 15, and 30 minutes will all attempt to run all of the notifications and orchestrations at the same time at the 30-minute mark. Using numbers such as 7, 17, 31, and 39 will balance the load better.

  • More Threads. The scheduler uses one thread each time a job is triggered to run on its schedule. If more jobs are triggered to run than there are available threads, a group of jobs will be picked up, one for each thread while the other jobs wait for their own thread. These other jobs will wait for a thread to be returned by a job that has completed execution. Increasing the threads will allow more jobs to run at one time, but might tax system resources with the added activity.

  • Clustering. Additional schedulers can pick up jobs that might otherwise become misfires. If the first scheduler has 10 threads, misfires could result when more than 10 jobs are scheduled to run at one time. Adding a second scheduler with 10 threads doubles the number of simultaneous jobs that can be running. There is no explicit load balancing between the two schedulers. The first one to examine scheduled jobs runs as many as it can. Only the remaining scheduled jobs are picked up by the next scheduler. In the preceding example of two schedulers with 10 threads each, if there are 12 jobs that are running on the same schedule, the first server to acquire a set of jobs might acquire 10, leaving the remaining 2 for the next scheduler. Reducing the threads on the first server to 8 might mean that 4 jobs are picked up by the next scheduler. Multiple schedulers running at the same time do not know about each other, so attempting to load balance across schedulers is an administrative task. Also, if the two AIS Servers point to the same HTML Server, little or no benefit will be gained.

6.7 Testing Your Configuration

Testing the configuration requires tools such as Postman, SoapUI, or cURL to run RESTful services. This guide does not provide training in running the web services.

After the configuration is complete, the next step is to test that the scheduler is running. There is a RESTful web service that submits a job and validates that it is being picked up from the database.

6.7.1 Start Cluster Unit Test Service

The End Point for this example is:

<URL>/jderest/v2/scheduler/startClusterUnitTest

In this example, <URL> is the AIS Server where the scheduler runs.

In the following example, replace the values that look like <xxxx> with the appropriate values.

Service Body

{
          "username": "<user>",
          "password": "<password>",
          "environment": "<environment>",
          "role": "<role>",
          "scheduleIntervalMinutes": 1,
          "schedulerHosts": [{
                "protocol": "<protocol>",
               "host": "<AIS host>",
               "port": "<port>",
                     }
          ]
}

For more information on REST APIs used for starting, stopping, and managing the scheduler, see "Scheduler Service" in the JD Edwards EnterpriseOne Tools REST API for the Application Interface Services Server Guide guide.

6.7.2 List Service

This job will be picked up each minute by the scheduler running on the AIS Server. Using the list service, you can see the job running.

The End Point for this example is:

<URL>/jderest/v2/scheduler/list

In this example, <URL> is the AIS Server where the scheduler runs.

Service Body

{
curl -X POST -H "Content-Type:application/json" http://ais_server_url/jderest/v2/scheduler/list -d {  
 "token": "044QF2SLgaM6vZX081eq8KsVi6XcJiiFL5un5ACH+eBGUg=MDE5MDEyMTY4NzY4NjcwMjI2NzExNzcyNDEwLjE1OS45OS43MzE0NzkxNDU4NDM4ODU="
}

Return Body

{
     "jobs": [
        {
            "jobLastExecutionBaseUrl": "http://an_ais_host:7101",
            "jobEndpointRequested": "UnitTest_a",
            "serviceShortEndpoint": "UnitTest_a",
            "serviceLongEndpoint": "Unit Test a",
            "omwServiceDescription": "Job For Unit Test ID = a",
            "omwScheduleDescription": "Run every 1 minutes",
            "scheduleIntervalMinutes": 1,
            "jobname": "a",
            "jobgroup": "DEFAULT"
        }
    ],
"scheduler": {
     "isResilient": false,
     "isStarted": true,
     "runningSinceUTC": "2019-03-28T18:08:35.305Z",
     "runningSinceUserLocale": "03/28/2019 12:08:35 PM",
     "schedulerInstanceJobsExecuted": 13
  }
}

Note:

Irrelevant values have been removed from the return body.

6.7.3 List Jobs Service (Release 9.2.3.4)

This service lists the status of one or more notifications and orchestrations that are scheduled to run.

The End Point for this example is:

<URL>/jderest/v2/scheduler/listJobs

In this example, <URL> is the AIS Server where the scheduler runs.

Service Body

{
    "username": "{{username}}",
    "password": "{{password}}",
    "environment": "{{environment}}",
    "role": "{{role}}",
    "schedulerServices": 
    [
        {
            "serviceEndpoint" : "{{JOBA}}"
        },
        {
            "serviceEndpoint" : "{{JOBB}}"
        }
    ]
}

Return Body

{
    "scheduledInSchedulerCount": 2,
    "notScheduledInSchedulerCount": 0,
    "scheduledInScheduler": [
        {
            "jobname": "NTF_1903210002JDE.USER.JDV920.*ALL",
            "jobgroup": "NTF_1903210002JDE",
            "jobVersion": "V2",
            "jobStartTimeUserLocale": "06/17/2019 11:20:41 AM",
            "jobStartTimeUTC": "2019-06-17T17:20:41.124Z",
            "jobNextFireTimeUserLocale": "06/17/2019 11:20:41 AM",
            "jobNextFireTimeUTC": "2019-06-17T17:20:41.124Z",
            "jobNextFireTime": "2019-06-17 11:20:41 124",
            "jobLastExecutionTimeMillis": 0,
            "jobTotalExecutions": 0,
            "jobTotalCompletions": 0,
            "jobTotalErrors": 0,
            "jobConsecutiveErrors": 0,
            "jobPercentageErrors": 0,
            "udoServiceType": "NOTIFICATION",
            "jobEndpointRequested": "NTF_1903210002JDE",
            "omwServiceDescription": "\"A\" Search Type LI AB LI WL 1 and 2",
            "jobServicePath": [
                "jderest",
                "v2",
                "notification"
            ],
            "serviceShortEndpoint": "NTF_1903210002JDE",
            "serviceLongEndpoint": "Search Type LI AB LI WL  1 and 2",
            "jobLastExecutionBaseUrl": "http://127.0.0.1:7101",
            "jobBaseUrl": "http://127.0.0.1:7101",
            "omwScheduleDescription": "LISTS JOBS",
            "scheduleCronString": "0 0/1 * 1/1 * ? *",
            "scheduleIntervalMinutes": -1,
            "userName": "USER",
            "environment": "JDV920",
            "role": "*ALL"
        },
        {
            "jobname": "NTF_1903210003JDE.USER.JDV920.*ALL",
            "jobgroup": "NTF_1903210003JDE",
            "jobVersion": "V2",
            "jobStartTimeUserLocale": "06/17/2019 11:20:41 AM",
            "jobStartTimeUTC": "2019-06-17T17:20:41.214Z",
            "jobNextFireTimeUserLocale": "06/17/2019 11:20:41 AM",
            "jobNextFireTimeUTC": "2019-06-17T17:20:41.214Z",
            "jobNextFireTime": "2019-06-17 11:20:41 214",
            "jobLastExecutionTimeMillis": 0,
            "jobTotalExecutions": 0,
            "jobTotalCompletions": 0,
            "jobTotalErrors": 0,
            "jobConsecutiveErrors": 0,
            "jobPercentageErrors": 0,
            "udoServiceType": "NOTIFICATION",
            "jobEndpointRequested": "NTF_1903210003JDE",
            "omwServiceDescription": "\"B\" Search Type LI AB LI WL 2 and 3",
            "jobServicePath": [
                "jderest",
                "v2",
                "notification"
            ],
            "serviceShortEndpoint": "NTF_1903210003JDE",
            "serviceLongEndpoint": "Search Type LI AB LI WL 2 and 3",
            "jobLastExecutionBaseUrl": "http://127.0.0.1:7101",
            "jobBaseUrl": "http://127.0.0.1:7101",
            "omwScheduleDescription": "LISTS JOBS",
            "scheduleCronString": "0 0/1 * 1/1 * ? *",
            "scheduleIntervalMinutes": -1,
            "userName": "USER",
            "environment": "JDV920",
            "role": "*ALL"
        }
    ],
    "notScheduledInScheduler": [],
    "scheduler": {
        "isResilient": false,
        "isStarted": false,
        "schedulerInstanceJobsExecuted": 0
    }
}

6.7.4 List Executing Service (Release 9.2.3.4)

This service displays the attributes of all the notifications and orchestrations that are currently executing on the server.

The End Point for this example is:

<URL>/jderest/v2/scheduler/listExecuting

In this example, <URL> is the AIS Server where the scheduler runs.

Service Body

{
    "username": "{{username}}",
    "password": "{{password}}",
    "environment": "{{environment}}",
    "role": "{{role}}",
    "schedulerServices": 
    [
        {
            "serviceEndpoint" : "{{JOBA}}"
        },
        {
            "serviceEndpoint" : "{{JOBB}}"
        }
    ]
 
}

Return Body

{
    "executingJobs": [
        {
            "serviceShortEndpoint": "NTF_1903210001JDE",
            "serviceLongEndpoint": "Search Type LI AB LI WL 2 and 4",
            "omwServiceDescription": "\"C\" Search Type LI AB LI WL 2 and 4",
            "startTime": "2019-06-17T17:20:41.273Z",
            "fireTime": "2019-06-17 11:38:14 220",
            "fireTimeUTC": "2019-06-17T17:38:14.220Z",
            "fireTimeUserLocale": "06/17/2019 11:38:14 AM",
            "executionMillis": 9409,
            "exectionTime": "00:00:09.409"
        },
        {
            "serviceShortEndpoint": "NTF_1903210003JDE",
            "serviceLongEndpoint": "Search Type LI AB LI WL 2 and 3",
            "omwServiceDescription": "\"B\" Search Type LI AB LI WL 2 and 3",
            "startTime": "2019-06-17T17:20:41.214Z",
            "fireTime": "2019-06-17 11:38:14 220",
            "fireTimeUTC": "2019-06-17T17:38:14.220Z",
            "fireTimeUserLocale": "06/17/2019 11:38:14 AM",
            "executionMillis": 9410,
            "exectionTime": "00:00:09.410"
        },
        {
            "serviceShortEndpoint": "NTF_1903210002JDE",
            "serviceLongEndpoint": "Search Type LI AB LI WL  1 and 2",
            "omwServiceDescription": "\"A\" Search Type LI AB LI WL 1 and 2",
            "startTime": "2019-06-17T17:20:41.124Z",
            "fireTime": "2019-06-17 11:38:14 198",
            "fireTimeUTC": "2019-06-17T17:38:14.198Z",
            "fireTimeUserLocale": "06/17/2019 11:38:14 AM",
            "executionMillis": 9433,
            "exectionTime": "00:00:09.433"
        }
    ]
}

6.7.5 View Service (Release 9.2.3.4)

This service displays all the notifications and orchestrations that you have UDO permissions to. This enables you to determine whether a job is scheduled and also to find out whether a job is currently running. Using filterScheduled=true parameter, you can view only the notifications and orchestration that have a schedule attached.

The End Point for this example is:

<URL>/jderest/v2/scheduler/view

In this example, <URL> is the AIS Server where the scheduler runs.

Service Body

curl -X POST -H "Content-Type:application/json" http://ais_server_url/jderest/v2/scheduler/view filterScheduled=true -d 
{   
"token": "044QF2SLgaM6vZX081eq8KsVi6XcJiiFL5un5ACH+eBGUg=MDE5MDEyMTY4NzY4NjcwMjI2NzExNzcyNDEwLjE1OS45OS43MzE0NzkxNDU4NDM4ODU="
}

Return Body

{
"scheduler": {
"isResilient": true,
"isStarted": true,
"runningSinceUTC": "2019-03-04T20:23:37.613Z",
"schedulerInstanceJobsExecuted": 11
},
"services": [{
"environment": "JDV920",
"role": "*ALL",
"deviceName": "127.0.0.1",
"ssoEnabled": false,
"username": "USER",
"jobEndpointRequested": "NTF_1808170001TOOLS",
"scheduleIntervalMinutes": -1,
"jobServicePath": [
"jderest",
"v2",
"notification"
],
"scheduleCronString": "0 0/1 * 1/1 * ? *",
"omwScheduleDescription": "0 0/1 * 1/1 * ? *",
"omwScheduleObjectName": "SCH_1808170001TOOLS",
"omwScheduleLongName": "Cron Minutely",
"omwScheduleOwner": "USER",
"schedule": {
"name": "Cron Minutely",
"description": "0 0/1 * 1/1 * ? *",
"omwObjectName": "SCH_1808170001TOOLS",
"udoOwner": "USER",
"cronString": "0 0/1 * 1/1 * ? *",
"intervalType": "cronString"
},
"omwServiceObjectName": "NTF_1808170001TOOLS",
"omwServiceOwner": "USER",
"omwServiceDescription": "UserMailingType",
"serviceShortEndpoint": "NTF_1808170001TOOLS",
"serviceLongEndpoint": "UserMailingType",
"udoServiceType": "NOTIFICATION",
"serviceScheduledInScheduler": true,
"jobInScheduler": {
"jobname": "NTF_1808170001TOOLS.USER.JDV920.*ALL",
"jobgroup": "NTF_1808170001TOOLS",
"jobVersion": "V2",
                "jobStartTimeUserLocale": "03/26/2019 02:40:50 PM",
                "jobStartTimeUTC": "2019-03-26T20:40:50.843Z",
                "jobNextFireTimeUserLocale": "03/26/2019 02:42:00 PM",
                "jobNextFireTimeUTC": "2019-03-26T20:42:00.000Z",
                "jobNextFireTime": "2019-03-26 14:42:00 0",
"jobTotalExecutions": 86,
"jobTotalCompletions": 86,
"jobTotalErrors": 0,
"jobConsecutiveErrors": 0,
"jobPercentageErrors": 0,
"udoServiceType": "NOTIFICATION",
"jobEndpointRequested": "NTF_1808170001TOOLS",
"omwServiceDescription": "UserMailingType",
"jobServicePath": [
"jderest",
"v2",
"notification"
],
"serviceShortEndpoint": "NTF_1808170001TOOLS",
"serviceLongEndpoint": "UserMailingType",
"jobLastExecutionBaseUrl": "http://127.0.0.1:7101",
"jobBaseUrl": "http://127.0.0.1:7101",
"omwScheduleDescription": "0 0/1 * 1/1 * ? *",
"scheduleCronString": "0 0/1 * 1/1 * ? *",
"scheduleIntervalMinutes": -1,
"userName": "USER",
"environment": "JDV920",
"role": "*ALL"
},
"executingInSchedulerInstance": {
"serviceShortEndpoint": "NTF_1808170001TOOLS",
"serviceLongEndpoint": "UserMailingType",
"omwServiceDescription": "UserMailingType",
"startTimeUTC": "2019-03-04T16:12:29.124Z",
"fireTime": "2019-03-04 13:32:08 434",
"fireTimeUTC": "2019-03-04T20:32:08.434Z",
"exectionTime": "00:00:48.740",
"executionMillis": 48740
}
]
}

6.7.6 Analyze Unit Test State Service

To view the results, verifying a successful configuration, run the analyzeUnitTestState RESTful web service.

The End Point for this example is:

<URL>/jderest/v2/scheduler/analyzeUnitTestState

In this example, <URL> is the AIS Server where the scheduler runs.

In the service body, replace the values that look like <xxxx> with the appropriate values.

Service Body

{
          "username": "<user>",
          "password": "<password>",
          "environment": "<environment>",
          "role": "<role>",
}

Return Body

{
    "unitTestResults": {
        "testSucceeded": true,
        "errorEventIndex": [],
        "errorEvent": [],
        "unitTestEvents": [
            {
                "environment": "JDV920",
                "role": "*ALL",
                "jasserver": null,
                "token": null,
                "deviceName": null,
                "ssoEnabled": false,
                "ssoUniqueId": null,
                "psToken": null,
                "username": "jde",
                "jasProcessingInstruction": null,
                "unitTestJobBehavior": {
                    "unitTestStartupHost": {
                        "protocol": "http",
                        "host": "an_ais_host_1",
                        "port": "7101"
                    },
                    "id": "a",
                    "unitTestHostActions": [],
                    "unitTestHostCount": 1,
                    "runningOnSingleServer": true
                },
                "executionHost": "http:// an_ais_host_1:7101"
            },
            {
                "environment": "JDV920",
                "role": "*ALL",
                "jasserver": null,
                "token": null,
                "deviceName": null,
                "ssoEnabled": false,
                "ssoUniqueId": null,
                "psToken": null,
                "username": "jde",
                "jasProcessingInstruction": null,
                "unitTestJobBehavior": {
                    "unitTestStartupHost": {
                        "protocol": "http",
                        "host": " an_ais_host_2",
                        "port": "7101"
                    },
                    "id": "a",
                    "unitTestHostActions": [],
                    "unitTestHostCount": 1,
                    "runningOnSingleServer": true
                },
                "executionHost": "http:// an_ais_host_1:7101"
            }
        ],
        "jobsStopped": null
    }
}

The most important value in this return body is the "testSucceeded": "true" pair. Everything else describes details of the successful executions of the unit test by the scheduler. In this case, there are two "unitTestEvents," meaning that the unit test ran twice. Because it is successful, your scheduler resilience is properly configured. If a different payload is returned or the testSucceeded is false, check the logs for that AIS Server.

When you have completed your testing, stop the server using the stop RESTful web service described previously.

6.8 Configuring More Scheduler Instances

To configure more scheduler instances, follow the instructions for configuring the first instance on a separate AIS host machine.

Start the new scheduler using the REST APIs mentioned in Scheduler Administration.

Follow the same steps for testing a single instance, but change the list of hosts to include the new host. All the server clocks should be synchronized.

The End Point for this example is:

<URL>/jderest/v2/scheduler/startClusterUnitTest

In this example, <URL> is the AIS Server where the scheduler runs.

In the following example, replace the values that look like <xxxx> with the appropriate values.

Service Body

{
          "username": "<user>",
          "password": "<password>",
          "environment": "<environment>",
          "role": "<role>",
          "scheduleIntervalMinutes": 1,
          "schedulerHosts": [{
                "protocol": "<protocol>",
               "host": "<AIS host>",
               "port": "<port>",
                     },
{
                "protocol": "<protocol>",
               "host": "<AIS host 2>",
               "port": "<port>",
                     }
          ]
}

The results should be the same as with a single instance, except the "ID" values will iterate through the alphabet -- one sequential letter for each scheduler instance, i.e. unitTestEvent ID "a", then "b", then "c" for three hosts. Make sure that all previous unit test jobs have been stopped using the /stop service or the /stopjob service mentioned previously.