Using the Test Suites REST API

The Test Suites REST API provides the ability to maintain and assess user-defined test suites and review the results of the executed tests. Test suites are associated with one or more deployments or web projects and are populated with test cases. Each test case provides input data to be used for assessment and the expected assessment output. Test runs are created to assess all test cases in a suite against the most recent version of the associated deployments or web projects. The output of a test run provides details of the test cases assessed for each deployment or project version, identifying whether the assessed output matched the expected output.

The following diagram provides an overview of the components of the Test Suites REST API and how they interact:

REST API - Clients interact with the Hub Testing API as resources within the Hub REST API. The REST API can be used to:

• Create, read, modify, or delete test suites, and test cases.

• Create new test runs for processing, and cancel or delete test runs.

• Read, or delete test assessment results for test runs.

Test Runner - The Test Runner is a regularly scheduled background task managed by the Hub, responsible for initiating processing of test runs. The Test Runner will:

• Collect any new test runs that have not yet started processing.

• Initialise the test runs with new "pending" test case assessments for each combination of test case and most recent deployment/project version associated with the containing test suite.

• Add a new Test Processor task to the Hub job scheduler for each initialised test run.

Test Processor - A Test Processor is an ad-hoc background task created by the Test Runner when required, responsible for processing an individual test run. A Test Processor will:

• Batch and send "pending" test case assessments as test assessment requests to Determinations Server.

• Use the responses returned by Determinations Server to update the status and outcome details of the test case assessments.

• Manage the overall state of the test run.

Determinations Server - The Determinations Server Test Service handles test assessment requests sent by the Hub. For each request, the Test Service will:

• Reject the request if is not authenticated with a system-generated Hub-specific auth token.

• Retrieve and cache the identified deployment/project version using the existing Hub backplane channel.

• Assess each test case against the identified deployment/project version.

• Return a response containing the output from each assessed test case.

Test Cleaner - The Test Cleaner is a regularly scheduled asynchronous task managed by the Hub, responsible for cleaning invalidated test data. For performance reasons, client requests to delete test suites and test runs are handled as soft-deletions. Top-level records are invalidated, updating one database record only instead of performing a cascading deletion of a potentially large number of database records. When invoked, the Test Cleaner will:

• Invalidate expired test runs if configured.

• Delete blocks data associated with invalidated test runs and test suites.

Test case definition

The structure required for the input and expected output of a test case depends on the "testType" of the containing test suite:

• "decisionService": Uses the same format as the Decision Service REST API for web projects

• "batchAssess": Uses a format similar to the Batch Assess REST API for Policy Modeling deployments

The "input" represents the data that will be submitted for assessment by the identified deployment/project, and the "expectedOutput" represents the exact output expected by assessing the "input".

Decision service test case example

The following is an example of a REST resource describing a decision service test case, with the input and expected output:

{
   "name": "test-case-1",
   "description": "The first test case",
   "enabled": true,
   "tags": ["tag-1", "tag-2"],
   "input": {
      "parent-name": "Barry",
      "parent-height-cm": 195,
      "children": [
         {"child-name": "Harry", "child-height-cm": 180},
         {"child-name": "Kerry", "child-height-cm": 160}
      ]
   },
   "expectedOutput": {
      "parent-taller-than-all": true
      "children": [
         {"child-name": "Harry", "child-taller-than-parent": false},
         {"child-name": "Kerry", "child-taller-than-parent": false}
      ]
   }
}

The "input" and "expectedOutput" properties respectively use the format of an Input contract and Output contract of a decision service.

Batch assess test case example

The following is an example of a REST resource describing a batch assess test case, with input and expected output:

{
   "name": "test-case-1",
   "description": "The first test case",
   "enabled": true,
   "tags": ["tag-1", "tag-2"],
   "input": {
      "case": {
         "@id": 1,
         "parent-name": "Barry",
         "parent-height-cm": 195,
         "children": [
            {"@id": 1, "child-name": "Harry", "child-height-cm": 180},
            {"@id": 2, "child-name": "Kerry", "child-height-cm": 160}
         ]
      },
      "outcomes": ["parent-taller-than-all", "child-name", "child-taller-than-parent"]
   },
   "expectedOutput": {
      "@id": 1,
      "parent-taller-than-all": true
      "children": [
         {"@id": 1, "child-name": "Harry", "child-taller-than-parent": false},
         {"@id": 2, "child-name": "Kerry", "child-taller-than-parent": false}
      ]
   }
}

The format of the "input" data is similar to a single assessment expected by the Batch Assess REST API, but with a singular "case" property instead of the "cases" array property. The format of the "expectedOutput" property is that of a single case from within the "cases" array of a response from the Batch Assess REST API.

Test case assessments

The system will use the details of a test case to perform test case assessments against the current versions of the deployments/projects associated with the containing test suite. The Hub sends test cases to Determinations Server for assessment, and uses the responses received from Determinations Server to determine the outcome of the test case assessment.

The following table provides a list of the possible combinations of test case assessment state and outcomes for normal operation, and what the combinations represent:

Status	Outcome code	Description
pending	N/A	The test case has not been assessed yet
passed	ComparisonSuccess	Test case assessment was successful. The result state from Determinations Server was "OK". The output from Determinations Server matched the expected output from the test case.
failed	ComparisonFailure	Test case assessment failed as the output was not as expected. The result state from Determinations Server was "OK". The output from Determinations Server did not match the expected output from the test case.
failed	AssessedStateError	Test case assessment failed as Determinations Server could not perform the assessment. The result state from Determinations Server was "ERROR". The error details returned by Determinations Server are recorded with the test case assessment outcome.

The following table provides a list of additional combinations of test assessment state and outcome. These combinations indicate that an unexpected system error has prevented the test case assessment from being completed.

Status	Outcome code	Description
error	ConnectionError	Test assessment request failed due to a connection error when sending the request to Determinations Server. The error details are recorded with the test case assessment outcome.
error	ResponseError	Test assessment request failed as Determinations Server could not process the request. The error details returned by Determinations Server are recorded with the test assessment outcome.
error	ResponseInvalid	Test assessment request failed as the response was invalid. The response payload was not a valid JSON object.
error	ResponseCasesMissing	Test assessment request failed as the response was invalid. The response payload did not include the required "cases" property.
error	ResponseCasesInvalid	Test assessment request failed as the response was invalid. The "cases" property in the response payload was not a JSON array.
error	AssessedStateInvalid	The test assessment response was invalid. The result state value was not "OK" or "ERROR".
error	AssessedStateMissing	The test assessment response was invalid. The result state value was not provided.
error	AssessedOutputInvalid	The test assessment response was invalid. The output was not a valid JSON object.
error	AssessedOutputMissing	The test assessment response was invalid. The output was not provided.
error	AssessedResultInvalid	The test assessment response was invalid. The result was not a valid JSON object.
error	AssessedResultMissing	The test assessment response was invalid. The result was not provided.
error	SystemError	A system error was encountered when sending the test assessment to Determinations Server.

Test run lifecycle

Test runs have a well-defined lifecycle that is initiated by a client interacting with the REST API and is managed by the asynchronous Test Runner process. The following diagram describes the possible states of a test run, and the valid transitions between states:

Creating a test run

A client creates a new test run by submitting a valid POST request to the REST API:

• The new test run cannot be created if an active test run (status of "pending" or "running") already exists for the test suite

• The new test run is allocated the next run number for the test suite (N+1 where N is the latest run number for the test suite, or 1 if there are no test runs)

• The new test run is created with a status of "pending"

The status "pending" indicates that the test run has been created, but has not yet started processing.

Cancelling a test run

A client cancels an existing test run by submitting a valid PATCH request to the REST API:

• The test run must be currently active (status of "pending" or "running")

• The status of any "pending" test case assessments is changed to "cancelled"

• The test run status is changed to "cancelled"

A status of "cancelled" indicates that the test run was manually interrupted before processing was completed.

Starting a test run

The asynchronous Test Runner starts a test run when next invoked:

• Test case assessments with a status of "pending" are created for each combination of test case and the current version of all deployments/projects associated with the containing test run

• The test run status is changed to "running"

A status of "running" indicates that the test run has been started.

Processing a test run

For each test run started, the asynchronous Test Runner creates a Test Process to complete the test case assessments:

• Test case assessments for the test run are queued for processing

• Blocks of "pending" test case assessments are packaged and sent as test assessment requests to Determinations Server

• The assessed output provided by the response is recorded as the "actualOutput" property of the test case assessment resource

• The status of the assessment is determined by comparing the actual output of the test assess request to the expected output of the test case

The status remains "running" until all test case assessments have been processed.

Finishing a test run

Once all test case assessments have been assessed, the Test Process finishes the test run:

• The test run status is changed to "error" if:

  • The status of at least one contained test case assessment is "error", or

  • A system error occurred, interrupting the Test Process.

• The test run status is changed to "failed" if:

  • The Test Process did not encounter an error, and

  • The status of no contained test case assessments is "error", and

  • The status of at least one contained test case assessment is "failed".

• The test run status is changed to "passed" if:

  • The Test Process did not encounter an error, and

  • The status of no contained test case assessments is "error", and

  • The status of no contained test case assessments is "failed".

The status of "error", "failed", or "passed" indicates that the test run has been completed.

REST API resources

The Test Suites REST API has resources to represent Test Suites, Test Cases, Test Runs, Test Run Assessment Summaries and Test Run Assessment Results. The following diagram provides an overview of how these resources relate to each other, and how they relate to the existing Deployments REST API and Projects REST API resources: