Assertions and Semantic Judging in ATLAS

Choose the right evidence based on the type of response or behavior you want to verify.

• Fixed fallback message, fixed JSON, or protocol response: Exact-output and path assertions
• Narrative answer, summary, or recommendation: Path assertions and a semantic rubric
• Application response: Required structure and fields, route checks, and semantic judging where appropriate
• Important intermediate value: A node input or output assertion
• Required number of operations: A run-count assertion
• Per-run consumption limit: An optional AI-unit budget assertion

Path and Exact-Output Assertions

Path assertions identify nodes that must run and nodes that must not run. The following partial example checks an empty-data response and its route:

{
  "assertions": {
    "equals": "No team performance ratings found for the selected period."
  },
  "pathAssertions": {
    "mustExecute": [
      "GET_TEAM_PERFORMANCE_RATINGS",
      "NO_RATINGS_RETURN"
    ],
    "mustNotExecute": [
      "BUILD_TEAM_PERFORMANCE_DASHBOARD"
    ]
  }
}

Use exact checks for fixed responses. Avoid exact string comparisons for model-generated answers whose wording can vary. Assertion suggestions are diagnostic guidance. They don’t authorize replacing deliberate assertions.

Semantic Judging

A semantic rubric defines observable quality criteria. For example, tests can require that a response meet the following criteria:

• Uses the supplied data
• Answers the user’s request
• Includes required records or explanations
• Avoids unsupported facts
• Meets the required response contract

A result of needs-judge means required evaluation remains incomplete. Use the following prompts with your coding assistant:

Attach all available local judge results for <workflow name>.

Refresh the reports and show remaining failed or needs-judge cases.

Attaching an existing judge result updates reports without rerunning the workflow. A suite is complete only when required judging is complete and unresolved failures are addressed.

Intermediate-Node Assertions

Use node assertions when final-output and path checks don’t establish an important intermediate contract. Examples include a Code node returning required fields or a Business Object response containing an expected item.

Node assertions use JSON Pointer paths into the selected input or output. /result is appropriate only when the actual data contains a result property. Describe the requirement in business terms and review generated field paths when precision matters.

Run-Count Assertions

Use counts when the number of operations is part of the expected behavior.

• nodeCounts: Counts observed operations of a workflow node
• referenceNodeCounts: Counts child-node runs for a particular reference caller
• exact: Requires one specified count
• minimum / maximum: Defines lower and upper bounds

Counts reflect observed runtime operations rather than fixture counts. Conversation tests can check individual steps and the complete conversation.

AI-Unit Budget Assertions

An optional budget assertion checks the consumption for one complete test run.

{
  "aiUnitAssertions": {
    "id": "workflow-ai-unit-budget",
    "name": "Workflow stays within the AI-unit budget",
    "maximumPerExecution": 20,
    "enforcement": "fail",
    "onMissingEvidence": "warn"
  }
}

• maximumPerExecution: Maximum AI units for the complete run
• enforcement: Fail the test, warn, or report an overage
• onMissingEvidence: Fail, warn, or skip when metering evidence is unavailable

A conversation uses one root-level budget for aggregate consumption across its steps. Budgets are opt-in. Ordinary generation, updates, and synchronization preserve existing budgets without automatically adding or changing them.

During an interactive local-judge run, ATLAS can explain an exceeded budget and recommend reductions. This analysis is advisory; it doesn’t modify the workflow or rerun operation or judging. It isn’t generated for remote judging.