Work with ATLAS

Get Started

Use your coding assistant to generate, run, and evaluate ATLAS tests for your workflow.

• Describe the workflow, scenario, and expected result to your coding assistant:

  Generate and run ATLAS tests for <workflow name>. Cover the main success path, important branches, and empty-data cases. Use approved representative data, complete required local judging, and show the test files and final suite report.

• To inspect proposed coverage without creating or running tests, use the following prompt:

  Show me the test synchronization plan for <workflow name>. Do not change files or run tests

A preview displays planned work. An execution request asks the assistant to complete the applicable generation, data preparation, execution, judging, and reporting.

ATLAS Lifecycle

Use this lifecycle after creating or materially changing a workflow:

1. Inspect impact: Identify missing, outdated, or affected scenarios.
2. Create or update tests: Preserve valid custom coverage, metadata, and assertions.
3. Prepare replay data: Record representative data or prepare intentional generated data.
4. Validate behavior: Check routes, response structures, and deterministic assertions.
5. Complete semantic judging: Evaluate flexible responses when quality criteria apply.
6. Review reports: Resolve failures and required pending work.
7. Optimize separately: Compare models or evaluate consumption after establishing a reliable baseline.

Design Useful Coverage

Create tests for important behavior rather than one test for every node.

• Primary path: A valid request produces the expected business result.
• Conditional branch: Different inputs select the appropriate route.
• Empty or missing data: The workflow returns the defined no-results response.
• Invalid input or exception: The workflow follows its expected fallback or error path.
• Application stage: Startup, query, action, or communication requests satisfy their distinct contracts.
• Stateful interaction: A later conversation turn uses earlier information correctly.

Ask the coding assistant to suggest scenarios:

Suggest useful ATLAS scenarios for <workflow name>. Include primary paths, branch outcomes, and empty-data cases. Explain the input and expected result for each scenario.

Prepare Scenario Inputs

Provide meaningful inputs for the intended scenario.

• For Query scenarios, provide a useful user question.
• For REST scenarios, include required trigger parameters.
• For application scenarios, include the required stage and application context.

ATLAS prioritizes explicit values and preserves existing inputs validated against the current workflow. For new coverage, it can use matching run profiles, trigger metadata, and routing conditions.

The operation must confirm an inferred route. If a required value can’t be determined safely, ATLAS reports the missing information.

Name and Tag Tests

Use descriptive names that identify the scenario. Existing names and filenames are preserved; generated tests use stable scenario identity for synchronization.

• smoke: Critical paths that provide focused confidence
• regression: Expected behavior checked during routine development
• release-upgrade: Coverage required for release or upgrade validation
• integration: Current connected-service validation

Create and Maintain ATLAS Tests

Use the following prompts with your coding assistant to create an individual test:

Create a smoke-tagged ATLAS test for <workflow name> that verifies <scenario>.

Record approved representative data, run the test, complete required judging, and show the test and report locations.

After an intended workflow change, use the following prompt:

Update the existing <test name> test for <workflow name>. Preserve valid tags, custom metadata, and deliberate assertions. Run the affected test and show the results.

Generated tests represent scenarios maintained by synchronization. Custom tests represent user-owned coverage. Unmatched tests are reported for review rather than automatically deleted.

Don’t rerecord fixtures solely because a test fails. First determine whether the workflow changed intentionally, the test became outdated, or a defect needs correction.

Record and Manage Replay Data

Choose the data approach that matches the scenario.

• record-now: Approved representative live data is available for recording.
• record-later: The test can be defined now, but recording must happen later.
• model-generated: An empty state, fallback, or difficult-to-reproduce condition needs intentional synthetic data.
• baseline-replay: Another ready test uses the same data-producing path and differs primarily in trigger context.

Use the following prompt to record data:

Record approved representative replay data for <test name>. Report whether the retained data is raw or compacted.

Use the following prompt to prepare generated data:

Prepare clearly marked synthetic data for <test name>to exercise the empty-result path. Apply it, validate file replay, and show the report.

Fixtures retain information about their origin:

• raw: Complete recorded data retained within the available budget
• model-compacted: A smaller representative snapshot of real recorded data
• model-generated: Intentionally synthetic data

Compacted real data must remain distinguishable from generated data. Baseline reuse is appropriate only when the tests genuinely share the same data-producing path.

Recorded Data Masking

ATLAS can replace selected sensitive values in recorded test data. Masking is field-driven; it doesn’t assume a universal list of sensitive fields across products.

Define and Apply the Scope

Name the exact fields to mask, or ask the assistant to propose fields for an approved category. Review the proposed scope before applying it.

• Inspect: Identify field paths and types without displaying recorded values.
• Define: Establish explicit masking rules and exclusions.
• Preview: Show affected paths and replacement counts without modifying the test.
• Apply: Write masked data and validate deterministic file replay. Restore the original file if replay fails.

Consistent, type-compatible replacements preserve relationships across related test data, assertions, and judge criteria.

Inspect recordable fields for <workflow name>.Propose a reusable masking profile for <named sensitive fields>.Preview the affected paths before applying it.

Reuse and Maintain Policies

Create the masking policy before recording whenever sensitive values must not enter persistent recordings or compaction requests.

• Legacy: test/masking-profiles/<profile>.json
• App package: tests/ai/self/<module-name>/masking-profiles/<profile>.json

When a policy changes, update its reference and prepare fresh data. Don’t remask old replacement values. Refresh a complete conversation when needed to avoid mixing data prepared under different policies.

Raw preparation data is temporary. Ready source-controlled tests must contain only the final approved data and must not depend on temporary authoring files.