1. How do I use AI Agent Studio?
  2. Economic Regression Testing

Economic Regression Testing

Economic regression testing evaluates projected consumption against an approved allocation.

  • AI-unit budget assertion: The assertion covers one complete test run. 
  • Economic Profile: The profile includes representative outcomes, production volume, allocation, and accepted baselines.

The runtime economic control plane is the allocation authority. User-defined or model-recommended allocations are provisional alternatives requiring review. ATLAS doesn’t modify control-plane allocations automatically.

Establish an Economic Profile

Establish an economic profile to define representative outcomes, validate expected consumption, and create approved baselines for economic evaluation. Follow these steps:
  1. Complete representative functional coverage.
  2. Verify allocation authority and warning thresholds.
  3. Prepare a profile proposal.
  4. Review selected outcomes, expected volume, evidence, and projected consumption.
  5. Explicitly approve the proposal.
  6. Establish separate accepted file-replay and live baselines.

Interpret File and Live Measurements

Interpret economic measurements based on the selected data source to understand how consumption is calculated against the approved allocation. Use the following guidelines:
  • Live: Current consumption per representative outcome multiplied by expected volume
  • File replay: Replay measurements translated using accepted outcome-level live-to-replay relationships

Both evaluate against the same approved allocation.

Application profiles use representative backing-workflow tests and the number of invocations per application interaction. Panel tests are excluded from economic calculations.

Warnings and Enforcement

Profiles support up to three utilization warning thresholds. If no usable threshold is available, the default warning is 80%; reaching 100% is over budget.

  • warn: Reports economic health without changing the test command’s exit status
  • fail: Can fail economic evaluation and block CI under the applicable release's command policy

After changing workflows, representative outcomes, or production volume, synchronize tests and collect fresh evidence. Review the impact before explicitly accepting a new baseline.