Model Comparison and Optimization

Establish a reliable baseline before comparing models.

Compare a Known Candidate

Use identical fixtures and separate run labels. Use the following prompt with your coding assistant:

Run the file-replay suite for <workflow name> with the current model as baseline-current-model.

Run the same fixtures with <candidate model> as candidate-model. Complete required judging and compare the reports.

Review quality and contract compliance before accepting cost or latency improvements.

Evaluate Model Placement

Use a node-level sweep when the best placement hasn’t been identified. Use the following prompt with your coding assistant:

Propose a node-level model-optimization sweep for <workflow name>. Run fresh candidate profiles using file replay, complete required judging, and summarize the recommendations.

Use a workflow-wide sweep only for a deliberate global comparison. Evaluation uses runtime-only overrides. Apply a validated recommendation only after review and approval, then rerun the complete suite.

Analyze Token Usage

Analyze token consumption in a completed test report to identify opportunities to reduce usage while maintaining response quality and correct workflow routing. Use the following prompt with your coding assistant:

Analyze token usage for the completed <test name> report. Recommend prompt, grounding, or orchestration reductions that preserve quality and routing.