Run suites and compare results

Evaluate one pinned Agent across an immutable Dataset, then set baselines and detect regressions.

A Suite starts ordinary Agent Runs for Dataset cases, applies the selected Scorers, and reduces their verdicts into one content-addressed result. Fresh mode uses real calls against evaluation-bound Resources. Replay mode re-executes captured Runs under the candidate while their immutable journals answer recorded world effects. Shadow mode runs the candidate on a fork at each captured production Run's pre-run head, scores the production and candidate outputs with the same Scorer versions, and reports paired deltas. All three use the production runtime, Policy, and accounting paths.

Before you begin

Publish a Dataset, create at least one Scorer, and deploy the Agent you want to measure. Bind effectful capabilities to safe evaluation Resources or deny them with Policy. Pick a concurrency limit that the Agent's dependencies can sustain and a hard USD budget that covers the expected case volume.

Steps

1. Open Evals → Suites and choose Run Suite.
2. Select the exact Dataset, Agent, and Scorer.
3. Choose Fresh for a capability test, Replay for a journal-controlled regression test, or Shadow for a paired candidate-versus-production comparison over captured Runs.
4. For Replay, choose whether a changed Model request should reuse its recorded response or run live. Divergent non-Model effects are never performed: read-only effects reuse the recording and effectful operations fail closed.
5. Give the Suite a caller-stable ID. Repeating the same request with that ID is safe.
6. Set concurrency and the hard charge ceiling, then start the Suite. You can leave the page; the Suite continues durably.

Shadow requires every Dataset case to retain its captured source Run. The production side is never re-executed or charged again. Only the candidate runs on the fork; scoring cost for both outputs and candidate execution cost count toward the Suite budget.

The equivalent CLI request is explicit about every pinned dependency:

constal evals suites start --body @suite.json
constal evals suites get support-v17

{
  "suiteId": "support-v17",
  "dataset": { "crn": "DATASET_CRN", "version": "3", "hash": "DATASET_HASH" },
  "subject": { "agent": "AGENT_CRN", "bundle": "current" },
  "mode": "fresh",
  "scorers": [{ "crn": "SCORER_CRN", "version": "2", "hash": "SCORER_HASH" }],
  "concurrency": 4,
  "budgetMicroUsd": 10000000
}

To measure a Judge against human labels, include that Judge in scorers and add calibration. Every Dataset case must store expected.score; expected.pass is optional. The Judge scores the Agent output while the Suite derives agreement from the independent human label—it never asks the Judge to grade its own verdict.

"calibration": {
  "judge": { "crn": "JUDGE_SCORER_CRN", "version": "4", "hash": "JUDGE_SCORER_HASH" },
  "tolerance": 0.1
}

Verify

Wait for succeeded, then inspect total, completed, errored, per-Scorer pass rate and score distribution, settled cost, calibration metrics when requested, and the verdict manifest reference. A high error count is not a low score; fix execution failures before treating the aggregate as evidence.

Compare a successful Suite result with a baseline using their result hashes:

constal evals compare --body '{"suite":"SUITE_RESULT_HASH","baseline":"BASELINE_RESULT_HASH"}'
constal evals baseline --body '{"agent":"AGENT_CRN","suite":"SUITE_RESULT_HASH"}'

Comparison aligns cases by caseId, reports per-Scorer deltas, and marks statistically significant changes when the available paired data supports that conclusion.

Replay results also include a divergence manifest. Each entry identifies the journal position, old and candidate request hashes, a bounded structural diff, the effect class, and whether the harness used a recorded response, made a live Model call, or refused an unsafe operation. Treat a replay-unsafe error as a real regression in the candidate's effect behavior.

Shadow results contain a content-addressed production-side baseline in compare.baseline. Candidate and production verdicts align by caseId and Scorer CRN, so each delta is computed from pairs rather than unrelated cohort averages. A Shadow Suite rejects mixed production identities: every source case must name the same Agent bundle, Policy, Resource resolution, and Model policy.

For a production canary, configure the eval door to sample constal.rollout.cohort=candidate and constal.rollout.cohort=control under the same pinned Scorers. Only new, selector-eligible Sessions enter the split, and assignment is stable for the Session. Use paired Shadow evidence when case-level significance is required; cohort samples remain useful for production-only reliability and cost signals. Promotion and rollback are explicit Agent rollout controls rather than a second execution path.

Next steps

Set a reviewed result as the Agent baseline. Agent deployment policies use Replay suites so candidate code is evaluated before promotion without repeating recorded side effects. If failures expose missing examples, add them to the Dataset draft and publish a new version. If the behavior is learnable from examples or rewards, continue to SFT or RL training.