# Run suites and compare results

> Evaluate one pinned Agent across an immutable Dataset, then set baselines and detect regressions.

A Suite starts ordinary Agent Runs for Dataset cases, applies the selected Scorers, and reduces their verdicts into one content-addressed result. **Fresh** mode uses real calls against evaluation-bound Resources. **Replay** mode re-executes captured Runs under the candidate while their immutable journals answer recorded world effects. **Shadow** mode runs the candidate on a fork at each captured production Run's pre-run head, scores the production and candidate outputs with the same Scorer versions, and reports paired deltas. All three use the production runtime, Policy, and accounting paths.

## Before you begin {#before-you-begin}

Publish a [Dataset](/docs/evals/datasets.md), create at least one [Scorer](/docs/evals/scorers.md), and deploy the Agent you want to measure. Bind effectful capabilities to safe evaluation Resources or deny them with Policy. Pick a concurrency limit that the Agent's dependencies can sustain and a hard USD budget that covers the expected case volume.

## Steps {#steps}

1. Open **Evals → Suites** and choose **Run Suite**.
2. Select the exact Dataset, Agent, and Scorer.
3. Choose **Fresh** for a capability test, **Replay** for a journal-controlled regression test, or **Shadow** for a paired candidate-versus-production comparison over captured Runs.
4. For Replay, choose whether a changed Model request should reuse its recorded response or run live. Divergent non-Model effects are never performed: read-only effects reuse the recording and effectful operations fail closed.
5. Give the Suite a caller-stable ID. Repeating the same request with that ID is safe.
6. Set concurrency and the hard charge ceiling, then start the Suite. You can leave the page; the Suite continues durably.

Shadow requires every Dataset case to retain its captured source Run. The production side is never re-executed or charged again. Only the candidate runs on the fork; scoring cost for both outputs and candidate execution cost count toward the Suite budget.

The equivalent CLI request is explicit about every pinned dependency:

```sh
constal evals suites start --body @suite.json
constal evals suites get support-v17
```

```json suite.json
{
  "suiteId": "support-v17",
  "dataset": { "crn": "DATASET_CRN", "version": "3", "hash": "DATASET_HASH" },
  "subject": { "agent": "AGENT_CRN", "bundle": "current" },
  "mode": "fresh",
  "scorers": [{ "crn": "SCORER_CRN", "version": "2", "hash": "SCORER_HASH" }],
  "concurrency": 4,
  "budgetMicroUsd": 10000000
}
```

To measure a Judge against human labels, include that Judge in `scorers` and add `calibration`. Every Dataset case must store `expected.score`; `expected.pass` is optional. The Judge scores the Agent output while the Suite derives agreement from the independent human label—it never asks the Judge to grade its own verdict.

```json
"calibration": {
  "judge": { "crn": "JUDGE_SCORER_CRN", "version": "4", "hash": "JUDGE_SCORER_HASH" },
  "tolerance": 0.1
}
```

## Verify {#verify}

Wait for `succeeded`, then inspect total, completed, errored, per-Scorer pass rate and score distribution, settled cost, calibration metrics when requested, and the verdict manifest reference. A high error count is not a low score; fix execution failures before treating the aggregate as evidence.

Compare a successful Suite result with a baseline using their result hashes:

```sh
constal evals compare --body '{"suite":"SUITE_RESULT_HASH","baseline":"BASELINE_RESULT_HASH"}'
constal evals baseline --body '{"agent":"AGENT_CRN","suite":"SUITE_RESULT_HASH"}'
```

Comparison aligns cases by `caseId`, reports per-Scorer deltas, and marks statistically significant changes when the available paired data supports that conclusion.

Replay results also include a divergence manifest. Each entry identifies the journal position, old and candidate request hashes, a bounded structural diff, the effect class, and whether the harness used a recorded response, made a live Model call, or refused an unsafe operation. Treat a replay-unsafe error as a real regression in the candidate's effect behavior.

Shadow results contain a content-addressed production-side baseline in `compare.baseline`. Candidate and production verdicts align by `caseId` and Scorer CRN, so each delta is computed from pairs rather than unrelated cohort averages. A Shadow Suite rejects mixed production identities: every source case must name the same Agent bundle, Policy, Resource resolution, and Model policy.

For a production canary, configure the eval door to sample `constal.rollout.cohort=candidate` and `constal.rollout.cohort=control` under the same pinned Scorers. Only new, selector-eligible Sessions enter the split, and assignment is stable for the Session. Use paired Shadow evidence when case-level significance is required; cohort samples remain useful for production-only reliability and cost signals. Promotion and rollback are explicit Agent rollout controls rather than a second execution path.

## Next steps {#next-steps}

Set a reviewed result as the Agent baseline. Agent deployment policies use Replay suites so candidate code is evaluated before promotion without repeating recorded side effects. If failures expose missing examples, add them to the Dataset draft and publish a new version. If the behavior is learnable from examples or rewards, continue to [SFT or RL training](/docs/evals/training.md).