Evals

Measure Agent behavior with versioned cases and Scorers, prevent regressions, and turn proven data into trained Models.

Evals use the same Agents, Runs, Resources, Policies, budgets, and durable facts as production. There is no separate execution model to approximate what an Agent would do. A suite executes one pinned Agent revision in fresh, journal-backed replay, or paired shadow mode, records a verdict for every case, and produces a content-addressed result that can become a baseline or deployment gate.

The improvement loop

text

real interactions → Dataset version → Scorer → Suite
                                      ↓        ↓
                                online gate   baseline
                                      ↓        ↓
                                  trusted data → SFT or RL → candidate Model

A Dataset is an immutable version of representative inputs, expected outputs, context, tags, and provenance.
A Scorer is a versioned judgment. The same Scorer can judge a suite, guard a commit, or provide an RL reward.
A Suite evaluates one pinned Agent against every Dataset case and settles its actual cost.
A Schedule repeats one Suite definition on a durable cadence without overlapping runs.
A baseline is a successful Suite result selected for future comparisons or deployment gates.
A training job performs supervised fine-tuning or reinforcement learning from pinned Dataset and Scorer versions.

Choose a path

Goal	Guide
Build a representative case set	Create and publish Datasets
Define a deterministic quality signal	Create Scorers
Test a deployed Agent	Run suites and compare results
Continuously detect regressions	Run continuous evaluations
Produce adapted Model weights	Train with SFT or RL
Connect another training system	Build a Training Provider
Automate the workflow	Evals with the CLI
Validate recipes in code	Evals SDK types and validators
Embed evaluation in another product	Evals Platform API

The rule that keeps results honest

Every result names exactly what it measured: the Agent bundle, Policy, Resource resolutions, Dataset version and hash, Scorer versions and hashes, model policy, and input. Changing any of those inputs creates a different evaluation identity. Publishing a Dataset or Scorer version never mutates an older result.

Use fresh suites for end-to-end capability questions. They make real Model and Resource calls, so bind safe evaluation Resources and set a hard budget. Use replay suites for controlled regression questions: the source journal answers exact world effects without repeating them. Use shadow suites when you want a paired answer to “would this candidate have been better on the production cases?” The candidate runs on source forks while the already-recorded production outputs remain untouched. Use deterministic Scorers for cheap, explainable checks; promote a Suite to baseline only after reviewing its failures, divergences, and error rate.

Start in the Console

Open Evals. Create a Dataset, create a Scorer, and run a Suite. Add a Schedule when the same evidence should be refreshed continuously. Once the Dataset represents the behavior you want, open Training, choose an installed Training Provider, and start SFT or RL. A successful job produces checkpoints; importing one through a compatible Gateway creates a candidate Model without changing an Agent binding. Every long-running operation has a caller-stable ID, can be retried safely, and remains inspectable after the browser closes.