# Evals and training API

> Create versioned evaluation inputs, operate durable suites, compare baselines, and start metered SFT or RL jobs.

All Evals routes are namespace-scoped and use normal Platform authentication, authorization, request IDs, and JSON error responses. Dataset and Scorer references are immutable triples of `crn`, `version`, and `hash`. Suite and training IDs are caller-stable idempotency identities.

## Datasets and Scorers {#datasets-and-scorers}

```text
POST   /v1/namespaces/{namespace}/evals/datasets/{dataset}/draft
GET    /v1/namespaces/{namespace}/evals/datasets/{dataset}/draft
GET    /v1/namespaces/{namespace}/evals/datasets/{dataset}/cases
POST   /v1/namespaces/{namespace}/evals/datasets/{dataset}/cases
DELETE /v1/namespaces/{namespace}/evals/datasets/{dataset}/cases/{case}
POST   /v1/namespaces/{namespace}/evals/datasets/{dataset}/capture
POST   /v1/namespaces/{namespace}/evals/datasets/{dataset}/publish
POST   /v1/namespaces/{namespace}/evals/scorers
```

Published Datasets and Scorers are ordinary Resources and are read through the namespace Resource routes. Draft cases support bounded keyset pagination with `after` and `limit`.

## Suites, comparison, and baselines {#suites}

```text
GET  /v1/namespaces/{namespace}/evals/suites?after={cursor}&limit={count}
POST /v1/namespaces/{namespace}/evals/suites
GET  /v1/namespaces/{namespace}/evals/suites/{suite}
POST /v1/namespaces/{namespace}/evals/suites/{suite}/cancel
POST /v1/namespaces/{namespace}/evals/compare
POST /v1/namespaces/{namespace}/evals/baseline
```

Starting a Suite returns `202` with its durable identity unless it has already completed. Read the job route for authoritative progress and result. Comparison and baseline bodies use content-addressed Suite result hashes, not the mutable job row.

The start body sets `mode` to `fresh`, `replay`, or `shadow`. Replay and Shadow cases must have been captured from completed Runs. `onDivergence` may be `recorded` (default) or `live`; it applies only to Replay and the live option applies only to changed Model requests. A Replay result includes `divergencesRef`, an immutable manifest of request hashes, bounded diffs, effect classes, and resolutions. Divergent effectful non-Model operations fail closed.

A Shadow Suite validates each production source, forks the candidate at the captured pre-run head, and scores both outputs with the same pinned Scorers. Its `compare` value points to the immutable production-side Suite artifact and contains paired per-Scorer deltas and significance flags. Mixed production subject identities are rejected rather than averaged together.

When `calibration` names one of the pinned Judge Scorers, every Dataset case must have a human `expected.score` and may include `expected.pass`. The Suite result includes exact and within-tolerance agreement, mean absolute error, correlation, and Cohen's kappa. Missing labels or Judge verdicts fail the Suite rather than silently shrinking the sample.

## Continuous schedules {#schedules}

```text
GET    /v1/namespaces/{namespace}/evals/schedules
GET    /v1/namespaces/{namespace}/evals/schedules/{schedule}
PUT    /v1/namespaces/{namespace}/evals/schedules/{schedule}
DELETE /v1/namespaces/{namespace}/evals/schedules/{schedule}
POST   /v1/namespaces/{namespace}/evals/schedules/{schedule}/run
```

The PUT body contains `state`, `intervalMs`, a Suite request without `suiteId`, `training`, and `expectedHash`. Set `training` to `null` for evaluation only. Otherwise it contains a complete pinned training recipe, an output Model prefix, and a gate with minimum completed cases, maximum errors, and optional aggregate Scorer thresholds. Use a null `expectedHash` only to create a new Schedule; use the returned hash for later compare-and-set changes. DELETE accepts `expectedHash` in JSON or `If-Match`. The run endpoint accepts `eventId`, also available through `Idempotency-Key`, and starts one deterministic immediate Suite without shifting the cadence.

A qualifying successful Suite starts at most one training job per exact Schedule revision. `lastTraining` exposes the qualifying Suite fact, derived job and output IDs, Agent run, attempts, state, and skip or failure reason. Launch retries keep the same deterministic identity. Training uses the normal eval Agent, training Resource, Policy authorization, budget reservation, measured usage settlement, and billing path. A successful job produces candidate weights only; the API never promotes or binds them implicitly.

## Training {#training}

```text
GET  /v1/namespaces/{namespace}/evals/training-models
GET  /v1/namespaces/{namespace}/evals/training-jobs?after={cursor}&limit={count}
POST /v1/namespaces/{namespace}/evals/training-jobs
GET  /v1/namespaces/{namespace}/evals/training-jobs/{job}
POST /v1/namespaces/{namespace}/evals/training-jobs/{job}/cancel
```

The model catalog includes the exact provider model identifier, context window, and pinned per-million-token prices used for settlement. A training status includes phase, method, base and output Model IDs, step and epoch progress, exact usage units, settled micro-USD cost, retry state, terminal error, timestamps, and final fact when available.

## Request discipline {#request-discipline}

Send the same job ID and body when retrying an uncertain start request. Do not create a second ID after a timeout. Treat `queued`, `running`, and `evaluating` as non-terminal. Bound list requests and follow the returned continuation cursor. Keep the API key outside request bodies and logs; every referenced Resource still passes the caller's Policy checks.