Evals SDK types and validators

Validate Scorers, Suite inputs, and training recipes in code before sending them through the Platform API.

The SDK exports the canonical Evals types and pure validators used by the runtime. Use them at build time to reject malformed rules and recipes close to their source. Use the Platform API or CLI to create Resources and start durable jobs.

Validate a deterministic Scorer

scorer.ts

import { codeScorerIR, evaluateCodeScorer } from "@constal/sdk";

export const grounded = codeScorerIR({
  v: 1,
  rule: {
    op: "citations-in-source",
    textPath: "/output/answer",
    sourcesPath: "/output/sources",
  },
});

export function testGrounded(output: unknown) {
  return evaluateCodeScorer(grounded, {
    case: { caseId: "local", input: {} },
    output,
    run: { id: "local", session: "local" },
  });
}

evaluateCodeScorer returns score, pass, and structured failure evidence. It never performs I/O or Model calls.

Validate a training recipe

recipe.ts

import {
  startTrainingRequest,
  trainingRecipe,
  type DatasetRef,
  type ScorerRef,
  type TrainingProviderPin,
} from "@constal/sdk";

export function rlRecipe(dataset: DatasetRef, reward: ScorerRef) {
  return trainingRecipe({
    method: "rl",
    baseModel: "openai/gpt-oss-20b",
    dataset,
    validationDataset: dataset,
    scorers: [reward],
    loraRank: 32,
    learningRate: 0.00005,
    epochs: 1,
    batchSize: 4,
    groupSize: 4,
    temperature: 0.7,
    seed: 42,
    checkpointEvery: 25,
    evalEvery: 25,
    maxTrainTokens: 1_000_000,
    maxSampleTokens: 500_000,
    budgetMicroUsd: 50_000_000,
  });
}

export function sftRequest(provider: TrainingProviderPin, dataset: DatasetRef) {
  return startTrainingRequest({
    jobId: "support-sft-v1",
    provider,
    outputModelId: "support-model-v1",
    recipe: trainingRecipe({
      method: "sft",
      baseModel: "acme/base-20b",
      dataset,
      validationDataset: dataset,
      scorers: [],
      loraRank: 32,
      learningRate: 0.0001,
      epochs: 1,
      batchSize: 4,
      groupSize: 1,
      temperature: 0,
      seed: 42,
      checkpointEvery: 25,
      evalEvery: 25,
      maxTrainTokens: 1_000_000,
      maxSampleTokens: 500_000,
      budgetMicroUsd: 50_000_000,
    }),
  });
}

The recipe validator accepts a provider model identifier, exact Dataset and Scorer references, bounded numeric settings, and method-compatible values. In particular, RL needs a Scorer and SFT requires groupSize: 1. Admission resolves the selected provider, verifies that its exact revision advertises the model and method, and inserts or validates the same provider pin before invocation.

Declare a deployment suite gate

gate.ts

import type { AgentDeployPolicy, DatasetRef, ScorerRef } from "@constal/sdk";

export const deployEvals = (
  dataset: DatasetRef,
  scorer: ScorerRef,
  baseline: string,
): AgentDeployPolicy => ({
  suites: [{
    id: "support-regression",
    dataset,
    scorers: [scorer],
    mode: "replay",
    concurrency: 4,
    budgetMicroUsd: 10_000_000,
    minPassRate: 0.95,
    baseline,
    maxRegression: 0.02,
  }],
});

The deployer resolves every reference, replays the captured Runs with the candidate immutable Agent identity, and refuses promotion when the gate fails. Deployment gates deliberately require replay mode so recorded side effects are not repeated before promotion. Keep references in generated deployment configuration rather than copying mutable display names into Agent code.