Create and calibrate Scorers

Define reusable code, Judge, or human judgments for suites, gates, validation, and reinforcement-learning rewards.

A Scorer turns an Agent artifact into a score, pass decision, and bounded evidence. Scorer identity is part of every verdict, so changing judgment behavior requires a new version. Code Scorers are validated deterministic rule trees. Judge Scorers delegate semantic judgment to one immutable Agent version and rubric. Human Scorers route a structured labeling task to a Channel.

Before you begin

Decide what one signal means. Prefer several narrow Scorers—schema validity, correct intent, required citations, safe tool order—over one opaque “quality” rule. Suite subjects have three roots: /case for the Dataset row, /output for the Agent result, and /run for Run identity.

Steps

1. Open Evals → Scorers and choose Create Scorer.
2. Name one measurable signal and select what it applies to. Choose Case for offline suites and training.
3. Choose Code for deterministic rules, Judge for semantic judgment by an Agent, or Human for structured labels delivered through a Channel.
4. For a Judge, pin the Judge Agent and write a narrow rubric with an explicit numeric scale and passing threshold.
5. Create version 1. When a rule, rubric, Agent, scale, or label schema changes, keep the ID and publish a new version.

This rule passes when the Agent returns the expected intent:

{
  "v": 1,
  "rule": {
    "op": "equals",
    "path": "/output/intent",
    "value": "order_status"
  }
}

An output-shape rule can use JSON Schema:

{
  "v": 1,
  "rule": {
    "op": "schema",
    "path": "/output",
    "schema": {
      "type": "object",
      "required": ["answer", "citations"],
      "properties": {
        "answer": { "type": "string" },
        "citations": { "type": "array" }
      }
    }
  }
}

Verify

Validate the same rule locally with the SDK before publishing it:

import { codeScorerIR, evaluateCodeScorer } from "@constal/sdk";

const scorer = codeScorerIR({
  v: 1,
  rule: { op: "range", path: "/output/confidence", min: 0.8, max: 1 },
});

const verdict = evaluateCodeScorer(scorer, {
  case: { input: "..." }, output: { confidence: 0.93 }, run: { id: "run-1" },
});
if (!verdict.pass) throw new Error(JSON.stringify(verdict.evidence));

The Console only lists the Scorer after the rule is accepted and its immutable rule artifact is stored. A malformed pointer, regular expression, schema, or unknown operation is rejected before Resource creation.

Calibrate a Judge before using it as a gate or reward. Put human labels in every calibration case as expected: { "score": 0.9, "pass": true }, run the exact Judge Scorer as a Suite Scorer, and enable Judge calibration with an absolute tolerance. The result records exact agreement, within-tolerance agreement, mean absolute error, correlation, and Cohen's kappa when boolean labels are available. Calibration is bound to the exact Judge Scorer version and fails closed if any case lacks a numeric human label or Judge verdict.

Next steps

Use the Scorer in a Suite. Once a signal has enough coverage and few false positives, use its exact version in a deployment gate or as an RL reward. Keep the same version pinned when comparing candidates.