# Create and calibrate Scorers

> Define reusable code, Judge, or human judgments for suites, gates, validation, and reinforcement-learning rewards.

A Scorer turns an Agent artifact into a score, pass decision, and bounded evidence. Scorer identity is part of every verdict, so changing judgment behavior requires a new version. Code Scorers are validated deterministic rule trees. Judge Scorers delegate semantic judgment to one immutable Agent version and rubric. Human Scorers route a structured labeling task to a Channel.

## Before you begin {#before-you-begin}

Decide what one signal means. Prefer several narrow Scorers—schema validity, correct intent, required citations, safe tool order—over one opaque “quality” rule. Suite subjects have three roots: `/case` for the Dataset row, `/output` for the Agent result, and `/run` for Run identity.

## Steps {#steps}

1. Open **Evals → Scorers** and choose **Create Scorer**.
2. Name one measurable signal and select what it applies to. Choose **Case** for offline suites and training.
3. Choose **Code** for deterministic rules, **Judge** for semantic judgment by an Agent, or **Human** for structured labels delivered through a Channel.
4. For a Judge, pin the Judge Agent and write a narrow rubric with an explicit numeric scale and passing threshold.
5. Create version `1`. When a rule, rubric, Agent, scale, or label schema changes, keep the ID and publish a new version.

This rule passes when the Agent returns the expected intent:

```json
{
  "v": 1,
  "rule": {
    "op": "equals",
    "path": "/output/intent",
    "value": "order_status"
  }
}
```

An output-shape rule can use JSON Schema:

```json
{
  "v": 1,
  "rule": {
    "op": "schema",
    "path": "/output",
    "schema": {
      "type": "object",
      "required": ["answer", "citations"],
      "properties": {
        "answer": { "type": "string" },
        "citations": { "type": "array" }
      }
    }
  }
}
```

## Verify {#verify}

Validate the same rule locally with the SDK before publishing it:

```ts
import { codeScorerIR, evaluateCodeScorer } from "@constal/sdk";

const scorer = codeScorerIR({
  v: 1,
  rule: { op: "range", path: "/output/confidence", min: 0.8, max: 1 },
});

const verdict = evaluateCodeScorer(scorer, {
  case: { input: "..." }, output: { confidence: 0.93 }, run: { id: "run-1" },
});
if (!verdict.pass) throw new Error(JSON.stringify(verdict.evidence));
```

The Console only lists the Scorer after the rule is accepted and its immutable rule artifact is stored. A malformed pointer, regular expression, schema, or unknown operation is rejected before Resource creation.

Calibrate a Judge before using it as a gate or reward. Put human labels in every calibration case as `expected: { "score": 0.9, "pass": true }`, run the exact Judge Scorer as a Suite Scorer, and enable Judge calibration with an absolute tolerance. The result records exact agreement, within-tolerance agreement, mean absolute error, correlation, and Cohen's kappa when boolean labels are available. Calibration is bound to the exact Judge Scorer version and fails closed if any case lacks a numeric human label or Judge verdict.

## Next steps {#next-steps}

Use the Scorer in a [Suite](/docs/evals/suites.md). Once a signal has enough coverage and few false positives, use its exact version in a deployment gate or as an [RL reward](/docs/evals/training.md). Keep the same version pinned when comparing candidates.