Run continuous evaluations

Repeat one pinned Suite on a durable cadence without overlapping work or creating catch-up storms.

A Schedule is a versioned instruction to start the existing Suite workflow at a fixed interval. It does not create another execution engine: every scheduled run still uses the normal eval Agent, Resources, Policy checks, budget enforcement, verdicts, and accounting. The Schedule stores its own revision history and the exact Suite request used for each run.

Before you begin

Publish a Dataset, create the Scorers, and deploy the Agent you want to observe. Choose Replay for stable regression evidence, Fresh for live dependency coverage, or Shadow for paired candidate-versus-production evidence. A Schedule may point to the Agent's current bundle; each run resolves that bundle independently so a deployment can be detected by the next evaluation.

Steps

1. Open Evals → Schedules and choose Create Schedule.
2. Give the Schedule a stable ID and choose its cadence. Cadence may be five minutes through thirty days.
3. Select the immutable Dataset and Scorer versions, Agent, mode, concurrency, and per-run charge ceiling.
4. Under After a qualifying Suite, leave training off for monitoring only or choose Train a candidate. Set minimum completed cases, maximum errors, optional Scorer thresholds, SFT or RL, and a hard training budget.
5. Create it as Enabled to arm the next run or Paused to save the definition without running it.
6. Use the same Schedule ID with its current hash when changing the definition. That compare-and-set prevents one operator from silently overwriting another operator's revision.

The CLI accepts the same complete request as the Console:

constal evals schedules apply daily-support --body @schedule.json
constal evals schedules get daily-support

{
  "state": "enabled",
  "intervalMs": 86400000,
  "suite": {
    "dataset": { "crn": "DATASET_CRN", "version": "3", "hash": "DATASET_HASH" },
    "subject": { "agent": "AGENT_CRN", "bundle": "current" },
    "mode": "replay",
    "scorers": [{ "crn": "SCORER_CRN", "version": "2", "hash": "SCORER_HASH" }],
    "concurrency": 4,
    "budgetMicroUsd": 10000000,
    "onDivergence": "recorded"
  },
  "training": null,
  "expectedHash": null
}

Set training to a complete automation definition when this Schedule should close the improvement loop:

{
  "outputModelPrefix": "support-candidate",
  "gate": {
    "minimumCompletedCases": 20,
    "maximumErroredCases": 0,
    "scorer": { "crn": "SCORER_CRN", "version": "2", "hash": "SCORER_HASH" },
    "minimumPassRate": 0.9,
    "minimumMeanScore": null
  },
  "recipe": {
    "method": "rl",
    "baseModel": "openai/gpt-oss-20b",
    "dataset": { "crn": "DATASET_CRN", "version": "3", "hash": "DATASET_HASH" },
    "validationDataset": { "crn": "DATASET_CRN", "version": "3", "hash": "DATASET_HASH" },
    "scorers": [{ "crn": "SCORER_CRN", "version": "2", "hash": "SCORER_HASH" }],
    "loraRank": 32,
    "learningRate": 0.0001,
    "epochs": 1,
    "batchSize": 4,
    "groupSize": 4,
    "temperature": 0.7,
    "seed": 42,
    "checkpointEvery": 25,
    "evalEvery": 25,
    "maxTrainTokens": 1000000,
    "maxSampleTokens": 500000,
    "budgetMicroUsd": 50000000
  }
}

All Dataset and Scorer references are exact Resource pins. RL Scorers must be deterministic code Scorers. A successful Suite that clears the gate consumes the Schedule revision's one training opportunity. The platform derives a stable job ID and candidate Model ID from the Schedule revision, so an uncertain launch retries the same job rather than charging for a duplicate. A Suite that misses the gate is recorded as skipped; a later Suite in the same revision may still qualify.

Verify

The Schedule should show its next run, last run state, automatic training state, and current definition hash. A scheduled run has a deterministic Suite ID derived from the Schedule revision and due time, so retrying an uncertain launch cannot duplicate the Suite. If the preceding Suite is still active, the scheduler advances to the next cadence rather than overlapping it. Missed intervals are skipped rather than replayed as a burst.

Trigger one immediate run without changing the cadence when validating a new Schedule:

constal evals schedules run daily-support --event-id initial-check

The event ID is an idempotency key. A failed launch remains visible in lastRun; the scheduler does not turn an infrastructure failure into a successful-looking evaluation. Automatic training progress remains visible in lastTraining, including the qualifying Suite result, deterministic job and output IDs, attempts, terminal state, and reason for a skip or failure.

Next steps

Inspect completed Suite results, compare them with a reviewed baseline, and add newly discovered failures to the Dataset draft. A trained output is always a candidate: the scheduler never binds it to an Agent or promotes it automatically. Validate it with a fresh Suite before an explicit deployment decision. Pause a Schedule while changing dependencies. Delete only when its cadence is no longer needed; immutable revision and run records remain part of the audit trail.