Evals with the CLI

Create Datasets and Scorers, operate suites and baselines, and automate metered SFT or RL jobs.

The CLI exposes task-oriented commands over the same Platform API used by the Console. JSON input may be inline, read from @FILE, or streamed from @-. Use files for pinned references and training recipes so code review captures every semantic input.

Dataset workflow

constal evals datasets create support-regression \
  --body '{"displayName":"Support regression","description":"Representative support behavior"}'

constal evals datasets add-case support-regression --body @case.json
constal evals datasets cases support-regression --limit 100
constal evals datasets publish support-regression --body '{}'
constal evals datasets list

Draft case writes are idempotent by caseId. Publishing creates an immutable version and keeps the draft available for the next version.

Scorer and Suite workflow

constal evals scorers create --body @scorer.json
constal evals scorers list

constal evals suites start --body @suite.json
constal evals suites get support-v17
constal evals suites list --limit 50

Cancel a job only when its remaining work is no longer useful:

constal evals suites cancel support-v17

Use Suite result hashes—not display IDs—for comparison and baseline operations:

constal evals compare --body @comparison.json
constal evals baseline --body @baseline.json

Continuous schedules

constal evals schedules apply daily-support --body @schedule.json
constal evals schedules get daily-support
constal evals schedules list
constal evals schedules run daily-support --event-id release-2026-08-23

Updating or deleting a Schedule uses its current definition hash as expectedHash. A manual run's event ID is caller-stable; reuse it after an uncertain response so the platform resolves the same Suite identity.

Training workflow

constal evals training providers
constal evals training models \
  --provider "$TRAINING_PROVIDER_CRN" \
  --hash "$TRAINING_PROVIDER_HASH"
constal evals training start --body @training.json
constal evals training get support-rl-v2
constal evals training list --limit 50

The start response is a durable job reference, not a promise that training has already completed. Poll get for authoritative phase, step, usage, cost, retry state, error, checkpoints, and final result. Cancellation is also idempotent:

constal evals training cancel support-rl-v2

A successful job exposes content-addressed checkpoints. Ask the API for exact compatible Gateway revisions, then submit a separate Model creation body:

constal evals training checkpoint-gateways support-rl-v2 \
  --step 400 \
  --ref "$CHECKPOINT_REF"

constal evals training create-model support-rl-v2 \
  --body @candidate-model.json

The Model import is durable and idempotent. It creates a normal Model Resource with training provenance; it does not update an Agent binding.

Automation rules

Choose caller-stable Suite and training IDs, commit request JSON, and retry transport failures with the same inputs. Never generate a new job ID merely because a request timed out. Set explicit concurrency, token ceilings, and budgetMicroUsd in every automated workflow. Read paged lists using the returned continuation cursor rather than assuming the most recent page is complete.

Run constal help evals training start or the complete CLI reference for global authentication, namespace, endpoint, output, and JSON options.