Common workflows

Create a new eval case

Goal: create the required directory structure and a case file in eval/ from an existing recording. Commands:

mkdir -p eval/cases
cat > eval/cases/task-02.yaml <<'YAML'
video: ../../../recordings/task-02.mov
description: Replace this note with what task-02 should cover.
sampling:
  every_s: 0.5
targets:
  step_2:
    samples:
    - range: [0.0, 6.0]
      expect: false
    - range: [6.0, 12.0]
      expect: true
YAML

Expected result: eval/cases/task-02.yaml points to task-02.mov in the sibling recordings/ directory and labels the expected value before and after the six-second mark. Note: If you prefer colocated fixtures, put the recording in eval/cases/ and set video: to the filename, such as task-02.mov.

Seed draft expectations

Hand-labeling is the clearest way to start. When a case has many samples, you can instead omit expect and ask an adapter to propose the missing values:

uv run --env-file .env glasskit eval seed --case task-02
uv run glasskit eval review --case task-02

seed uses the same adapter as run by default, fills only omitted expectations, and preserves existing labels. Use --adapter or --adapter-command to label with a different adapter, and use --case or --target to narrow the work. Treat generated values as proposals and review them before relying on the eval.

Review and correct expectations in the browser

Goal: make recorded-video expectations faster to verify and correct. Without the review UI, verifying expectations means juggling a media player and YAML editor while matching timestamps by hand. The review UI puts the video, expanded sample schedule, and editing controls in one place, making corrections faster and less error-prone. Command:

uv run glasskit eval review --eval-dir eval

To jump directly to a failure reported by a separate eval run, include its case, target, and timestamp:

uv run glasskit eval review --eval-dir eval --case task-01 --target step_1 --time 7.4

The command opens a local browser UI where you can compare labeled moments with their source video and add, move, edit, or delete samples. Changes are saved automatically to the case file.

Run one case while debugging

Goal: run a focused eval and print every sample result. Command:

uv run glasskit eval run --case task-01 --target step_1 --verbose --keep-going --save-failures --output-json eval/runs/results.json --artifacts-dir eval/runs/artifacts

Expected output: focused case and target progress, every selected sample result, a final summary, and a per-target table. Note: --keep-going records adapter evaluation errors and comparison errors as sample results instead of aborting on the first sample error. Every completed adapter result is also checkpointed, so when there is reusable progress the printed glasskit eval run --resume ... command can retry adapter errors without rerunning completed samples. --save-failures writes JPEG frames and per-result JSON for failed or errored samples. Treat eval/runs/ as disposable output and add it to your app repo’s .gitignore if you keep generated eval reports out of source control.

Measure nondeterministic stability

Goal: run the same selected eval three times and identify samples whose outcomes vary. With --repeat N, GlassKit Eval executes the same selected sample schedule N times. Each complete repetition is called a trial, and each evaluation of a sample within a trial is an attempt. Command:

uv run --env-file .env glasskit eval run --concurrency 2 --repeat 3 --max-flaky-samples 0 --output-json eval/runs/repeated-results.json

Expected output: three sequential trial progress sections, a per-trial quality table, minimum/mean/maximum trial pass rates, per-target stability, and a table of flaky or consistently failing samples. The command constructs and closes a fresh evaluator for every trial. --concurrency 2 still permits at most two individual evaluations in flight because trials themselves are never run concurrently. Every trial uses the same filters and selected schedule. Quality gates such as --min-pass-rate apply independently to each trial, and the run fails if any trial fails one; results are never pooled before applying a quality gate. --max-flaky-samples 0 checks only whether sample statuses vary across trials, so combine it with a correctness gate such as --min-pass-rate 0.9 when both stability and quality should affect the exit code. Repeating an eval multiplies its adapter work and provider cost by the repeat count.

Enforce CI quality gates

Goal: make the command fail when quality drops below your threshold. Command:

uv run glasskit eval run --min-pass-rate 0.9 --min-target-pass-rate 0.85 --max-failures 3 --output-json eval/runs/results.json

Expected behavior: the process exits 0 when every configured quality gate passes, 1 when the eval completed but one or more gates failed, and 2 for setup or runtime errors that abort the run. Note: Threshold defaults are intentionally unset. Without --min-pass-rate, --min-target-pass-rate, --max-failures, or YAML thresholds, failed comparisons are visible in the report but do not fail the command. Always configure a gate for CI.

Use an adapter config file

Goal: pass runtime settings to your adapter without putting them in case files. Save portable adapter settings in eval/adapter.yaml; GlassKit discovers this file automatically:

uv run --env-file .env glasskit eval run

Example eval/adapter.yaml:

api_url: https://example.test/v1
model: vision-checker
jpeg_quality: 90

Use --adapter-config PATH to load a different YAML or JSON object instead. GlassKit does not expand environment variables inside adapter config files. Read secrets from environment variables in your adapter.

Get started

Guides and reference

Create a new eval case

Seed draft expectations

Review and correct expectations in the browser

Run one case while debugging

Measure nondeterministic stability

Enforce CI quality gates

Use an adapter config file

​Create a new eval case

​Seed draft expectations

​Review and correct expectations in the browser

​Run one case while debugging

​Measure nondeterministic stability

​Enforce CI quality gates

​Use an adapter config file

Create a new eval case

Seed draft expectations

Review and correct expectations in the browser

Run one case while debugging

Measure nondeterministic stability

Enforce CI quality gates

Use an adapter config file