--help.
glasskit
Purpose: top-level command group.
Commands:
glasskit eval
Purpose: command group for recorded-video evals.
glasskit eval video-store
Purpose: manage videos backed by an S3-compatible cloud object store. Ordinary eval commands fetch cloud videos automatically.
glasskit eval seed
Purpose: fill missing expectations in selected draft cases using an adapter, or explicitly replace existing expectations in the selected scope.
When
field is present on a sample block, seed extracts that path from the adapter’s observation and writes the extracted value as expect; otherwise it writes the complete observation. Existing expectations outside the selected filters, and inside the filters without --replace, are preserved. Ignored samples are never seeded, even with --replace, and may omit expect. Other missing expectations outside the selected filters may remain draft; run, validate, and list-samples reject draft samples only when they fall inside those commands’ selected scope.
Each successful adapter result is durably checkpointed before the command advances. If seeding is interrupted or an adapter call fails after at least one result succeeds, the case file is left unchanged and the error output prints an exact glasskit eval seed --resume ... command. Setup failures and attempts with no successful results do not print a resume command, because rerunning the original command repeats no completed adapter work. Resume evaluates only checkpointed errors and unfinished samples, including when the original operation used --replace. Once all selected expectations are available, seed validates and atomically replaces the complete case YAML. Resume does not automatically retry calls; each invocation makes at most one new attempt for each selected pending sample. Resume also checks its inputs for changes; if it detects one, resuming stops with a checkpoint inputs changed error and the operation must be restarted.
Exit behavior: exits 0 after seeding or when the selected scope has nothing to seed, 1 when --keep-going attempted the selected scope but one or more expectations remain incomplete, 2 for invalid input, an adapter failure that aborted evaluation, or a case file that cannot be updated, and 130 when interrupted with Ctrl+C. Interrupted and incomplete operations retain checkpoints only when they contain successful adapter results.
glasskit eval review
Purpose: launch the local eval review UI without loading or running an adapter.
Because edits are saved directly to the case file, commit or copy case files before editing if you want an easy way to review or undo the changes. Saving may reformat the YAML and remove ordinary YAML comments; values stored in sample
comment and ignore fields are preserved.
Exit behavior: exits 0 after a normal Ctrl+C shutdown and 2 for an invalid eval path or selector, invalid option combination, or failure to load or start the review UI. Failure to open the browser is nonfatal because the printed URL remains usable.
glasskit eval export-frames
Purpose: export the exact display-oriented frames GlassKit would pass to an adapter at arbitrary points in one case.
Each image is named
at-<timestamp>s.png. A frame with the same destination name is replaced, and the command prints only the absolute path of each written image, in requested order. Timestamps do not need to be declared samples, and draft or ignored samples do not prevent export.
Frame selection is identical to eval execution: GlassKit chooses the nearest decoded frame, chooses the earlier frame on a tie, and applies the video’s display rotation and reflection before writing a lossless RGB PNG. A time beyond the source video duration is rejected.
Exit behavior: exits 0 after exporting every requested frame and 2 for an invalid eval path, case, timestamp, video, or destination.
glasskit eval run
Purpose: execute selected eval samples and apply quality gates, with optional repetition for measuring stability.
--at, --from, and --until select samples already scheduled in the case; they do not create samples at arbitrary video times. Repeat --at to select multiple timestamps. Each requested timestamp must be present among the samples chosen by --case and --target. --from is inclusive, --until is exclusive, either range bound may be used alone, and when both are given --until must be greater than --from. All three options require --case, and --at cannot be combined with either range bound. Only selected samples are sent to the adapter, and quality gates apply to the selected results.
Every completed sample result is durably checkpointed. If a fail-fast run is interrupted after at least one adapter evaluation completes, its error output prints an exact glasskit eval run --resume ... command. With --keep-going, the normal report still contains adapter errors and fails the automatic adapter_errors gate, while its summary prints a resume command only when the checkpoint contains completed adapter work. Setup failures and attempts where every adapter call fails do not print a resume command. Resume reuses successful evaluations, ordinary comparison failures, ignored samples, and comparison-error results; it evaluates only adapter errors and unfinished samples. No adapter call is retried automatically. A resumed run writes a JSON report or failure artifacts only when the original invocation requested them. Resume also checks its inputs for changes; if it detects one, resuming stops with a checkpoint inputs changed error.
Exit behavior: exits 0 when every configured gate passes, 1 when the eval completed but one or more gates failed, 2 when setup or runtime errors abort the run, and 130 when interrupted with Ctrl+C.
glasskit eval validate
Purpose: check an eval directory without evaluating sample observations. A normal run performs the same eval structure, video, and sample-time checks before calling the adapter, so a separate validation step is not required. Use validate when you want an inexpensive standalone check, such as in a configuration-only CI job or before using a slow or paid adapter.
When
--adapter or --adapter-command is provided, validation also constructs and closes that adapter. It does not evaluate a sample or verify the adapter’s observations. Without either option, validation checks only the eval directory and selected cases.
Exit behavior: exits 0 when validation passes, 1 when validation fails, and 2 for CLI usage errors such as combining --adapter with --adapter-command.
glasskit eval list-samples
Purpose: inspect the expanded sample schedule when debugging ranges, timestamp filters, fields, or comparison modes.
The table includes each sample’s case, target, timestamp, expectation, comparison mode, field, and source. Range blocks are half-open: for example,
range: [1.0, 2.0] with every_s: 0.5 produces samples at 1.0 and 1.5, not 2.0.
Exit behavior: exits 0 when the samples can be listed and 2 when the eval directory cannot be loaded.
Configuration
glasskit eval has no global config file. Eval configuration lives in the eval config file and case files within the eval directory.
Default values at a glance:
<eval-dir>/config.yaml supports eval-level thresholds and named video_stores. Cloud video store examples and credential behavior are documented in Cloud-stored Videos. Thresholds use this form:
glasskit eval does not treat a missing min_pass_rate as 1.0, 0.0, or the current pass rate; it skips that pass-rate gate. If every quality threshold is omitted, ordinary failed comparisons still appear in the console report and JSON output, but they do not fail glasskit eval run. If another gate is configured, such as max_failures or a per-target min_pass_rate, ordinary failed comparisons can still fail the run through that gate.
A configured gate with no matching results fails: a per_target threshold that names a target absent from the selected results fails at a 0% pass rate rather than passing silently. Per-target gates are skipped only for targets excluded by --case, --target, or time filters. Pass rates count errored samples in their denominator, so with --keep-going, adapter errors lower pass-rate gates in addition to tripping the automatic adapter_errors gate.
With --repeat, quality gates are calculated separately for every trial, and the overall run fails if any trial fails one. Results are never pooled before applying a quality gate. Flaky samples do not fail the run unless --max-flaky-samples is configured. A stable failure satisfies --max-flaky-samples 0, so combine stability and quality gates when correctness also matters.
Adapter evaluation errors, non-JSON adapter observations, and unexpected comparison exceptions abort the run with exit code 2 by default. Completed results remain in the printed checkpoint. With --keep-going, those sample-level errors are recorded as results with status error, and the automatic adapter_errors gate makes the completed run fail with exit code 1. Adapter-error results remain resumable; comparison errors remain completed diagnostic results. Adapter setup, loading, and close errors still abort the command, but any sample results completed before those errors remain checkpointed.
Validation, listing, and running require expect on every non-ignored sample in their selected scope. Ignored samples may omit it. If one of those commands reports draft samples, use glasskit eval seed to propose their expectations or label them manually. Filters are applied before this check, so a focused command can operate on a ready target while another target in the same case remains draft.
Threshold precedence:
With
--repeat, the quality-gate precedence above is resolved the same way for each trial, then each gate is evaluated independently against that trial’s results.
Other precedence rules:
Environment variables
glasskit eval reads one CLI-specific environment variable: GLASSKIT_EVAL_CACHE_DIR overrides the per-user cache directory for downloaded cloud videos described in Cloud-stored Videos. It does not read user input from stdin.
Adapters may read any environment variables your app needs, such as API keys, backend URLs, or feature flags. Command adapters inherit the GlassKit Eval process environment, and GlassKit Eval reserves their stdin and stdout for the process protocol. Keep secrets out of case files and adapter config files. With uv, pass a dotenv file to uv run:
Output formats
Human-readable output is printed as tables to stdout. Pass--output-json PATH to retain a machine-readable report after the run. Each sample result records the complete adapter observation in observed and the value selected by field in observed_value, so diagnostic metadata remains available even when only one nested value determines pass/fail. The report’s checkpoint object records its checkpoint path, whether this invocation resumed it, and how many adapter-error results remain resumable. See the JSON output reference for the complete report format, a repeated-run example, and result-structure semantics.
Checkpoints contain adapter configuration and observations and are written with owner-only file permissions. Treat <eval-dir>/runs/ as sensitive disposable state: keep it out of version control, retain an incomplete checkpoint only while recovery is useful, and remove it manually when it is no longer needed.
--save-failures writes artifacts for every failed or errored sample attempt. To prevent repeated executions from overwriting one another, files are grouped under <eval-dir>/runs/failures/trial-NNN/ by default or <artifacts-dir>/failures/trial-NNN/ when --artifacts-dir is provided. A run without --repeat uses trial-001. Each saved result includes a JPEG frame and a JSON metadata file named with the case, target, sample index, and timestamp; the metadata also records its one-based trial number.
Exit codes
Support
Questions, bug reports, feature requests, and pull requests are welcome. Use whichever path is easiest:- Discord: https://discord.gg/v5ayGKhPNP
- GitHub issues and pull requests: https://github.com/RealComputer/GlassKit