> ## Documentation Index > Fetch the complete documentation index at: https://glasskit.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # CLI reference > Reference GlassKit Eval commands, options, configuration, output formats, and exit codes. Every command supports `--help`. ## `glasskit` Purpose: top-level command group. ```sh theme={null} glasskit --help ``` Options: | Option | Default | Description | | ---------------------- | ------- | ----------------------------------------------- | | `--version` | None | Show the installed GlassKit version and exit. | | `--install-completion` | None | Install shell completion for the current shell. | | `--show-completion` | None | Print shell completion setup text. | | `--help` | None | Show help and exit. | Commands: | Command | Description | | ------- | -------------------------- | | `eval` | Recorded-video eval tools. | ## `glasskit eval` Purpose: command group for recorded-video evals. ```sh theme={null} glasskit eval --help ``` Commands: | Command | Description | | --------------- | ------------------------------------------------------------------------------------------------------ | | `run` | Decode selected frames, call the adapter, compare observations, apply gates, and report results. | | `seed` | Run a labeling adapter and write proposed expectations into selected draft samples. | | `review` | Open the local browser UI for inspecting and correcting timed expectations. | | `validate` | Check eval structure, videos, sample times, and optional adapter construction without running samples. | | `list-samples` | Print the expanded sample schedule for inspection or debugging. | | `export-frames` | Export the eval-decoded image at one or more case timestamps. | | `video-store` | Pull, upload, and prune cached videos backed by cloud object storage. | ## `glasskit eval video-store` Purpose: manage videos backed by an S3-compatible cloud object store. Ordinary eval commands fetch cloud videos automatically. ```sh theme={null} glasskit eval video-store --help ``` Commands: | Command | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `pull` | Download all selected cloud videos. Accepts `--eval-dir` and optional `--case`. | | `upload SOURCE --store NAME` | Upload a local video through the named store's authenticated S3 API and print a case-file `video:` block. Accepts optional `--key` and `--eval-dir`. | | `prune-cache` | Remove abandoned partial downloads older than one hour. Add `--all` to remove downloaded videos too. Operates on the current user's cache across eval directories. | ## `glasskit eval seed` Purpose: fill missing expectations in selected draft cases using an adapter, or explicitly replace existing expectations in the selected scope. ```sh theme={null} glasskit eval seed --case task-01 --target step_1 ``` Options: | Option | Default | Description | | ------------------------ | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--adapter TEXT` | `/adapter.py:create_evaluator` | Labeling adapter target in `:` form. It uses the same contract as the run adapter. | | `--adapter-command TEXT` | None | Process labeling adapter command. Mutually exclusive with `--adapter`; when set, the Python adapter default is not loaded. | | `--eval-dir PATH` | `eval` | Eval directory. | | `--case TEXT` | All cases | Only seed one case by filename or stem. | | `--target TEXT` | All targets | Only seed this target id from the selected cases. Repeat the option to seed multiple targets. Every requested target must exist in the selected case scope. May be used with or without `--case`. | | `--adapter-config PATH` | `/adapter.yaml` when present | YAML or JSON object passed to the selected adapter in its `config` field. | | `--concurrency INTEGER` | `1` | Maximum concurrent per-sample `evaluate` calls within a target. Must be greater than zero. Ignored for adapters using `evaluate_many`, which control their own batch execution. | | `--replace` | `false` | Evaluate and replace existing expectations in the selected scope as well as filling missing ones. | | `--keep-going` | `false` | Checkpoint per-sample adapter or field-extraction errors and continue evaluating. The case YAML remains unchanged unless every selected expectation succeeds. | | `--resume PATH` | None | Resume an incomplete seed checkpoint by its printed path or checkpoint id. The checkpoint restores the original adapter, filters, config, concurrency, and seed options, so it cannot be combined with overrides; only `--eval-dir` may be repeated, to locate a checkpoint given by id. | | `--verbose` | `false` | Print every proposed expectation and set the factory config object's `verbose` field. | When `field` is present on a sample block, `seed` extracts that path from the adapter's observation and writes the extracted value as `expect`; otherwise it writes the complete observation. Existing expectations outside the selected filters, and inside the filters without `--replace`, are preserved. Ignored samples are never seeded, even with `--replace`, and may omit `expect`. Other missing expectations outside the selected filters may remain draft; `run`, `validate`, and `list-samples` reject draft samples only when they fall inside those commands' selected scope. Each successful adapter result is durably checkpointed before the command advances. If seeding is interrupted or an adapter call fails after at least one result succeeds, the case file is left unchanged and the error output prints an exact `glasskit eval seed --resume ...` command. Setup failures and attempts with no successful results do not print a resume command, because rerunning the original command repeats no completed adapter work. Resume evaluates only checkpointed errors and unfinished samples, including when the original operation used `--replace`. Once all selected expectations are available, `seed` validates and atomically replaces the complete case YAML. Resume does not automatically retry calls; each invocation makes at most one new attempt for each selected pending sample. Resume also checks its inputs for changes; if it detects one, resuming stops with a `checkpoint inputs changed` error and the operation must be restarted. Exit behavior: exits `0` after seeding or when the selected scope has nothing to seed, `1` when `--keep-going` attempted the selected scope but one or more expectations remain incomplete, `2` for invalid input, an adapter failure that aborted evaluation, or a case file that cannot be updated, and `130` when interrupted with `Ctrl+C`. Interrupted and incomplete operations retain checkpoints only when they contain successful adapter results. ## `glasskit eval review` Purpose: launch the local eval review UI without loading or running an adapter. ```sh theme={null} glasskit eval review --eval-dir eval --case task-01 --target step_1 --time 7.4 ``` Options: | Option | Default | Description | | ----------------- | ------------ | -------------------------------------------------------------------------------------------- | | `--eval-dir PATH` | `eval` | Eval directory. | | `--case TEXT` | First case | Initially open one case by filename or stem. It does not hide other cases. | | `--target TEXT` | First target | Initially focus one target from `--case`. Requires `--case` and does not hide other targets. | | `--time FLOAT` | None | Initially seek to a finite, nonnegative time in the selected case. Requires `--case`. | | `--port INTEGER` | `0` | Loopback port. `0` chooses an available port. | | `--no-open` | `false` | Print the URL without opening the default browser. | Because edits are saved directly to the case file, commit or copy case files before editing if you want an easy way to review or undo the changes. Saving may reformat the YAML and remove ordinary YAML comments; values stored in sample `comment` and `ignore` fields are preserved. Exit behavior: exits `0` after a normal `Ctrl+C` shutdown and `2` for an invalid eval path or selector, invalid option combination, or failure to load or start the review UI. Failure to open the browser is nonfatal because the printed URL remains usable. ## `glasskit eval export-frames` Purpose: export the exact display-oriented frames GlassKit would pass to an adapter at arbitrary points in one case. ```sh theme={null} glasskit eval export-frames --case task-01 --at 7.5 --at 8.0 ``` Options: | Option | Default | Description | | ------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `--eval-dir PATH` | `eval` | Eval directory. | | `--case TEXT` | Required | Case filename or stem containing the source video. | | `--at FLOAT` | Required | Nonnegative frame time in seconds. Repeat to export multiple times in one video decode. Duplicate times are exported once. | | `--output-dir PATH` | `/runs/frames//` | Directory for exported PNGs. | Each image is named `at-s.png`. A frame with the same destination name is replaced, and the command prints only the absolute path of each written image, in requested order. Timestamps do not need to be declared samples, and draft or ignored samples do not prevent export. Frame selection is identical to eval execution: GlassKit chooses the nearest decoded frame, chooses the earlier frame on a tie, and applies the video's display rotation and reflection before writing a lossless RGB PNG. A time beyond the source video duration is rejected. Exit behavior: exits `0` after exporting every requested frame and `2` for an invalid eval path, case, timestamp, video, or destination. ## `glasskit eval run` Purpose: execute selected eval samples and apply quality gates, with optional repetition for measuring stability. ```sh theme={null} glasskit eval run --case task-01 --output-json eval/runs/results.json ``` Options: | Option | Default | Description | | ------------------------------ | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--adapter TEXT` | `/adapter.py:create_evaluator` | Adapter target in `:` form. | | `--adapter-command TEXT` | None | NDJSON process adapter command. Mutually exclusive with `--adapter`; when set, the Python adapter default is not loaded. | | `--eval-dir PATH` | `eval` | Eval directory. | | `--case TEXT` | All cases | Only run one case by filename or stem. Do not include path separators. | | `--target TEXT` | All targets | Only run this target id from the selected cases. Repeat the option to run multiple targets. Every requested target must exist in the selected case scope. May be used with or without `--case`. | | `--at FLOAT` | None | Only run samples scheduled at this time in seconds. Repeat to select multiple times. Requires `--case` and cannot be combined with `--from` or `--until`. | | `--from FLOAT` | None | Only run samples scheduled at or after this time in seconds. Requires `--case`. | | `--until FLOAT` | None | Only run samples scheduled before this time in seconds. Requires `--case`. | | `--adapter-config PATH` | `/adapter.yaml` when present | YAML or JSON object passed to the selected adapter in its `config` field. | | `--concurrency INTEGER` | `1` | Maximum concurrent per-sample `evaluate` calls within a target. Must be greater than zero. Ignored for adapters using `evaluate_many`, which control their own batch execution. | | `--repeat INTEGER` | `1` | Number of complete executions. Values above `1` run sequential trials with a fresh evaluator for each one. | | `--min-pass-rate FLOAT` | None | Pass-rate gate from `0.0` to `1.0`. Overrides eval-level `thresholds.min_pass_rate` and suppresses case-level gates when set. | | `--min-target-pass-rate FLOAT` | None | Uniform per-target pass-rate gate for targets present in the selected results. Replaces eval-level `thresholds.per_target` gates. | | `--max-failures INTEGER` | None | Maximum failed comparisons. Overrides eval-level `thresholds.max_failures` and suppresses case-level gates when set. | | `--max-flaky-samples INTEGER` | None | Cross-trial maximum number of samples whose status varies. Must be nonnegative and requires `--repeat` of at least `2`. | | `--keep-going` | `false` | Record adapter evaluation or comparison errors as sample results and continue. | | `--resume PATH` | None | Resume an incomplete run checkpoint by its printed path or checkpoint id. The checkpoint restores the original adapter and run options, so it cannot be combined with overrides; only `--eval-dir` may be repeated, to locate a checkpoint given by id. | | `--verbose` | `false` | Print every sample result and set the factory config object's `verbose` field. | | `--output-json PATH` | None | Write a machine-readable JSON report. | | `--artifacts-dir PATH` | None | Base directory for generated artifacts. Failure artifacts are written below its `failures/` subdirectory; when omitted, the base is `/runs/`. | | `--save-failures` | `false` | Save failed or errored sample frames and per-result JSON. | | `--allow-empty` | `false` | Allow evals or cases with no samples. | `--at`, `--from`, and `--until` select samples already scheduled in the case; they do not create samples at arbitrary video times. Repeat `--at` to select multiple timestamps. Each requested timestamp must be present among the samples chosen by `--case` and `--target`. `--from` is inclusive, `--until` is exclusive, either range bound may be used alone, and when both are given `--until` must be greater than `--from`. All three options require `--case`, and `--at` cannot be combined with either range bound. Only selected samples are sent to the adapter, and quality gates apply to the selected results. Every completed sample result is durably checkpointed. If a fail-fast run is interrupted after at least one adapter evaluation completes, its error output prints an exact `glasskit eval run --resume ...` command. With `--keep-going`, the normal report still contains adapter errors and fails the automatic `adapter_errors` gate, while its summary prints a resume command only when the checkpoint contains completed adapter work. Setup failures and attempts where every adapter call fails do not print a resume command. Resume reuses successful evaluations, ordinary comparison failures, ignored samples, and comparison-error results; it evaluates only adapter errors and unfinished samples. No adapter call is retried automatically. A resumed run writes a JSON report or failure artifacts only when the original invocation requested them. Resume also checks its inputs for changes; if it detects one, resuming stops with a `checkpoint inputs changed` error. Exit behavior: exits `0` when every configured gate passes, `1` when the eval completed but one or more gates failed, `2` when setup or runtime errors abort the run, and `130` when interrupted with `Ctrl+C`. ## `glasskit eval validate` Purpose: check an eval directory without evaluating sample observations. A normal `run` performs the same eval structure, video, and sample-time checks before calling the adapter, so a separate validation step is not required. Use `validate` when you want an inexpensive standalone check, such as in a configuration-only CI job or before using a slow or paid adapter. ```sh theme={null} glasskit eval validate --adapter eval/adapter.py:create_evaluator ``` Options: | Option | Default | Description | | ------------------------ | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--eval-dir PATH` | `eval` | Eval directory. | | `--adapter TEXT` | None | Optional Python adapter target to verify. | | `--adapter-command TEXT` | None | Optional process adapter command to verify. Mutually exclusive with `--adapter`. | | `--case TEXT` | All cases | Only validate one case by filename or stem. | | `--target TEXT` | All targets | Only validate this target id from the selected cases. Repeat the option to validate multiple targets. Every requested target must exist in the selected case scope. May be used with or without `--case`. | | `--adapter-config PATH` | `/adapter.yaml` when present | YAML or JSON object passed to the selected adapter during validation. | | `--allow-empty` | `false` | Allow evals or cases with no samples. | When `--adapter` or `--adapter-command` is provided, validation also constructs and closes that adapter. It does not evaluate a sample or verify the adapter's observations. Without either option, validation checks only the eval directory and selected cases. Exit behavior: exits `0` when validation passes, `1` when validation fails, and `2` for CLI usage errors such as combining `--adapter` with `--adapter-command`. ## `glasskit eval list-samples` Purpose: inspect the expanded sample schedule when debugging ranges, timestamp filters, fields, or comparison modes. ```sh theme={null} glasskit eval list-samples --case task-01 ``` Options: | Option | Default | Description | | ----------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--eval-dir PATH` | `eval` | Eval directory. | | `--case TEXT` | All cases | Only list one case by filename or stem. | | `--target TEXT` | All targets | Only list this target id from the selected cases. Repeat the option to list multiple targets. Every requested target must exist in the selected case scope. May be used with or without `--case`. | | `--at FLOAT` | None | Only list samples scheduled at this time in seconds. Repeat to select multiple times. Requires `--case` and cannot be combined with `--from` or `--until`. | | `--from FLOAT` | None | Only list samples scheduled at or after this time in seconds. Requires `--case`. | | `--until FLOAT` | None | Only list samples scheduled before this time in seconds. Requires `--case` and must be greater than `--from` when both are set. | | `--allow-empty` | `false` | Allow evals or cases with no samples. | The table includes each sample's case, target, timestamp, expectation, comparison mode, field, and source. Range blocks are half-open: for example, `range: [1.0, 2.0]` with `every_s: 0.5` produces samples at `1.0` and `1.5`, not `2.0`. Exit behavior: exits `0` when the samples can be listed and `2` when the eval directory cannot be loaded. ## Configuration `glasskit eval` has no global config file. Eval configuration lives in the eval config file and case files within the eval directory. Default values at a glance: | Area | Default When Omitted | | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Eval directory | `eval` from the command's working directory. | | `seed` adapter | Python target `/adapter.py:create_evaluator`; replaced when `--adapter-command` is set. | | `run` adapter | Python target `/adapter.py:create_evaluator`; replaced when `--adapter-command` is set. | | Individual evaluation concurrency | `1`. Increase with `seed --concurrency` or `run --concurrency`. | | `/adapter.yaml` | Optional. Missing file means the adapter receives an empty config object. | | `/config.yaml` | Optional. Missing file means no eval-level thresholds or cloud video stores. | | Case `sampling.every_s` | `0.5` seconds. | | Sample block `every_s` | Inherits the case `sampling.every_s`. | | Sample `field` | Inherits target or case `sample_defaults.field`; otherwise compares the whole adapter observation. | | Sample `compare.mode` | Inherits target or case `sample_defaults.compare`; otherwise inferred from `expect`: non-boolean numbers use `numeric`; booleans, strings, `null`, arrays, and objects use `exact`. | | Numeric `compare.tolerance` | `0.0`. | | `targets..config` | Empty object. Use this as the default place for adapter-specific target metadata. The final adapter target config also includes matching optional metadata (other than `id` and `label`) from `workflow.targets`, with `targets..config` taking precedence. | | Threshold keys | Unset. Missing `min_pass_rate`, `max_failures`, and `per_target..min_pass_rate` keys create no corresponding gate. | | Adapter config | `/adapter.yaml` when present, otherwise an empty object; `--adapter-config` overrides discovery. | | Failure artifacts | Saved only with `--save-failures`; stored below `/runs/failures/` by default. | | Frame exports | Written by `export-frames` below `/runs/frames//` by default. | | Checkpoints | Created automatically below `/runs/checkpoints/`; completed adapter evaluations are fsynced before the command advances, and new error-only checkpoints are discarded. | `/config.yaml` supports eval-level thresholds and named `video_stores`. Cloud video store examples and credential behavior are documented in [Cloud-stored Videos](/docs/eval/cases#cloud-stored-videos). Thresholds use this form: ```yaml theme={null} thresholds: min_pass_rate: 0.9 max_failures: 5 per_target: step_1: min_pass_rate: 0.95 ``` All threshold keys default to unset. `glasskit eval` does not treat a missing `min_pass_rate` as `1.0`, `0.0`, or the current pass rate; it skips that pass-rate gate. If every quality threshold is omitted, ordinary failed comparisons still appear in the console report and JSON output, but they do not fail `glasskit eval run`. If another gate is configured, such as `max_failures` or a per-target `min_pass_rate`, ordinary failed comparisons can still fail the run through that gate. A configured gate with no matching results fails: a `per_target` threshold that names a target absent from the selected results fails at a 0% pass rate rather than passing silently. Per-target gates are skipped only for targets excluded by `--case`, `--target`, or time filters. Pass rates count errored samples in their denominator, so with `--keep-going`, adapter errors lower pass-rate gates in addition to tripping the automatic `adapter_errors` gate. With `--repeat`, quality gates are calculated separately for every trial, and the overall run fails if any trial fails one. Results are never pooled before applying a quality gate. Flaky samples do not fail the run unless `--max-flaky-samples` is configured. A stable failure satisfies `--max-flaky-samples 0`, so combine stability and quality gates when correctness also matters. Adapter evaluation errors, non-JSON adapter observations, and unexpected comparison exceptions abort the run with exit code `2` by default. Completed results remain in the printed checkpoint. With `--keep-going`, those sample-level errors are recorded as results with status `error`, and the automatic `adapter_errors` gate makes the completed run fail with exit code `1`. Adapter-error results remain resumable; comparison errors remain completed diagnostic results. Adapter setup, loading, and close errors still abort the command, but any sample results completed before those errors remain checkpointed. Validation, listing, and running require `expect` on every non-ignored sample in their selected scope. Ignored samples may omit it. If one of those commands reports draft samples, use `glasskit eval seed` to propose their expectations or label them manually. Filters are applied before this check, so a focused command can operate on a ready target while another target in the same case remains draft. Threshold precedence: | Source | Applies To | Notes | | -------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--min-pass-rate` | Selected results | Overrides eval-level `thresholds.min_pass_rate`. When set, case-level gates are not applied. | | `--max-failures` | Selected results | Overrides eval-level `thresholds.max_failures`. When set, case-level gates are not applied. | | `--min-target-pass-rate` | Selected targets | Adds the same per-target pass-rate gate for each target present in the selected results and replaces eval-level `thresholds.per_target` gates. Case-level gates still apply unless `--min-pass-rate` or `--max-failures` is set. | | `--max-flaky-samples` | Repeated run | Counts logical samples with more than one distinct `passed`, `failed`, or `error` status across trials. It does not measure whether stable outcomes are correct. | | `/config.yaml` | Selected results | Applies after CLI overrides. Eval-level per-target gates for targets outside a case, target, or time-window filtered run are skipped. | | `cases/.yaml` `thresholds` | That case | Applies per case unless `--min-pass-rate` or `--max-failures` is set. | With `--repeat`, the quality-gate precedence above is resolved the same way for each trial, then each gate is evaluated independently against that trial's results. Other precedence rules: | Area | Rule | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Range sampling | A sample block's `every_s` overrides case-level `sampling.every_s`. | | Sample settings | A sample block overrides target `sample_defaults`, which overrides case `sample_defaults`. A declared `compare` replaces the inherited comparison as one value. | | Target metadata | `targets..config` is the default place for adapter target metadata and overrides matching keys from optional `workflow.targets` metadata. | | Adapter config | `/adapter.yaml`, or the explicit `--adapter-config` override, is independent of the eval config file and case files and is passed only to the selected adapter. | ## Environment variables `glasskit eval` reads one CLI-specific environment variable: `GLASSKIT_EVAL_CACHE_DIR` overrides the per-user cache directory for downloaded cloud videos described in [Cloud-stored Videos](/docs/eval/cases#cloud-stored-videos). It does not read user input from stdin. Adapters may read any environment variables your app needs, such as API keys, backend URLs, or feature flags. Command adapters inherit the GlassKit Eval process environment, and GlassKit Eval reserves their stdin and stdout for the process protocol. Keep secrets out of case files and adapter config files. With `uv`, pass a dotenv file to `uv run`: ```sh theme={null} uv run --env-file .env glasskit eval run ``` ## Output formats Human-readable output is printed as tables to stdout. Pass `--output-json PATH` to retain a machine-readable report after the run. Each sample result records the complete adapter observation in `observed` and the value selected by `field` in `observed_value`, so diagnostic metadata remains available even when only one nested value determines pass/fail. The report's `checkpoint` object records its checkpoint path, whether this invocation resumed it, and how many adapter-error results remain resumable. See the [JSON output reference](https://github.com/RealComputer/GlassKit/blob/main/cli/JSON_OUTPUT.md) for the complete report format, a repeated-run example, and result-structure semantics. Checkpoints contain adapter configuration and observations and are written with owner-only file permissions. Treat `/runs/` as sensitive disposable state: keep it out of version control, retain an incomplete checkpoint only while recovery is useful, and remove it manually when it is no longer needed. `--save-failures` writes artifacts for every failed or errored sample attempt. To prevent repeated executions from overwriting one another, files are grouped under `/runs/failures/trial-NNN/` by default or `/failures/trial-NNN/` when `--artifacts-dir` is provided. A run without `--repeat` uses `trial-001`. Each saved result includes a JPEG frame and a JSON metadata file named with the case, target, sample index, and timestamp; the metadata also records its one-based trial number. ## Exit codes | Code | Meaning | Fix | | ----: | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `0` | Command succeeded. For `run`, every configured gate passed. | No action needed. | | `1` | Validation failed, `seed --keep-going` retained one or more incomplete expectations, or `run` completed but one or more gates failed. | Read the validation issues, incomplete-seed message, or gate tables. Use the printed resume command to retry only adapter errors and unfinished samples. | | `2` | A CLI usage error, setup error, config error, video error, adapter loading error, or adapter runtime error aborted the command. | Read the error message and validate the eval directory. If a checkpoint is printed, resume it after resolving the error; use `--keep-going` on a new operation when other samples should continue after an error. | | `130` | `run` or `seed` was interrupted with `Ctrl+C`. | Rerun the command, or use the printed `--resume` command when the checkpoint retained completed adapter work. | ## Support Questions, bug reports, feature requests, and pull requests are welcome. Use whichever path is easiest: * Discord: [https://discord.gg/v5ayGKhPNP](https://discord.gg/v5ayGKhPNP) * GitHub issues and pull requests: [https://github.com/RealComputer/GlassKit](https://github.com/RealComputer/GlassKit) For a real app-backed setup, see [this example](https://github.com/RealComputer/GlassKit/tree/main/examples/origami/backend).