> ## Documentation Index
> Fetch the complete documentation index at: https://glasskit.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Common workflows

> Create, seed, review, debug, repeat, and gate GlassKit Eval cases.

## Create a new eval case

Goal: create the required directory structure and a case file in `eval/` from an existing recording.

Commands:

```sh theme={null}
mkdir -p eval/cases
cat > eval/cases/task-02.yaml <<'YAML'
video: ../../../recordings/task-02.mov
description: Replace this note with what task-02 should cover.
sampling:
  every_s: 0.5
targets:
  step_2:
    samples:
    - range: [0.0, 6.0]
      expect: false
    - range: [6.0, 12.0]
      expect: true
YAML
```

Expected result: `eval/cases/task-02.yaml` points to `task-02.mov` in the sibling `recordings/` directory and labels the expected value before and after the six-second mark.

Note: If you prefer colocated fixtures, put the recording in `eval/cases/` and set `video:` to the filename, such as `task-02.mov`.

## Seed draft expectations

Hand-labeling is the clearest way to start. When a case has many samples, you can instead omit `expect` and ask an adapter to propose the missing values:

```sh theme={null}
uv run --env-file .env glasskit eval seed --case task-02
uv run glasskit eval review --case task-02
```

`seed` uses the same adapter as `run` by default, fills only omitted expectations, and preserves existing labels. Use `--adapter` or `--adapter-command` to label with a different adapter, and use `--case` or `--target` to narrow the work. Treat generated values as proposals and review them before relying on the eval.

## Review and correct expectations in the browser

Goal: make recorded-video expectations faster to verify and correct.

Without the review UI, verifying expectations means juggling a media player and YAML editor while matching timestamps by hand. The review UI puts the video, expanded sample schedule, and editing controls in one place, making corrections faster and less error-prone.

Command:

```sh theme={null}
uv run glasskit eval review --eval-dir eval
```

To jump directly to a failure reported by a separate eval run, include its case, target, and timestamp:

```sh theme={null}
uv run glasskit eval review --eval-dir eval --case task-01 --target step_1 --time 7.4
```

The command opens a local browser UI where you can compare labeled moments with their source video and add, move, edit, or delete samples. Changes are saved automatically to the case file.

## Run one case while debugging

Goal: run a focused eval and print every sample result.

Command:

```sh theme={null}
uv run glasskit eval run --case task-01 --target step_1 --verbose --keep-going --save-failures --output-json eval/runs/results.json --artifacts-dir eval/runs/artifacts
```

Expected output: focused case and target progress, every selected sample result, a final summary, and a per-target table.

Note: `--keep-going` records adapter evaluation errors and comparison errors as sample results instead of aborting on the first sample error. Every completed adapter result is also checkpointed, so when there is reusable progress the printed `glasskit eval run --resume ...` command can retry adapter errors without rerunning completed samples. `--save-failures` writes JPEG frames and per-result JSON for failed or errored samples. Treat `eval/runs/` as disposable output and add it to your app repo's `.gitignore` if you keep generated eval reports out of source control.

## Measure nondeterministic stability

Goal: run the same selected eval three times and identify samples whose outcomes vary.

With `--repeat N`, GlassKit Eval executes the same selected sample schedule `N` times. Each complete repetition is called a trial, and each evaluation of a sample within a trial is an attempt.

Command:

```sh theme={null}
uv run --env-file .env glasskit eval run --concurrency 2 --repeat 3 --max-flaky-samples 0 --output-json eval/runs/repeated-results.json
```

Expected output: three sequential trial progress sections, a per-trial quality table, minimum/mean/maximum trial pass rates, per-target stability, and a table of flaky or consistently failing samples. The command constructs and closes a fresh evaluator for every trial. `--concurrency 2` still permits at most two individual evaluations in flight because trials themselves are never run concurrently.

Every trial uses the same filters and selected schedule. Quality gates such as `--min-pass-rate` apply independently to each trial, and the run fails if any trial fails one; results are never pooled before applying a quality gate. `--max-flaky-samples 0` checks only whether sample statuses vary across trials, so combine it with a correctness gate such as `--min-pass-rate 0.9` when both stability and quality should affect the exit code. Repeating an eval multiplies its adapter work and provider cost by the repeat count.

## Enforce CI quality gates

Goal: make the command fail when quality drops below your threshold.

Command:

```sh theme={null}
uv run glasskit eval run --min-pass-rate 0.9 --min-target-pass-rate 0.85 --max-failures 3 --output-json eval/runs/results.json
```

Expected behavior: the process exits `0` when every configured quality gate passes, `1` when the eval completed but one or more gates failed, and `2` for setup or runtime errors that abort the run.

Note: Threshold defaults are intentionally unset. Without `--min-pass-rate`, `--min-target-pass-rate`, `--max-failures`, or YAML thresholds, failed comparisons are visible in the report but do not fail the command. Always configure a gate for CI.

## Use an adapter config file

Goal: pass runtime settings to your adapter without putting them in case files.

Save portable adapter settings in `eval/adapter.yaml`; GlassKit discovers this file automatically:

```sh theme={null}
uv run --env-file .env glasskit eval run
```

Example `eval/adapter.yaml`:

```yaml theme={null}
api_url: https://example.test/v1
model: vision-checker
jpeg_quality: 90
```

Use `--adapter-config PATH` to load a different YAML or JSON object instead. GlassKit does not expand environment variables inside adapter config files. Read secrets from environment variables in your adapter.