> ## Documentation Index > Fetch the complete documentation index at: https://glasskit.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Cases, videos, and comparisons > Define GlassKit Eval cases, store their videos, and configure result comparisons. ## Eval directory layout A typical layout keeps the eval directory and adapter code in the app repo while storing recordings outside the repo: ```text theme={null} recordings/ task-01.mp4 task-02.mp4 your-app-repo/ eval/ adapter.py adapter.yaml # Optional adapter config file config.yaml # Optional thresholds and cloud video stores cases/ task-01.yaml # Case file task-02.yaml ``` You can also keep videos next to the case file and reference them with a local filename such as `video: task-01.mp4`. Local paths are the simplest setup. The `video:` path in the case file is resolved relative to that file. If recordings are too large to keep locally or share through Git, use a cloud video store as described below. The adapter config file is optional and must be named `adapter.yaml` for automatic discovery. The eval config file is also optional and supports eval-level `thresholds` and named `video_stores`; it must be named `config.yaml`. Case files must live directly under `cases/` and use the `.yaml` suffix. Supported video suffixes are `.mp4`, `.mov`, `.m4v`, `.webm`, and `.mkv`. Timestamps in case files are seconds from the start of the decoded clip. ## Cloud-stored videos GlassKit supports AWS S3, Cloudflare R2, and other S3-compatible object stores. This keeps large recordings out of your app repository and makes them easier to share with a team. Eval commands download videos when needed and reuse cached copies on later runs. Define a named store in `/config.yaml`. For a private Cloudflare R2 store, configure credentials through environment variables: ```yaml theme={null} video_stores: team-videos: type: s3 bucket: team-eval-videos endpoint_url: https://.r2.cloudflarestorage.com region: auto access_key_id_env: EVAL_STORAGE_ACCESS_KEY_ID secret_access_key_env: EVAL_STORAGE_SECRET_ACCESS_KEY ``` Keep the credential values in an ignored `.env` file or your team's secret manager: ```dotenv theme={null} EVAL_STORAGE_ACCESS_KEY_ID=... EVAL_STORAGE_SECRET_ACCESS_KEY=... ``` Upload a recording from the directory containing your eval setup: ```sh theme={null} uv run --env-file .env glasskit eval video-store upload recordings/task-01.mp4 --store team-videos ``` The command prints a `video:` block to copy into the case file: ```yaml theme={null} video: store: team-videos key: abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.mp4 sha256: abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789 targets: step_1: samples: - at: 0 expect: false ``` The `store`, `key`, and `sha256` values identify the uploaded video. Copy them as printed rather than writing them by hand. Ordinary `run`, `seed`, `validate`, `export-frames`, and `review` commands download cloud videos automatically. For AWS S3, omit `endpoint_url` and use the bucket's AWS region. You can also omit the custom credential variable names to use the standard AWS credential configuration: ```yaml theme={null} video_stores: team-evals: type: s3 bucket: team-eval-videos region: us-east-1 ``` `access_key_id_env` and `secret_access_key_env` must be set together, and temporary credentials can add `session_token_env`. Only `bucket` is required: `type` defaults to `s3` and `region` defaults to `us-east-1`. Omit `--key` when uploading to let GlassKit use `` as the object key. Uploading is idempotent: when the destination object already exists with a matching size and SHA-256, the command reports that and prints the same `video:` block; any other existing object at the key is refused rather than overwritten. Use `pull` when you want to download selected videos ahead of time: ```sh theme={null} uv run --env-file .env glasskit eval video-store pull uv run --env-file .env glasskit eval video-store pull --case task-01 ``` `list-samples` validates cloud references without downloading videos. Downloads are stored in a per-user cache outside the eval directory — `~/Library/Caches/glasskit/eval/videos` on macOS, `$XDG_CACHE_HOME/glasskit/eval/videos` on Linux when `XDG_CACHE_HOME` is set (otherwise `~/.cache/glasskit/eval/videos`), and `%LOCALAPPDATA%\GlassKit\Cache\eval\videos` on Windows — and shared by all of that user's eval directories. Set `GLASSKIT_EVAL_CACHE_DIR` to override the location. To clear downloaded videos, run `glasskit eval video-store prune-cache --all`; they will be downloaded again when needed. ### Public downloads For a public repository, you may want anyone to run the eval without storage credentials while allowing only maintainers to upload. Expose the bucket through a public HTTP URL and add it to the store: ```yaml theme={null} video_stores: public-evals: type: s3 bucket: public-eval-videos endpoint_url: https:// region: public_base_url: https:// access_key_id_env: EVAL_STORAGE_ACCESS_KEY_ID secret_access_key_env: EVAL_STORAGE_SECRET_ACCESS_KEY ``` Downloads then use `public_base_url` without credentials. Uploads still require the configured credentials. ## Case file reference Here is a representative case file: ```yaml theme={null} video: task-01.mp4 description: Step 1 should be detected after the bracket is seated. sampling: every_s: 0.5 targets: step_1: label: Step 1 config: prompt_id: workflow.step_1 reference_image: assets/step_1.png samples: - range: [0.0, 6.8] expect: false comment: The bracket is not seated yet. - range: [7.4, 11.8] every_s: 0.25 field: result.matches expect: true - at: 11.9 expect: true ignore: Difficult frame with known flaky observations. step_2: label: Step 2 samples: - at: [4.0, 6.0] # Two discrete samples, not a range. expect: false thresholds: min_pass_rate: 0.9 max_failures: 2 per_target: step_1: min_pass_rate: 0.95 ``` Case fields: | Field | Required | Description | | ------------------ | -------: | ------------------------------------------------------------------------------------------------------------------------------------- | | `video` | Yes | Local path resolved relative to the case file, or an object with required `store`, `key`, and `sha256` fields for a cloud video. | | `description` | No | Human-readable case note. | | `sampling.every_s` | No | Default range sampling interval in seconds. Defaults to `0.5`; must be greater than `0`. | | `sample_defaults` | No | Case-wide defaults for sample `field` and `compare`. Target defaults override these values, and sample blocks override both scopes. | | `workflow.targets` | No | Optional advanced target metadata list for imported or generated workflow definitions. | | `targets` | Yes | Mapping of target id to target definition. Must contain at least one target. | | `thresholds` | No | Case-level gates: `min_pass_rate`, `max_failures`, and `per_target..min_pass_rate`. Omitted keys create no gate for that key. | Target fields: | Field | Required | Description | | ----------------- | -------: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `label` | No | Display name shown in reports. | | `config` | No | Adapter-specific metadata for the target. Use this as the default place for prompt IDs, rubric IDs, reference assets, confidence thresholds, or other target-specific settings. Defaults to an empty object. Values override matching keys from `workflow.targets`. | | `sample_defaults` | No | Target-wide defaults for sample `field` and `compare`. These override case defaults. | | `samples` | Yes | List of sample blocks. Empty lists are invalid unless `--allow-empty` is used. | Most evals should put adapter metadata directly under `targets..config`. `workflow.targets` is useful when an eval is generated from or synchronized with an app workflow manifest and workflow-owned metadata should stay separate from eval-owned samples, expectations, and per-case overrides. Each workflow target needs an `id`; `label` and extra metadata keys are allowed. Entries are matched by `id`, and their metadata keys other than `id` and `label` are merged into the adapter target config before `targets..config` is applied. A workflow `label` is used as the target's display label when the target does not define one, and entries whose `id` matches no target are ignored: ```yaml theme={null} workflow: targets: - id: step_1 app_step_id: 123 prompt_id: workflow.step_1 targets: step_1: config: confidence_threshold: 0.85 samples: - at: 8.0 expect: true ``` Sample block fields: | Field | Required | Description | | --------- | -------------------------------: | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `range` | Conditionally | Two-element `[start, end]` interval in seconds. Exactly one of `range` or `at` is required. The interval is half-open. | | `at` | Conditionally | One timestamp or a list of timestamps in seconds. Exactly one of `range` or `at` is required. Lists are sorted during expansion. | | `expect` | For non-ignored runnable samples | JSON-like expected value: `null`, boolean, finite number, string, array, or object with string keys. Omit it to create a draft sample for `seed`; explicit `null` is a labeled expectation. Ignored samples may omit it without becoming drafts. | | `every_s` | No | Per-block range sampling interval. Defaults to `sampling.every_s` for the case, which defaults to `0.5`. | | `field` | No | Dot-separated path to extract from the adapter observation before comparison. An omitted value inherits target or case `sample_defaults`; without a default, the whole observation is compared. Explicit `null` clears an inherited field. | | `compare` | No | Comparison config with `mode` and optional `tolerance`. An omitted value inherits target or case `sample_defaults`; without a default, mode is inferred from `expect` and numeric tolerance is `0.0`. Explicit `null` clears an inherited comparison. | | `comment` | No | Human-readable note retained with the expectation. It does not affect adapter calls or comparison. | | `ignore` | No | Nonempty reason for ignoring this block. Ignored samples do not need `expect`; they are reported but are not decoded, sent to the adapter, seeded, or included in pass rates, failure counts, or quality gates. | Sample times must be finite and nonnegative. Ranges must have `end` greater than `start`. Overlapping or duplicate samples for the same target are invalid; overlap is checked on the declared `at` times and `range` intervals, so two blocks with overlapping ranges are rejected even when their expanded samples would not collide. Expansion is capped at 10,000 samples across all targets in one case; pathological ranges are rejected before their samples are materialized. Unknown keys anywhere in a case file are validation errors, so a misspelled field name fails fast instead of being silently ignored; only `workflow.targets` entries accept extra metadata keys. Use `ignore` for a known exceptional sample that should remain documented without affecting a run. An ignored `at` list or `range` ignores every expanded sample in that block; use a single `at` timestamp when only one sample is exceptional. Sample settings use the precedence `sample block > target sample_defaults > case sample_defaults > built-in behavior`. `compare` is inherited or replaced as one complete value rather than merged key by key, so an override never retains an unrelated tolerance from a broader scope. Only `field` and `compare` can be defaulted; expectations, locations, comments, and ignore reasons remain explicit sample-block properties. For example, these defaults apply a structured result envelope and subset comparison to every target in the case, while the `confidence` target replaces both settings: ```yaml theme={null} sample_defaults: field: result compare: mode: json_subset targets: object_detection: samples: - range: [180.0, 182.0] expect: object: coffee_mug color: red confidence: sample_defaults: field: result.confidence compare: mode: numeric tolerance: 0.05 samples: - at: 182.0 expect: 0.9 ``` ## Comparison reference The adapter observation and the sample `expect` value must both be JSON-like. For simple checks, return only the value you want compared and omit `field`. Use `field` when the adapter naturally returns a structured result but only one nested value should determine correctness. For example, an adapter can return its result alongside diagnostic metadata; selecting the result with `field` makes it the seeded and compared value while preserving the complete adapter response in machine-readable reports and saved failure artifacts. Field paths are dot-separated. Mapping keys are matched by name, and list indexes can be addressed with nonnegative numeric path parts such as `detections.0.label`. Missing fields fail the sample with an `adapter observation is missing configured field: ...` reason. Supported comparison modes: | Mode | Description | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `exact` | Observed value must equal `expect`. Booleans only match booleans. | | `numeric` | Observed and expected values must be numbers. `tolerance` defaults to `0.0`. | | `json_subset` | Every expected key and value must be present in the observed object. For arrays, expected items are matched one-for-one against observed items, so duplicate expected items require duplicate observed matches. | | `set_equals` | Observed and expected arrays are compared as unordered JSON sets. | | `set_contains_any` | At least one expected array item must be present in the observed array. | | `set_contains_all` | Every expected array item must be present in the observed array. | Default comparison modes are inferred from `expect`: booleans, strings, and `null` use `exact`; numbers use `numeric`; arrays and objects use `exact`. Example: ```yaml theme={null} targets: detector: samples: - at: 2.0 field: result.matches expect: true - at: 3.0 field: result.confidence expect: 0.8 compare: mode: numeric tolerance: 0.05 - at: 4.0 field: detected_classes expect: - bracket - fastener compare: mode: set_contains_all ```