Cases, videos, and comparisons

Eval directory layout

A typical layout keeps the eval directory and adapter code in the app repo while storing recordings outside the repo:

recordings/
  task-01.mp4
  task-02.mp4
your-app-repo/
  eval/
    adapter.py
    adapter.yaml # Optional adapter config file
    config.yaml # Optional thresholds and cloud video stores
    cases/
      task-01.yaml # Case file
      task-02.yaml

You can also keep videos next to the case file and reference them with a local filename such as video: task-01.mp4. Local paths are the simplest setup. The video: path in the case file is resolved relative to that file. If recordings are too large to keep locally or share through Git, use a cloud video store as described below. The adapter config file is optional and must be named adapter.yaml for automatic discovery. The eval config file is also optional and supports eval-level thresholds and named video_stores; it must be named config.yaml. Case files must live directly under cases/ and use the .yaml suffix. Supported video suffixes are .mp4, .mov, .m4v, .webm, and .mkv. Timestamps in case files are seconds from the start of the decoded clip.

Cloud-stored videos

GlassKit supports AWS S3, Cloudflare R2, and other S3-compatible object stores. This keeps large recordings out of your app repository and makes them easier to share with a team. Eval commands download videos when needed and reuse cached copies on later runs. Define a named store in <eval-dir>/config.yaml. For a private Cloudflare R2 store, configure credentials through environment variables:

video_stores:
  team-videos:
    type: s3
    bucket: team-eval-videos
    endpoint_url: https://<ACCOUNT_ID>.r2.cloudflarestorage.com
    region: auto
    access_key_id_env: EVAL_STORAGE_ACCESS_KEY_ID
    secret_access_key_env: EVAL_STORAGE_SECRET_ACCESS_KEY

Keep the credential values in an ignored .env file or your team’s secret manager:

EVAL_STORAGE_ACCESS_KEY_ID=...
EVAL_STORAGE_SECRET_ACCESS_KEY=...

Upload a recording from the directory containing your eval setup:

uv run --env-file .env glasskit eval video-store upload recordings/task-01.mp4 --store team-videos

The command prints a video: block to copy into the case file:

video:
  store: team-videos
  key: abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.mp4
  sha256: abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789
targets:
  step_1:
    samples:
    - at: 0
      expect: false

The store, key, and sha256 values identify the uploaded video. Copy them as printed rather than writing them by hand. Ordinary run, seed, validate, export-frames, and review commands download cloud videos automatically. For AWS S3, omit endpoint_url and use the bucket’s AWS region. You can also omit the custom credential variable names to use the standard AWS credential configuration:

video_stores:
  team-evals:
    type: s3
    bucket: team-eval-videos
    region: us-east-1

access_key_id_env and secret_access_key_env must be set together, and temporary credentials can add session_token_env. Only bucket is required: type defaults to s3 and region defaults to us-east-1. Omit --key when uploading to let GlassKit use <sha256><extension> as the object key. Uploading is idempotent: when the destination object already exists with a matching size and SHA-256, the command reports that and prints the same video: block; any other existing object at the key is refused rather than overwritten. Use pull when you want to download selected videos ahead of time:

uv run --env-file .env glasskit eval video-store pull
uv run --env-file .env glasskit eval video-store pull --case task-01

list-samples validates cloud references without downloading videos. Downloads are stored in a per-user cache outside the eval directory — ~/Library/Caches/glasskit/eval/videos on macOS, $XDG_CACHE_HOME/glasskit/eval/videos on Linux when XDG_CACHE_HOME is set (otherwise ~/.cache/glasskit/eval/videos), and %LOCALAPPDATA%\GlassKit\Cache\eval\videos on Windows — and shared by all of that user’s eval directories. Set GLASSKIT_EVAL_CACHE_DIR to override the location. To clear downloaded videos, run glasskit eval video-store prune-cache --all; they will be downloaded again when needed.

Public downloads

For a public repository, you may want anyone to run the eval without storage credentials while allowing only maintainers to upload. Expose the bucket through a public HTTP URL and add it to the store:

video_stores:
  public-evals:
    type: s3
    bucket: public-eval-videos
    endpoint_url: https://<S3_API_ENDPOINT>
    region: <REGION>
    public_base_url: https://<PUBLIC_BUCKET_HOST>
    access_key_id_env: EVAL_STORAGE_ACCESS_KEY_ID
    secret_access_key_env: EVAL_STORAGE_SECRET_ACCESS_KEY

Downloads then use public_base_url without credentials. Uploads still require the configured credentials.

Case file reference

Here is a representative case file:

video: task-01.mp4
description: Step 1 should be detected after the bracket is seated.
sampling:
  every_s: 0.5
targets:
  step_1:
    label: Step 1
    config:
      prompt_id: workflow.step_1
      reference_image: assets/step_1.png
    samples:
    - range: [0.0, 6.8]
      expect: false
      comment: The bracket is not seated yet.
    - range: [7.4, 11.8]
      every_s: 0.25
      field: result.matches
      expect: true
    - at: 11.9
      expect: true
      ignore: Difficult frame with known flaky observations.
  step_2:
    label: Step 2
    samples:
    - at: [4.0, 6.0] # Two discrete samples, not a range.
      expect: false
thresholds:
  min_pass_rate: 0.9
  max_failures: 2
  per_target:
    step_1:
      min_pass_rate: 0.95

Case fields:

Field	Required	Description
`video`	Yes	Local path resolved relative to the case file, or an object with required `store`, `key`, and `sha256` fields for a cloud video.
`description`	No	Human-readable case note.
`sampling.every_s`	No	Default range sampling interval in seconds. Defaults to `0.5`; must be greater than `0`.
`sample_defaults`	No	Case-wide defaults for sample `field` and `compare`. Target defaults override these values, and sample blocks override both scopes.
`workflow.targets`	No	Optional advanced target metadata list for imported or generated workflow definitions.
`targets`	Yes	Mapping of target id to target definition. Must contain at least one target.
`thresholds`	No	Case-level gates: `min_pass_rate`, `max_failures`, and `per_target.<target>.min_pass_rate`. Omitted keys create no gate for that key.

Target fields:

Field	Required	Description
`label`	No	Display name shown in reports.
`config`	No	Adapter-specific metadata for the target. Use this as the default place for prompt IDs, rubric IDs, reference assets, confidence thresholds, or other target-specific settings. Defaults to an empty object. Values override matching keys from `workflow.targets`.
`sample_defaults`	No	Target-wide defaults for sample `field` and `compare`. These override case defaults.
`samples`	Yes	List of sample blocks. Empty lists are invalid unless `--allow-empty` is used.

Most evals should put adapter metadata directly under targets.<id>.config. workflow.targets is useful when an eval is generated from or synchronized with an app workflow manifest and workflow-owned metadata should stay separate from eval-owned samples, expectations, and per-case overrides. Each workflow target needs an id; label and extra metadata keys are allowed. Entries are matched by id, and their metadata keys other than id and label are merged into the adapter target config before targets.<id>.config is applied. A workflow label is used as the target’s display label when the target does not define one, and entries whose id matches no target are ignored:

workflow:
  targets:
  - id: step_1
    app_step_id: 123
    prompt_id: workflow.step_1
targets:
  step_1:
    config:
      confidence_threshold: 0.85
    samples:
    - at: 8.0
      expect: true

Sample block fields:

Field	Required	Description
`range`	Conditionally	Two-element `[start, end]` interval in seconds. Exactly one of `range` or `at` is required. The interval is half-open.
`at`	Conditionally	One timestamp or a list of timestamps in seconds. Exactly one of `range` or `at` is required. Lists are sorted during expansion.
`expect`	For non-ignored runnable samples	JSON-like expected value: `null`, boolean, finite number, string, array, or object with string keys. Omit it to create a draft sample for `seed`; explicit `null` is a labeled expectation. Ignored samples may omit it without becoming drafts.
`every_s`	No	Per-block range sampling interval. Defaults to `sampling.every_s` for the case, which defaults to `0.5`.
`field`	No	Dot-separated path to extract from the adapter observation before comparison. An omitted value inherits target or case `sample_defaults`; without a default, the whole observation is compared. Explicit `null` clears an inherited field.
`compare`	No	Comparison config with `mode` and optional `tolerance`. An omitted value inherits target or case `sample_defaults`; without a default, mode is inferred from `expect` and numeric tolerance is `0.0`. Explicit `null` clears an inherited comparison.
`comment`	No	Human-readable note retained with the expectation. It does not affect adapter calls or comparison.
`ignore`	No	Nonempty reason for ignoring this block. Ignored samples do not need `expect`; they are reported but are not decoded, sent to the adapter, seeded, or included in pass rates, failure counts, or quality gates.

Sample times must be finite and nonnegative. Ranges must have end greater than start. Overlapping or duplicate samples for the same target are invalid; overlap is checked on the declared at times and range intervals, so two blocks with overlapping ranges are rejected even when their expanded samples would not collide. Expansion is capped at 10,000 samples across all targets in one case; pathological ranges are rejected before their samples are materialized. Unknown keys anywhere in a case file are validation errors, so a misspelled field name fails fast instead of being silently ignored; only workflow.targets entries accept extra metadata keys. Use ignore for a known exceptional sample that should remain documented without affecting a run. An ignored at list or range ignores every expanded sample in that block; use a single at timestamp when only one sample is exceptional. Sample settings use the precedence sample block > target sample_defaults > case sample_defaults > built-in behavior. compare is inherited or replaced as one complete value rather than merged key by key, so an override never retains an unrelated tolerance from a broader scope. Only field and compare can be defaulted; expectations, locations, comments, and ignore reasons remain explicit sample-block properties. For example, these defaults apply a structured result envelope and subset comparison to every target in the case, while the confidence target replaces both settings:

sample_defaults:
  field: result
  compare:
    mode: json_subset
targets:
  object_detection:
    samples:
    - range: [180.0, 182.0]
      expect:
        object: coffee_mug
        color: red
  confidence:
    sample_defaults:
      field: result.confidence
      compare:
        mode: numeric
        tolerance: 0.05
    samples:
    - at: 182.0
      expect: 0.9

Comparison reference

The adapter observation and the sample expect value must both be JSON-like. For simple checks, return only the value you want compared and omit field. Use field when the adapter naturally returns a structured result but only one nested value should determine correctness. For example, an adapter can return its result alongside diagnostic metadata; selecting the result with field makes it the seeded and compared value while preserving the complete adapter response in machine-readable reports and saved failure artifacts. Field paths are dot-separated. Mapping keys are matched by name, and list indexes can be addressed with nonnegative numeric path parts such as detections.0.label. Missing fields fail the sample with an adapter observation is missing configured field: ... reason. Supported comparison modes:

Mode	Description
`exact`	Observed value must equal `expect`. Booleans only match booleans.
`numeric`	Observed and expected values must be numbers. `tolerance` defaults to `0.0`.
`json_subset`	Every expected key and value must be present in the observed object. For arrays, expected items are matched one-for-one against observed items, so duplicate expected items require duplicate observed matches.
`set_equals`	Observed and expected arrays are compared as unordered JSON sets.
`set_contains_any`	At least one expected array item must be present in the observed array.
`set_contains_all`	Every expected array item must be present in the observed array.

Default comparison modes are inferred from expect: booleans, strings, and null use exact; numbers use numeric; arrays and objects use exact. Example:

targets:
  detector:
    samples:
    - at: 2.0
      field: result.matches
      expect: true
    - at: 3.0
      field: result.confidence
      expect: 0.8
      compare:
        mode: numeric
        tolerance: 0.05
    - at: 4.0
      field: detected_classes
      expect:
      - bracket
      - fastener
      compare:
        mode: set_contains_all

Get started

Guides and reference

Cases, videos, and comparisons

Eval directory layout

Cloud-stored videos

Public downloads

Case file reference

Comparison reference

​Eval directory layout

​Cloud-stored videos

​Public downloads

​Case file reference

​Comparison reference

Eval directory layout

Cloud-stored videos

Public downloads

Case file reference

Comparison reference