Skip to main content

​
Installation

The Python package is glasskit.ai; it provides the glasskit console command. The package requires Python 3.12 or newer and is designed to run with uv. Add it to your app repo’s dev dependencies:
Or run it once without adding the dependency:
The uv run ... examples below assume the package has been added to your project. If you use the one-off form, replace uv run with uv run --with glasskit.ai.

​
Quickstart

Start in your app repository. Any recording that is at least a few seconds long works for this walkthrough. This example copies an MP4 recording into eval/cases/ so the case file can reference it by filename; Eval Directory Layout lists the supported formats. Create the eval directory and write a case file that points at the recording:
Create eval/adapter.py with a placeholder evaluator so you can verify that the eval wiring works before connecting a model pipeline:
Run the eval:
Expected result: run prints case progress, a summary, and a per-target table. Recordings do not have to live inside the repo; Eval Directory Layout shows how to reference a shared recordings/ directory, and Cloud-stored Videos covers recordings too large to keep locally.

​
Core concepts

An eval directory is a collection of draft or runnable cases. By default, glasskit eval uses eval/ in the current working directory. A case file is one YAML file under <eval-dir>/cases/. The case name is the filename stem. A video is declared by each case with video:. It can be a local path resolved relative to the case file or an object in a named cloud video store. A target is one thing the adapter should evaluate, such as step_1, ready_state, or detected_objects. A sample is one timestamp, or one timestamp expanded from a range. A runnable, non-ignored sample has an expected JSON-like value. A draft sample omits expect until glasskit eval seed proposes one or you add one manually; expect: null is a real expectation and is not a draft. Ignored samples may omit expect because they are not evaluated. An adapter is your Python bridge from the CLI to your app’s logic. The CLI decodes frames and calls the adapter; the adapter returns observations. A gate is a quality bar, such as a minimum pass rate or maximum failure count, that turns eval results into a pass/fail signal for CI. Because model-based checks may not always reach 100%, gates let you choose the right bar for your app.