Skip to main content
GlassKit Eval turns recorded video into repeatable evals for apps that make decisions from images. Label the moments that matter, connect your app through a language-agnostic adapter, and rerun the same checks locally or in CI. The eval loop is not tied to glasses or a particular model provider. It works for robotics, camera automation, video-analysis pipelines, multimodal model features, and other systems that can evaluate a sampled frame and return a JSON-like observation. Use GlassKit Eval through the glasskit eval command group. This is its user manual; for contributor implementation notes, see AGENTS.md.
GlassKit Eval terminal output showing the run summary and per-target results

​
Why use this?

Vision-based apps often turn camera input into a structured decision: whether a workflow step is complete, which objects are present, what state a scene is in, or what action should happen next. Recreating those scenes by hand for every prompt, model, or app logic change is slow and makes regressions difficult to reproduce. With glasskit eval, you provide a recording, label expected outputs at selected moments, and replay the same checks whenever the app changes. The adapter boundary lets the eval exercise existing application logic regardless of its implementation language, while quality gates turn the results into a useful local or CI signal. GlassKit Eval is a good fit when your behavior can be tested from sampled video frames and expressed as JSON-like outputs. It is intentionally frame-oriented; apps that require continuous video, audio, or other sensor streams may need an adapter that reconstructs that context.

​
How it works

Every eval command is a view of one pipeline:
  1. A case file names a video and declares samples — single at timestamps or range blocks expanded every every_s seconds. Labeled samples include an expected JSON-like value; drafts omit expect, and ignored samples may omit it.
  2. glasskit eval run checks the eval structure, videos, and sample times, expands the schedule, and decodes the frame nearest each scheduled timestamp.
  3. Each frame goes to your adapter, which runs your app’s logic and returns a JSON-like observation.
  4. The CLI extracts the configured field from the observation, if any, and compares the value against the sample’s expect using the sample’s comparison settings, recording a pass or fail.
  5. Completed runs are printed as tables and optionally written as a JSON report, and any configured quality gates turn them into the run’s exit code — the CI signal.
The labeling commands work on the same pipeline: seed sends draft samples through the adapter and writes the results back as proposed expect values instead of comparing them, and review opens a browser UI for checking and editing labels against the video.