Adapter reference

GlassKit Eval supports in-process Python adapters and language-neutral command adapters. Both expose the same individual or batch evaluation behavior to the runner. Use a Python adapter when the app logic is importable by Python, or --adapter-command when the adapter should run in its own process, such as a JavaScript or TypeScript backend.

Python adapters

By default, glasskit eval seed and glasskit eval run load <eval-dir>/adapter.py:create_evaluator. With the default eval directory, that is eval/adapter.py:create_evaluator. Use --adapter <module-or-file>:<callable> to choose another adapter target. The module side can be an import path such as my_app.eval_adapter or a file path such as eval/adapter.py. The callable side can name a function, class, or nested attribute such as create_evaluator or EvalAdapters.step_checker. Keep the adapter thin by reusing as much of the app’s runtime logic as practical and adding only the wrappers needed for recorded-video evaluation. The recommended adapter shape is a factory that accepts one config argument and returns an evaluator object:

from __future__ import annotations

import os
from typing import Any


def create_evaluator(config: Any) -> "Evaluator":
    settings = dict(config.config)
    return Evaluator(
        api_key=os.environ["MODEL_API_KEY"],
        model=settings.get("model", "default-model"),
        verbose=bool(config.verbose),
    )


class Evaluator:
    def __init__(self, *, api_key: str, model: str, verbose: bool) -> None:
        self._api_key = api_key
        self._model = model
        self._verbose = verbose

    async def evaluate(self, sample: Any, target: Any) -> bool:
        return await call_model_backend(
            api_key=self._api_key,
            model=self._model,
            image=sample.image,
            prompt_id=target.config.get("prompt_id", target.id),
            timestamp_s=sample.timestamp_s,
        )

    async def close(self) -> None:
        await close_model_client()

Adapter factories may be synchronous or asynchronous. No-argument factories are supported, but they do not receive the factory config object. If the factory needs --adapter-config, --artifacts-dir, --verbose, or the eval directory, define it with one required argument.

Individual and batch evaluation

An evaluator chooses one of two execution strategies by implementing evaluate or evaluate_many. Both methods may be synchronous or asynchronous.

Strategy	Adapter method	GlassKit Eval execution	Use when
Individual	`evaluate(sample, target)`	Calls the method once per sample, with at most `--concurrency` calls in flight for the current target.	Each sample maps to an independent request or local operation. This is the recommended default for ordinary model APIs.
Batch	`evaluate_many(samples, target)`	Calls the method once per target with that target’s selected decoded samples. GlassKit Eval does not schedule the samples inside the batch.	The provider has a real multi-input endpoint, or the adapter can materially reuse work across the target’s samples.

Implement at least one strategy. If an evaluator implements both methods, evaluate_many takes precedence. Batch evaluation must return exactly one JSON-like observation per input sample in the same order. A batch adapter owns any chunking or internal concurrency it needs; --concurrency does not fan out calls inside evaluate_many. Samples with an ignore reason are omitted before either strategy runs. They are not decoded and are not present in the samples list passed to evaluate_many. GlassKit Eval schedules the remaining samples in case-file declaration order and passes batch samples in that order. During seed and resumed runs, only samples that still need work are passed, so a batch adapter must not assume it always receives a target’s complete sample set. Prefer evaluate when the work consists of independent calls, even if those calls should overlap. GlassKit Eval bounds synchronous and asynchronous calls by --concurrency and restores deterministic sample order after calls finish. With --keep-going, an individual call failure becomes an error only for that sample. Use evaluate_many only for actual batch behavior. If a batch call fails, GlassKit Eval cannot attribute the failure to one input, so --keep-going records an error for every sample in that target batch. The optional close() method is called after the run or adapter validation check and may also be synchronous or asynchronous. With --repeat, GlassKit Eval creates fresh evaluator instances sequentially and closes each trial before calling the evaluator factory for the next one. Simple function adapters are also supported when the first two positional argument names are either image, target_id or sample, target:

def evaluate_frame(image, target_id):
    return target_id == "step_1"

Factory config fields:

Field	Description
`eval_dir`	Resolved eval directory path.
`config`	Mapping loaded from the discovered `adapter.yaml` or an explicit `--adapter-config`, or an empty mapping.
`artifacts_dir`	Path from `--artifacts-dir`, or `None`.
`verbose`	Boolean from `--verbose`.

Sample fields passed to the evaluator:

Field	Description
`image`	Display-oriented RGB `PIL.Image.Image` for the nearest decoded frame at the requested timestamp.
`timestamp_s`	Requested sample timestamp in seconds from the start of the clip, from `at` or the expanded `range`.
`frame_index`	Zero-based decoded video frame index chosen for that timestamp.
`sample_index`	Case-local sample index.
`video_path`	Local video file path as a string. For cloud-stored videos this is the downloaded cache file.
`case_name`	Case filename stem.

Frame sampling is timestamp-based. sample.timestamp_s is always the requested eval time, not the actual media timestamp of the selected frame. sample.image is the decoded frame whose timestamp is closest to that requested time, with ties choosing the earlier frame. GlassKit applies the source video’s display rotation and reflection before handing the frame to an adapter, so its pixels and dimensions match normal video playback. For variable-frame-rate videos, glasskit eval uses each frame’s media timestamp when available; if a video lacks frame timestamps, it estimates them from the frame index and average frame rate. sample.image is closed when the evaluate call returns. Call sample.image.copy() if the adapter needs to keep the frame afterward. Target fields passed to the evaluator:

Field	Description
`id`	Target id from the case file.
`index`	Target’s zero-based order in the case file.
`label`	Optional target label.
`config`	Adapter-specific target metadata from `targets.<id>.config`, plus matching metadata keys other than `id` and `label` from `workflow.targets`; `targets.<id>.config` wins on conflicts.

Adapter return values must be JSON-like: None, boolean, finite number, string, array, or object with string keys.

Command adapters

Use --adapter-command when the app is easier to call from its own runtime, such as a JavaScript or TypeScript backend:

glasskit eval run --adapter-command "node eval/adapter.js"

GlassKit Eval parses the command into an argument list, then starts it directly without a shell. Pipes, redirects, variable expansion, and command substitution are therefore unavailable. The command inherits the current working directory and environment, so it can import the app normally and read the same secrets and configuration. Start from the complete JavaScript file below. Its editable application section passes a factory to runGlassKitAdapter; the protocol function handles communication with GlassKit Eval. Stdout belongs to that function, so write application and dependency logs to stderr with console.error(). GlassKit Eval mirrors adapter stderr to its own stderr and quotes the most recent output in error messages. The factory runs once per eval trial and receives this context:

Field	Description
`evalDir`	Absolute eval directory path.
`config`	Object loaded from the discovered `adapter.yaml` or an explicit `--adapter-config`, or an empty object.
`artifactsDir`	Absolute path from `--artifacts-dir`, or `null`.
`verbose`	Boolean from `--verbose`.

Return an object with at least one evaluation method:

Method	Purpose
`evaluate({sample, target, signal})`	Evaluate one sample. Use this for ordinary independent backend calls.
`evaluateMany({samples, target, signal})`	Evaluate one target’s samples as a real batch and return one observation per sample in the same order. If both evaluation methods exist, this one takes precedence.
`close()`	Optional cleanup for app clients and other resources.

The individual and batch scheduling behavior is the same as described above. Multiple evaluate calls can be active at once according to --concurrency, so shared app clients and mutable state must support that. Pass the provided AbortSignal through to backend calls when possible. Throw an error to report a failed call; otherwise return a JSON-compatible observation. Command-adapter samples contain the same information as Python samples, using lower camel case for field names:

Field	Description
`image`	`{mimeType, bytes, width, height}`, where `bytes` is a Node.js `Buffer` containing the display-oriented lossless PNG. On the wire this field arrives as `dataBase64`, a base64 string that the protocol function below decodes into `bytes` before calling the app.
`timestampS`	Requested sample timestamp in seconds.
`frameIndex`	Zero-based decoded video frame index selected for that timestamp.
`sampleIndex`	Case-local sample index.
`videoPath`	Local video file path as a string; for cloud-stored videos, the downloaded cache file. Do not decode it again; `image` is already the selected frame.
`caseName`	Case filename stem.

Targets use the id, index, label, and config fields described above. GlassKit-owned fields use lower camel case; keys inside the user-provided factory and target config objects are preserved unchanged. glasskit eval validate --adapter-command ... constructs and closes the adapter without evaluating samples. After answering the final close request, the adapter process must exit promptly with status 0; GlassKit Eval waits about five seconds before terminating the process and its children. Protocol messages are limited to 256 MiB in each direction, which bounds how many PNG frames one evaluateMany batch can carry. In this eval/adapter.js, replace createAppClient and its methods with thin calls into the app, then keep the marked protocol function unchanged. Application clients stay in the factory’s closure, and supported methods are detected automatically. The example uses an ECMAScript module; use .mjs or set "type": "module" in the app’s package.json when needed. For adapters in other languages, use the JavaScript implementation below as an executable protocol reference.

// Application code: replace these calls with the app's imports and logic.
await runGlassKitAdapter(async (context) => {
  const app = await createAppClient(context.config);

  return {
    async evaluate({ sample, target, signal }) {
      return await app.evaluateFrame({
        image: sample.image.bytes,
        mimeType: sample.image.mimeType,
        promptId: target.config.promptId ?? target.id,
        timestampS: sample.timestampS,
        signal,
      });
    },

    // If the app has a real multi-input API, implement
    // evaluateMany({ samples, target, signal }) instead.

    async close() {
      await app.close();
    },
  };
});

// ---- GlassKit Eval protocol ----
async function runGlassKitAdapter(createEvaluator) {
  const { createInterface } = await import("node:readline");
  const lines = createInterface({ input: process.stdin, crlfDelay: Infinity });
  const active = new Map();
  let evaluator;
  let closing = false;
  let outputTail = Promise.resolve();

  function send(message) {
    const line = `${JSON.stringify(message)}\n`;
    outputTail = outputTail.then(
      () =>
        new Promise((resolve, reject) => {
          process.stdout.write(line, "utf8", (error) => {
            if (error) reject(error);
            else resolve();
          });
        }),
    );
    return outputTail;
  }

  function errorPayload(error) {
    return {
      message: error instanceof Error ? error.message : String(error),
      ...(error instanceof Error && error.stack ? { stack: error.stack } : {}),
    };
  }

  function sampleForApp(sample) {
    const { dataBase64, ...image } = sample.image;
    return {
      ...sample,
      image: { ...image, bytes: Buffer.from(dataBase64, "base64") },
    };
  }

  async function respond(request, operation) {
    try {
      await send({ id: request.id, result: await operation() });
    } catch (error) {
      await send({ id: request.id, error: errorPayload(error) });
    }
  }

  async function initialize(request) {
    await respond(request, async () => {
      if (request.params.protocolVersion !== 1) {
        throw new Error(
          `unsupported protocol version: ${request.params.protocolVersion}`,
        );
      }
      evaluator = await createEvaluator(request.params.config);
      const capabilities = {
        evaluate: typeof evaluator?.evaluate === "function",
        evaluateMany: typeof evaluator?.evaluateMany === "function",
      };
      if (!capabilities.evaluate && !capabilities.evaluateMany) {
        throw new Error("adapter must implement evaluate or evaluateMany");
      }
      return { protocolVersion: 1, capabilities };
    });
  }

  function startEvaluation(request) {
    const controller = new AbortController();
    const operation = async () => {
      if (!evaluator) throw new Error("adapter is not initialized");
      if (request.method === "evaluate") {
        return await evaluator.evaluate({
          sample: sampleForApp(request.params.sample),
          target: request.params.target,
          signal: controller.signal,
        });
      }
      return await evaluator.evaluateMany({
        samples: request.params.samples.map(sampleForApp),
        target: request.params.target,
        signal: controller.signal,
      });
    };
    const promise = respond(request, operation);
    active.set(request.id, { controller, promise });
    promise.then(
      () => active.delete(request.id),
      (error) => {
        active.delete(request.id);
        console.error("Could not write GlassKit Eval adapter response:", error);
        process.exitCode = 1;
      },
    );
  }

  async function closeEvaluator() {
    const currentEvaluator = evaluator;
    evaluator = undefined;
    if (typeof currentEvaluator?.close === "function") {
      await Promise.resolve(currentEvaluator.close());
    }
  }

  async function close(request) {
    closing = true;
    await Promise.allSettled([...active.values()].map(({ promise }) => promise));
    await respond(request, async () => {
      await closeEvaluator();
      return null;
    });
    lines.close();
    process.stdin.pause();
  }

  for await (const line of lines) {
    let request;
    try {
      request = JSON.parse(line);
    } catch (error) {
      console.error("Invalid GlassKit Eval protocol request:", error);
      process.exitCode = 1;
      break;
    }
    if (request.method === "cancel") {
      active.get(request.params.id)?.controller.abort();
    } else if (request.method === "initialize") {
      await initialize(request);
    } else if (
      request.method === "evaluate" ||
      request.method === "evaluateMany"
    ) {
      startEvaluation(request);
    } else if (request.method === "close") {
      await close(request);
    } else {
      await send({
        id: request.id,
        error: { message: `unknown method: ${request.method}` },
      });
    }
  }

  if (!closing) {
    for (const { controller } of active.values()) controller.abort();
    await Promise.allSettled([...active.values()].map(({ promise }) => promise));
    await closeEvaluator();
  }
  await outputTail;
}

Get started

Guides and reference

Adapter reference

Python adapters

Individual and batch evaluation

Command adapters

​Python adapters

​Individual and batch evaluation

​Command adapters

Python adapters

Individual and batch evaluation

Command adapters