> ## Documentation Index > Fetch the complete documentation index at: https://glasskit.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Adapter reference > Connect Python or language-neutral command adapters to GlassKit Eval. GlassKit Eval supports in-process Python adapters and language-neutral command adapters. Both expose the same individual or batch evaluation behavior to the runner. Use a Python adapter when the app logic is importable by Python, or `--adapter-command` when the adapter should run in its own process, such as a JavaScript or TypeScript backend. ## Python adapters By default, `glasskit eval seed` and `glasskit eval run` load `/adapter.py:create_evaluator`. With the default eval directory, that is `eval/adapter.py:create_evaluator`. Use `--adapter :` to choose another adapter target. The module side can be an import path such as `my_app.eval_adapter` or a file path such as `eval/adapter.py`. The callable side can name a function, class, or nested attribute such as `create_evaluator` or `EvalAdapters.step_checker`. Keep the adapter thin by reusing as much of the app's runtime logic as practical and adding only the wrappers needed for recorded-video evaluation. The recommended adapter shape is a factory that accepts one config argument and returns an evaluator object: ```python theme={null} from __future__ import annotations import os from typing import Any def create_evaluator(config: Any) -> "Evaluator": settings = dict(config.config) return Evaluator( api_key=os.environ["MODEL_API_KEY"], model=settings.get("model", "default-model"), verbose=bool(config.verbose), ) class Evaluator: def __init__(self, *, api_key: str, model: str, verbose: bool) -> None: self._api_key = api_key self._model = model self._verbose = verbose async def evaluate(self, sample: Any, target: Any) -> bool: return await call_model_backend( api_key=self._api_key, model=self._model, image=sample.image, prompt_id=target.config.get("prompt_id", target.id), timestamp_s=sample.timestamp_s, ) async def close(self) -> None: await close_model_client() ``` Adapter factories may be synchronous or asynchronous. No-argument factories are supported, but they do not receive the factory config object. If the factory needs `--adapter-config`, `--artifacts-dir`, `--verbose`, or the eval directory, define it with one required argument. ## Individual and batch evaluation An evaluator chooses one of two execution strategies by implementing `evaluate` or `evaluate_many`. Both methods may be synchronous or asynchronous. | Strategy | Adapter method | GlassKit Eval execution | Use when | | ---------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | Individual | `evaluate(sample, target)` | Calls the method once per sample, with at most `--concurrency` calls in flight for the current target. | Each sample maps to an independent request or local operation. This is the recommended default for ordinary model APIs. | | Batch | `evaluate_many(samples, target)` | Calls the method once per target with that target's selected decoded samples. GlassKit Eval does not schedule the samples inside the batch. | The provider has a real multi-input endpoint, or the adapter can materially reuse work across the target's samples. | Implement at least one strategy. If an evaluator implements both methods, `evaluate_many` takes precedence. Batch evaluation must return exactly one JSON-like observation per input sample in the same order. A batch adapter owns any chunking or internal concurrency it needs; `--concurrency` does not fan out calls inside `evaluate_many`. Samples with an `ignore` reason are omitted before either strategy runs. They are not decoded and are not present in the `samples` list passed to `evaluate_many`. GlassKit Eval schedules the remaining samples in case-file declaration order and passes batch samples in that order. During `seed` and resumed runs, only samples that still need work are passed, so a batch adapter must not assume it always receives a target's complete sample set. Prefer `evaluate` when the work consists of independent calls, even if those calls should overlap. GlassKit Eval bounds synchronous and asynchronous calls by `--concurrency` and restores deterministic sample order after calls finish. With `--keep-going`, an individual call failure becomes an error only for that sample. Use `evaluate_many` only for actual batch behavior. If a batch call fails, GlassKit Eval cannot attribute the failure to one input, so `--keep-going` records an error for every sample in that target batch. The optional `close()` method is called after the run or adapter validation check and may also be synchronous or asynchronous. With `--repeat`, GlassKit Eval creates fresh evaluator instances sequentially and closes each trial before calling the evaluator factory for the next one. Simple function adapters are also supported when the first two positional argument names are either `image, target_id` or `sample, target`: ```python theme={null} def evaluate_frame(image, target_id): return target_id == "step_1" ``` Factory `config` fields: | Field | Description | | --------------- | --------------------------------------------------------------------------------------------------------- | | `eval_dir` | Resolved eval directory path. | | `config` | Mapping loaded from the discovered `adapter.yaml` or an explicit `--adapter-config`, or an empty mapping. | | `artifacts_dir` | Path from `--artifacts-dir`, or `None`. | | `verbose` | Boolean from `--verbose`. | Sample fields passed to the evaluator: | Field | Description | | -------------- | ---------------------------------------------------------------------------------------------------- | | `image` | Display-oriented RGB `PIL.Image.Image` for the nearest decoded frame at the requested timestamp. | | `timestamp_s` | Requested sample timestamp in seconds from the start of the clip, from `at` or the expanded `range`. | | `frame_index` | Zero-based decoded video frame index chosen for that timestamp. | | `sample_index` | Case-local sample index. | | `video_path` | Local video file path as a string. For cloud-stored videos this is the downloaded cache file. | | `case_name` | Case filename stem. | Frame sampling is timestamp-based. `sample.timestamp_s` is always the requested eval time, not the actual media timestamp of the selected frame. `sample.image` is the decoded frame whose timestamp is closest to that requested time, with ties choosing the earlier frame. GlassKit applies the source video's display rotation and reflection before handing the frame to an adapter, so its pixels and dimensions match normal video playback. For variable-frame-rate videos, `glasskit eval` uses each frame's media timestamp when available; if a video lacks frame timestamps, it estimates them from the frame index and average frame rate. `sample.image` is closed when the evaluate call returns. Call `sample.image.copy()` if the adapter needs to keep the frame afterward. Target fields passed to the evaluator: | Field | Description | | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | Target id from the case file. | | `index` | Target's zero-based order in the case file. | | `label` | Optional target label. | | `config` | Adapter-specific target metadata from `targets..config`, plus matching metadata keys other than `id` and `label` from `workflow.targets`; `targets..config` wins on conflicts. | Adapter return values must be JSON-like: `None`, boolean, finite number, string, array, or object with string keys. ## Command adapters Use `--adapter-command` when the app is easier to call from its own runtime, such as a JavaScript or TypeScript backend: ```sh theme={null} glasskit eval run --adapter-command "node eval/adapter.js" ``` GlassKit Eval parses the command into an argument list, then starts it directly without a shell. Pipes, redirects, variable expansion, and command substitution are therefore unavailable. The command inherits the current working directory and environment, so it can import the app normally and read the same secrets and configuration. Start from the complete JavaScript file below. Its editable application section passes a factory to `runGlassKitAdapter`; the protocol function handles communication with GlassKit Eval. Stdout belongs to that function, so write application and dependency logs to stderr with `console.error()`. GlassKit Eval mirrors adapter stderr to its own stderr and quotes the most recent output in error messages. The factory runs once per eval trial and receives this context: | Field | Description | | -------------- | ------------------------------------------------------------------------------------------------------- | | `evalDir` | Absolute eval directory path. | | `config` | Object loaded from the discovered `adapter.yaml` or an explicit `--adapter-config`, or an empty object. | | `artifactsDir` | Absolute path from `--artifacts-dir`, or `null`. | | `verbose` | Boolean from `--verbose`. | Return an object with at least one evaluation method: | Method | Purpose | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `evaluate({sample, target, signal})` | Evaluate one sample. Use this for ordinary independent backend calls. | | `evaluateMany({samples, target, signal})` | Evaluate one target's samples as a real batch and return one observation per sample in the same order. If both evaluation methods exist, this one takes precedence. | | `close()` | Optional cleanup for app clients and other resources. | The individual and batch scheduling behavior is the same as described above. Multiple `evaluate` calls can be active at once according to `--concurrency`, so shared app clients and mutable state must support that. Pass the provided `AbortSignal` through to backend calls when possible. Throw an error to report a failed call; otherwise return a JSON-compatible observation. Command-adapter samples contain the same information as Python samples, using lower camel case for field names: | Field | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image` | `{mimeType, bytes, width, height}`, where `bytes` is a Node.js `Buffer` containing the display-oriented lossless PNG. On the wire this field arrives as `dataBase64`, a base64 string that the protocol function below decodes into `bytes` before calling the app. | | `timestampS` | Requested sample timestamp in seconds. | | `frameIndex` | Zero-based decoded video frame index selected for that timestamp. | | `sampleIndex` | Case-local sample index. | | `videoPath` | Local video file path as a string; for cloud-stored videos, the downloaded cache file. Do not decode it again; `image` is already the selected frame. | | `caseName` | Case filename stem. | Targets use the `id`, `index`, `label`, and `config` fields described above. GlassKit-owned fields use lower camel case; keys inside the user-provided factory and target `config` objects are preserved unchanged. `glasskit eval validate --adapter-command ...` constructs and closes the adapter without evaluating samples. After answering the final `close` request, the adapter process must exit promptly with status `0`; GlassKit Eval waits about five seconds before terminating the process and its children. Protocol messages are limited to 256 MiB in each direction, which bounds how many PNG frames one `evaluateMany` batch can carry. In this `eval/adapter.js`, replace `createAppClient` and its methods with thin calls into the app, then keep the marked protocol function unchanged. Application clients stay in the factory's closure, and supported methods are detected automatically. The example uses an ECMAScript module; use `.mjs` or set `"type": "module"` in the app's `package.json` when needed. For adapters in other languages, use the JavaScript implementation below as an executable protocol reference. ```js theme={null} // Application code: replace these calls with the app's imports and logic. await runGlassKitAdapter(async (context) => { const app = await createAppClient(context.config); return { async evaluate({ sample, target, signal }) { return await app.evaluateFrame({ image: sample.image.bytes, mimeType: sample.image.mimeType, promptId: target.config.promptId ?? target.id, timestampS: sample.timestampS, signal, }); }, // If the app has a real multi-input API, implement // evaluateMany({ samples, target, signal }) instead. async close() { await app.close(); }, }; }); // ---- GlassKit Eval protocol ---- async function runGlassKitAdapter(createEvaluator) { const { createInterface } = await import("node:readline"); const lines = createInterface({ input: process.stdin, crlfDelay: Infinity }); const active = new Map(); let evaluator; let closing = false; let outputTail = Promise.resolve(); function send(message) { const line = `${JSON.stringify(message)}\n`; outputTail = outputTail.then( () => new Promise((resolve, reject) => { process.stdout.write(line, "utf8", (error) => { if (error) reject(error); else resolve(); }); }), ); return outputTail; } function errorPayload(error) { return { message: error instanceof Error ? error.message : String(error), ...(error instanceof Error && error.stack ? { stack: error.stack } : {}), }; } function sampleForApp(sample) { const { dataBase64, ...image } = sample.image; return { ...sample, image: { ...image, bytes: Buffer.from(dataBase64, "base64") }, }; } async function respond(request, operation) { try { await send({ id: request.id, result: await operation() }); } catch (error) { await send({ id: request.id, error: errorPayload(error) }); } } async function initialize(request) { await respond(request, async () => { if (request.params.protocolVersion !== 1) { throw new Error( `unsupported protocol version: ${request.params.protocolVersion}`, ); } evaluator = await createEvaluator(request.params.config); const capabilities = { evaluate: typeof evaluator?.evaluate === "function", evaluateMany: typeof evaluator?.evaluateMany === "function", }; if (!capabilities.evaluate && !capabilities.evaluateMany) { throw new Error("adapter must implement evaluate or evaluateMany"); } return { protocolVersion: 1, capabilities }; }); } function startEvaluation(request) { const controller = new AbortController(); const operation = async () => { if (!evaluator) throw new Error("adapter is not initialized"); if (request.method === "evaluate") { return await evaluator.evaluate({ sample: sampleForApp(request.params.sample), target: request.params.target, signal: controller.signal, }); } return await evaluator.evaluateMany({ samples: request.params.samples.map(sampleForApp), target: request.params.target, signal: controller.signal, }); }; const promise = respond(request, operation); active.set(request.id, { controller, promise }); promise.then( () => active.delete(request.id), (error) => { active.delete(request.id); console.error("Could not write GlassKit Eval adapter response:", error); process.exitCode = 1; }, ); } async function closeEvaluator() { const currentEvaluator = evaluator; evaluator = undefined; if (typeof currentEvaluator?.close === "function") { await Promise.resolve(currentEvaluator.close()); } } async function close(request) { closing = true; await Promise.allSettled([...active.values()].map(({ promise }) => promise)); await respond(request, async () => { await closeEvaluator(); return null; }); lines.close(); process.stdin.pause(); } for await (const line of lines) { let request; try { request = JSON.parse(line); } catch (error) { console.error("Invalid GlassKit Eval protocol request:", error); process.exitCode = 1; break; } if (request.method === "cancel") { active.get(request.params.id)?.controller.abort(); } else if (request.method === "initialize") { await initialize(request); } else if ( request.method === "evaluate" || request.method === "evaluateMany" ) { startEvaluation(request); } else if (request.method === "close") { await close(request); } else { await send({ id: request.id, error: { message: `unknown method: ${request.method}` }, }); } } if (!closing) { for (const { controller } of active.values()) controller.abort(); await Promise.allSettled([...active.values()].map(({ promise }) => promise)); await closeEvaluator(); } await outputTail; } ```