Report

Every run of Kasm Session Runtime writes a record of what it did, which mechanism ran each step, what it saw and the verdict to its output directory, KASM_RUNTIME_OUT (default /testbench/out). This page lists every file in that directory and every field of report.json. report.json is written for programs and report.md for people, and an orchestrator waits for DONE. Secrets and evidence says what the record never contains.

The output directory

For a single scenario (the no-argument invocation) and for mcp:

Path	What it holds
report.json	The full record, schema nix-testbench-report/1. Fields below
report.md	The same record for reading: verdict and reasons first
trace.jsonl	One JSON event per line, written as the run goes. The sidebar’s live view is a copy of it. See Sidebar protocol
screenshots/NN-<label>.jpg	Screenshots in the order taken, JPEG at evidence.jpeg_quality (85 by default)
screenshots/99-failure.png	The screen at the moment of failure, when there was one
logs/<path>.tail	The last evidence.log_tail_lines (200 by default) lines of /tmp/container-init-trace.jsonl and ~/.xsession-errors, named after the source path, secrets redacted
baseline-candidate/candidate-<YYYY-MM-DD>.png	Written when KASM_RUNTIME_BASELINE had no good-* image, for a person to promote
inbox/<run id>/messages.jsonl and attachments/	The human-in-the-loop journal and uploads, when the run had an inbox
DONE	Written last. Holds the verdict followed by a newline

For run, each scenario gets the layout above in its own directory, $KASM_RUNTIME_OUT/<name>/, and the top level holds two more files:

Path	What it holds
summary.json	schema, the worst-of verdict, and scenarios: one entry per file with name, file, verdict, outcome, and error when the scenario never ran
DONE	Written after summary.json. Holds the worst-of verdict

DONE

An orchestrator waits for DONE. The runtime writes it after the trace is closed and report.json and report.md are written, so once it exists the record is complete.

• Any DONE left from an earlier run in the same directory is deleted before anything else happens, including when configuration fails to load.
• If the trace or the report cannot be written, DONE is withheld and the exit code is 2.
• A configuration or scenario error before the run starts still produces a test-error report and a DONE.
• Under run, the top-level DONE is withheld if any scenario’s own report, DONE and exit code disagree, or its error report could not be written.

Verdict and outcome

verdict	Meaning	Exit code
pass	The app launched and stayed up, and every step and check passed	0
pass-with-concerns	As pass, with concerns recorded in reasons, such as baseline drift, an unavailable judgement, or an agent loop that ended without verifying its goal	0
app-failed	The app did not launch, crashed, showed an error dialog, or a step or check failed	1
test-error	The runtime could not reach a verdict: a configuration or scenario error, a failed screenshot, an escalation, or an interruption	2

outcome equals verdict, except under test-error, where it can be escalated (the model called escalate) or interrupted (a signal stopped the run).

The deterministic probes decide whether the app launched and stayed up. A model, when configured, judges observe, expect and screen checks and can fail them, but it cannot turn a failed probe into a pass, and its own goal-not-met is recorded as a concern. With no model the report says agent.deterministic_only: true, and judgements are recorded as not judged.

report.json

Fields marked optional are left out when they do not apply.

Top level

Field	Type	What it holds
schema	string	nix-testbench-report/1
app	string	The app under test
screen	string, optional	The captured screen size, WxH
verdict	string	See above
outcome	string	See above
reasons	array of strings	Why the verdict is what it is, including concerns
probes	object	boot, process, window: what each deterministic launch probe found, as text
atspi	object, optional	The accessibility probe. See below
cdp	object, optional	The browser connection: enabled, available, endpoint, browser, target_url, error
visual	object	What the screen showed. See below
steps	array	One row per step or tool call. See below
transfers	array, optional	File transfers: n, at (RFC 3339), action, ok, result (size and digest, or the failure). Never the content
grounding	array, optional	Every coordinate the model produced. See below
atspi_actions	array, optional	Every accessibility action. See below
grounding_audit	object, optional	A grounding_audit run’s measurement. See below
agentic	object	The agent loop. See below
timings	object	Durations in seconds: total_s, boot_s, launch_s, steps_s, plan_s, agentic_s, and model_s, the sum of every model call
agent	object	Which binary and configuration produced the report. See below
plan	object, optional	A plan: run: steps, completed, and when it fell back, failed_step (1-based) and reason
launch	object, optional	A launch: run: cmd, on_missing_window, window_missed, missed_reason, on_error_dialog, error_dialog_seen, error_dialog_reason, error_dialog_cleared
escalation	object, optional	When the model escalated: reason, paused_s, ended_by (operator when a person ended it), and before and after, each with process_alive and window_present
inbox	object, optional	The human-in-the-loop record: summary, messages, requests, actions, attachments. See People in the loop

steps[]

Field	What it holds
n	Row number
action	The step as written, or tool:<name>(<args>) for a tool call
backend	The mechanism that ran it: cdp, atspi, xtest, vision, shell, none, or rpc for a call made over MCP or HTTP
rung	For a click, where its coordinates came from: center, declared, literal, element or grounded
expect	The step’s expect, if any
result	What happened
duration_ms	Wall-clock time, always present; 0 means not measured
post	Each postcondition checked: kind, arg, held, detail, ms
url	The attached tab’s URL just after the row ran

visual

Field	What it holds
rendered, error_dialog, description	The model’s reading of the post-launch screen. With no model, description says so
error_dialog_title	An error dialog’s window title, found from X11 without a model. Wins over the model’s reading
error_dialog_text	The model’s description of an error dialog
baseline	compared, distance (perceptual-hash distance, or null), assessment (similar, drifted or very-different, with the distance and file, or why nothing was compared)
luminance	mean and stddev of a small greyscale thumbnail, computed without a model
render	rendered, blank, black or unknown, from the luminance

grounding[]

Field	What it holds
step	The steps row this belongs to
description	What the model was asked to find
found, x, y	Whether it answered, and the screen pixel it gave
space	How the answer was read: normalized or pixel
geometry, convention	The capture geometry and answer convention in force
clicked	Whether the runtime clicked there
verified	true only when the model confirmed the expected change
outcome	confirmed, contradicted, or unavailable when no judgement could be made
note	The model’s account, or why it failed

atspi and atspi_actions[]

Field	What it holds
atspi.status	ok, no-session-bus, no-a11y-bus, bus-alive-app-absent, error, or skipped-by-scenario
atspi.enabled_by_agent	The runtime switched accessibility on in this session
atspi.app_name, atspi.app_node_count	The matched application and how many nodes were walked
atspi.actionable	Actionable elements, each name and role
atspi.apps_seen, atspi.error, atspi.elapsed_ms	Other applications on the bus, any error, and the probe’s duration
atspi_actions[].step, request, matched, role	Which row, the name asked for, and the element it resolved to
atspi_actions[].method	submit-action (the app performed it), click-extents (clicked at the app’s published position, x and y), or none
atspi_actions[].ok, note	Whether it worked
atspi_actions[].verified, verify_note	An expect_state check’s result, when the step asked for one

agentic

Field	What it holds
enabled	Whether the agent loop ran
iterations	Model calls made
crash_retries, resolution_retries, tool_errors	Retries spent on malformed calls, on names that did not resolve, and consecutive tool failures at the end
context_recoveries	Times an oversized prompt was retried without the last tool result
degraded	The loop gave up before finishing
model_verdict, model_summary	The model’s own goal-met or goal-not-met and its summary. Advisory; the probes decide the verdict
compact_events	Each context compaction: step_index, phase, tokens_before, tokens_after, messages_before, messages_after
tool_set	Every tool the model could call, sorted
policy	When the scenario set a policy: source (skill or inline), skill_name, and hash of the rendered text
usage	Tokens reported by the model server, summed over every call: calls, prompt_tokens, completion_tokens, total_tokens. Left out when the server reports none

agent

Field	What it holds
version	The binary’s version
deterministic_only	true when no model was configured, or the model was given up on during the run
vision	When a model is configured: model, endpoint, capture_geometry, answer_convention, grounding_scale, function_calling, sampling, max_tokens, model_transport. Never the API key
legacy_env	TESTBENCH_* names the run was configured with
operator_config	Every operator-file block as resolved: config_file (the path, never the contents), runner, sampling, function_calling, context, max_tokens, model, baseline, findings, timing, evidence, tools, and transport

operator_config.transport records how the run could be reached: bind_addr, observe_token_required, observe_input, mcp_localhost_protection, cdp_allow_remote, config_strict, and redact_env_names (names only).

grounding_audit

Present only for grounding_audit: true. It records model, endpoint, geometry, convention, screen, app, describe, and per control in samples: element, role, described, the app’s rectangle (x, y, w, h), found, pred_x, pred_y, in_bbox, dist_px, latency_ms, note. The totals are attempted, found, in_bbox, median_px, p90_px, median_ms, the mean offset and spread (mean_dx, mean_dy, stdev_dx, stdev_dy), in_bbox_corrected (hits after removing the mean offset), systematic (a note when the offset looks like a frame-of-reference error), and skipped.

report.md

The heading names the app, and the verdict and reasons follow it. The remaining sections appear when they apply: Probes, Accessibility (AT-SPI) with the elements acted on, Plan, DOM backend (CDP), Human in the loop, Visual, the agentic transcript or the steps table (with each row’s backend, result, postconditions and time), Diagnosis, Log excerpts (the last evidence.report_log_lines lines of each log, 40 by default), and Timings & agent, which ends with run notes such as legacy variable names in use.

Next

• Secrets and evidence
• Command line
• Run a scenario and read the report