Secrets and evidence

Every run leaves a record in KASM_RUNTIME_OUT: report.json, report.md, the live trace.jsonl, screenshots and log tails. This page covers which secrets are kept out of that record, what the record contains, and where secrets can still appear. None of it depends on a model, except where noted.

Redacted values

The runtime replaces each of these values with [REDACTED]:

Value	How it is registered
Any `KASM_RUNTIME_SECRET_` or `TESTBENCH_SECRET_` variable	Read from the environment at start
`VNC_PW`, `VNC_VIEW_ONLY_PW`	Read from the environment at start
`KASM_RUNTIME_SESSION_TOKEN`	Read from the environment at start
`KASM_RUNTIME_INBOX_TOKEN`	Added at start, in both `run` and `mcp`
`KASM_RUNTIME_VISION_API_KEY`	Added at start, in both `run` and `mcp`
Variables named in `KASM_RUNTIME_REDACT_ENV` (comma-separated names)	Read from the environment at start
Any value a scenario expands from a `$NAME` reference in a `type`, `cdp_fill` or `provides:` value	Added at the moment it is expanded

Put credentials in a KASM_RUNTIME_SECRET_* variable and reference it from the scenario as $NAME. A literal written into the scenario file is not registered with the redactor.

Files redaction covers

Redaction runs on everything the runtime writes as text:

report.json: every string field in the report is rewritten. Two fields are exempt, the tool-name list and the policy identifier, because they come from closed sets.
report.md, including the failure narrative and notes.
trace.jsonl and the live /events stream: every event is redacted before it is written.
The run log on stderr, and the fatal report a run writes when it cannot start.
Log tails in logs/*.tail and the excerpts quoted in report.md, line by line as they are collected.
Captured shell output, before it is cut to evidence.shell_output_chars. The cut point moves so that it never splits a registered secret and leaves a partial prefix.
Inbox messages, reply values and request text, at the moment they arrive, so the inbox file, the trace, the report and the model’s context all hold the same redacted text. A typed reply whose redacted form would break its declared schema is refused with 422 rather than stored altered.

Values typed with type_text, input.type, cdp_fill or dom.fill are recorded by length only (“typed 12 character(s)”), whether they came from a secret or a literal.

Limits:

Redaction replaces exact substrings. A secret that appears transformed (base64-encoded, URL-encoded, split across lines, or in different case) is not recognised.
A short secret value also replaces every other occurrence of the same characters in the output.
The report records the vision endpoint URL in full (agent.vision.endpoint). Put an API key in KASM_RUNTIME_VISION_API_KEY and keep it out of the URL.

Places redaction does not reach

Screenshots. A secret visible on screen when a capture is taken is in the saved image, and the image is sent to the model when one is configured. The runtime takes captures at action boundaries on its own, and there is no per-step switch. The only remedy is keeping credentials off the screen.
Text the runtime sends to the model other than shell output. Page text and element lists from the browser, and accessibility names, go to the model as the page presents them. They are redacted only when they are written to the record.
Keystrokes seen by input observation. With KASM_RUNTIME_OBSERVE_INPUT on, every key the session receives is written to the trace verbatim, from any source, except while the runtime itself is typing: those events carry no key or text. A password a person types in the viewer is recorded key by key and is not a registered secret. Turning it on forces a token on the observe routes.

Contents of the report

report.json (schema nix-testbench-report/1) and its companions record, per run:

Record	Where
Verdict, outcome and the reasons for them	`verdict`, `outcome`, `reasons`; `DONE` holds the verdict and is written last
Deterministic probe results: process, window, logs	`probes`
Each step: action, the backend that ran it (`cdp`, `atspi`, `xtest`, `vision`, `shell`, `rpc`, `none`), the click rung, the result, the duration and any postcondition checks	`steps[]`
Each grounded click: what was asked for, where the model put it, whether it clicked and the verification outcome	`grounding[]`
Each file transfer: path, byte count and SHA-256, never content	`transfers[]`
The agentic loop: iterations, retries by class, whether it degraded, the model’s verdict and summary, the tool list offered, token usage	`agentic`
The configuration in force: runtime version, whether the run was deterministic-only, model name, endpoint, capture geometry, answer convention, sampling, the operator file’s effective values, legacy variable names in use	`agent`
Escalation: reason, how long the pause lasted, process and window state before and after	`escalation`
Screen geometry the run used	`screen`
Screenshots at step boundaries, and `99-failure.png` on failure	`screenshots/`
The last lines of the app’s logs (200 by default)	`logs/*.tail`
Every event as it happened, including the model’s text each turn	`trace.jsonl`

Tool results, call arguments and the model’s text are cut to evidence.tool_result_chars (default 300 bytes) in the step table and the trace. The model sees the full text during the run; the record keeps the shortened version.

Limits of the report

The report is not signed. It is written by a process running as the session user, into a directory that user and anyone holding the session token (through shell) can modify. Treat it as the runtime’s account of the run, and protect it after collection if you need it unaltered.
The report records what the runtime did and saw. It is evidence you can use in a review; it does not make a deployment compliant with any standard.