Run a scenario and read the report

This tutorial runs a scripted login against a page in Chrome inside a Kasm session, then reads the report Kasm Session Runtime writes. No model is configured at any point. Every step is a DOM action over the Chrome DevTools Protocol (CDP), and the page itself answers each check.

Of the three tutorials this one has the least setup. It shows what the runtime does on its own: it runs the steps, checks them, records which backend ran each one and decides the verdict. When you need a model lists what a model adds.

The run uses the 01-cdp-chrome example in the SDK.

Before you start

You need:

A Kasm Workspaces deployment you can reach.
A Kasm API key and secret that can set environment variables when it requests a session (request_kasm), plus the username of the user it acts as. Without an API key, Kasm drops the environment the example sends, including the Chrome flags that turn CDP on.
The username and password of a real Kasm user. The example uploads files into the session and downloads the report, and those two endpoints accept only Kasm’s login cookie.
A Chrome workspace image. The example defaults to the chrome:nix image, which has the runtime built in.
uv and python3 on your PATH.
No vision model. The script unsets KASM_RUNTIME_VISION_ENDPOINT and KASM_RUNTIME_VISION_MODEL so the run stays model-free even if your shell has them exported.

Steps

1. Fill in the settings file

In the SDK’s examples directory, copy settings.env.example to settings.env and fill in the Kasm block. Every line must stay an export: the provisioning helper runs as a separate process and only sees exported variables.

export KASM_URL=https://kasm.example.com
export KASM_API_KEY=...
export KASM_API_KEY_SECRET=...
export KASM_SESSION_USER=...
export KASM_USERNAME=...
export KASM_PASSWORD=...
export KASM_VERIFY_TLS=true

KASM_VERIFY_TLS also takes false or a path to a CA bundle, for a deployment with a self-signed certificate.

If you have only a username and password, the helper can mint a scoped API key from an account that holds global admin rights. Optional:

uv run lib/kasm_provision.py bootstrap-api-key

Paste the key and secret it prints into settings.env.

2. Look at the scenario

Open 01-cdp-chrome/scenario.yaml. It names the app, turns on CDP and lists eight steps:

app: chrome
window: "Chrom"
process: "chrome"
cdp: true
launch_timeout_s: 90

steps:
  - action: cdp_navigate "http://127.0.0.1:8901/login.html"
  - action: cdp_query "#username"
  - action: cdp_fill "#username" "demo-user"
  - action: cdp_fill "#password" "demo-password"
  - action: cdp_click "button.login"
  - action: cdp_wait "location.hash === '#/dashboard'" 15
  - action: cdp_wait "document.getElementById('who').textContent === 'demo-user'" 10
  - action: cdp_url

The login page is a local fixture. The script uploads it and serves it over loopback HTTP inside the session; Kasm’s browser images block file:// URLs by policy.

The two cdp_wait steps are the assertions. Each is a JavaScript predicate that must become true within the timeout, or the step fails. They check location.hash because the fixture, like many single-page apps, never changes its window title.

The same folder has scenario-stale-selector.yaml: the same flow with the login button’s selector left as it was before a rename. The script runs both, one after the other, in the same session.

3. Run it

cd 01-cdp-chrome
./run.sh

The script requests a session through the Kasm API, opens the session’s viewer in your browser, uploads the fixture and both scenario files, runs them with the runtime in the image, then downloads the output into out/ and destroys the session.

If the image is a kasm-nix image, the viewer shows the runtime’s sidebar, and each step appears there as it runs.

The script exits with status 1. The second scenario is broken on purpose, and the combined verdict across both scenarios is the worse of the two.

To keep the session running afterwards so you can look around, set KEEP_SESSION=1. The script then prints the command that destroys it. Optional:

KEEP_SESSION=1 ./run.sh

4. Read the summary

Open out/summary.json:

{
  "schema": "nix-testbench-report/1",
  "verdict": "app-failed",
  "scenarios": [
    { "name": "scenario", "file": "scenario.yaml", "verdict": "pass", "outcome": "pass" },
    { "name": "scenario-stale-selector", "file": "scenario-stale-selector.yaml", "verdict": "app-failed", "outcome": "app-failed" }
  ]
}

One scenario passed and one failed, so the run as a whole is app-failed. The script’s exit code follows the same verdict: 0 for pass or pass-with-concerns, 1 for app-failed, 2 for test-error, and 3 if the script could not get far enough to run the runtime.

5. Read the passing report

Open out/scenario/report.md. It starts with the verdict and its reasons, then the evidence.

The steps table, from a real run:

#	Action	Backend	Result
1	`cdp_navigate "http://127.0.0.1:8901/login.html"`	cdp	loaded http://127.0.0.1:8901/login.html
2	`cdp_query "#username"`	cdp	found `<input>` id=“username” name=“username” visible=true enabled=true
3	`cdp_fill "#username" "demo-user"`	cdp	filled #username with 9 character(s)
4	`cdp_fill "#password" "demo-password"`	cdp	filled #password with 13 character(s)
5	`cdp_click "button.login"`	cdp	clicked `<button>` id=“login-button” text=“Log in” visible=true enabled=true
6	`cdp_wait "location.hash === '#/dashboard'" 15`	cdp	predicate holds
7	`cdp_wait "document.getElementById('who').textContent === 'demo-user'" 10`	cdp	predicate holds
8	`cdp_url`	cdp	location.href = http://127.0.0.1:8901/login.html#/dashboard

Every row names the backend that ran it, here always cdp. Other runs show atspi, xtest, vision, shell or none, and the column matters when a step fails: a DOM click and a vision-guided click fail for different reasons.

Rows 3 and 4 record how many characters were filled. cdp_fill values are never written to the report, whether or not they are secrets.

The actual report table also has an Expect column. It reads -- here because no step asks for a visual check.

6. The report without a model

With no endpoint configured, the runtime runs in deterministic-only mode, and the report records that in three places.

The Visual section reads rendered: false, error_dialog: false and description: no visual judgment (deterministic-only mode). rendered: false here means nothing was judged, and is not a finding against the app.
The last section reads deterministic_only=true and note: vision endpoint not configured -- no visual judgment.
The verdict comes from the probes and the steps. The Probes table shows the boot check, the process (alive through settle) and the window the runtime matched. Those and the steps decide pass.

The report also has an Accessibility (AT-SPI) section and a DOM backend (CDP) section, which say whether each backend was available and, for CDP, which browser and tab the runtime attached to. This scenario uses only CDP.

7. Read the failure

Open out/scenario-stale-selector/report.md. The first line gives the reason:

step 4 (cdp_click "button#submit-login") failed: no element matches "button#submit-login" after 5s; candidates on the page: input#username[name=username], input#password[name=password], button#login-button "Log in"

The verdict is app-failed. The page answered and the element was absent, which is evidence about the app. A lost CDP connection would be test-error, a fault in the test setup.

The reason lists the candidates on the page. button#login-button "Log in" among them shows that the page loaded and the selector is stale.

8. Look at the other files

Each scenario’s folder under out/ holds:

File	What it is
`report.json`	The full report, schema `nix-testbench-report/1`
`report.md`	The same report for people: verdict and reasons first
`screenshots/NN-<label>.jpg`	One after launch and one per step, for example `02-step-1-cdp-navigate.jpg`
`screenshots/99-failure.png`	Only in the failed scenario: the screen at the moment of failure
`baseline-candidate/candidate-<date>.png`	Saved because no baseline existed to compare against
`trace.jsonl`	One event per line, in order, with timestamps
`logs/`	Log tails, when there are any
`DONE`	Written last; contains the verdict

The screenshots let you check the report against what was on screen. Promote the baseline candidate to a baseline if you want later runs compared against it.

Limits of this run

Everything happens inside one page’s DOM. A browser toolbar, an extension popup or a native dialog has no selector, and needs a key chord, a declared click target, the accessibility tree or a vision-guided click. How an action reaches the app lists those in order.
It is a single run and gives no reliability figure.
The steps check the DOM. Whether the page looked right is a separate question, answered by an observe: step or an expect: check, and those need a model.

Report reference: every field in report.json and the output folder.
Write a scenario: turn your own task into steps like these.
Drive a session from Claude Code: the same session, driven by your own agent instead of a script.