Run it with a local model

This tutorial serves a vision model on your own hardware and gives Kasm Session Runtime a goal instead of steps. The runtime’s own loop reads the page, decides what to do and does it, and every model call goes to your server. Nothing about the session is sent to a model provider.

This is the one tutorial that needs a model. A goal: scenario runs the runtime’s agentic loop, and the loop asks the model to choose each tool call and to judge the screen. See When you need a model.

The run uses two examples in the SDK: 00-model-server to stand up the model, and 07-agentic-discovery for the task.

Before you start

You need:

A Kasm Workspaces deployment you can reach.
A Kasm API key and secret that can set environment variables when it requests a session (request_kasm), plus the username of the user it acts as, and the username and password of a real Kasm user for uploads and downloads. The settings are the same as in Run a scenario and read the report.
A Chrome workspace image. The example defaults to chrome:nix, which has the runtime built in.
uv, python3 and curl on your machine.
A machine for the model with an NVIDIA GPU. setup.sh runs the model server in a container and needs Docker with the NVIDIA container toolkit on that machine. The task was measured with gemma-4-12b on one RTX 3090. On a Mac, or Linux without the NVIDIA container runtime, run llama.cpp’s llama-server yourself with a GPU build instead.
A network path from the session container to the model machine, and from the session to the internet: the task uses pcpartpicker.com.

A CPU is not enough for this task. A dense 12B model measured about 43 seconds per call on CPU, and the loop makes a call per step. The CPU model the examples offer, qwen3-vl-4b, has measured results for judging screens and locating things on them, and none for running the loop.

Steps

1. Fill in the settings file

In the SDK’s examples directory, copy settings.env.example to settings.env and fill in the Kasm block, as in the first tutorial. Keep every line an export.

2. Start the model server

cd 00-model-server
MODEL=gemma-4-12b ./setup.sh

setup.sh fetches the model weights and the vision projector (mmproj) from Hugging Face into ~/.cache/kasm-session-runtime/models/, starts llama.cpp’s server in Docker on port 8001, waits for it to answer /health, and writes five variables into ../settings.env:

KASM_RUNTIME_VISION_ENDPOINT
KASM_RUNTIME_VISION_MODEL
KASM_RUNTIME_VISION_CAPTURE_GEOMETRY
KASM_RUNTIME_VISION_ANSWER_CONVENTION
KASM_RUNTIME_VISION_API_KEY

The server is published on every interface of the host so that a session container can reach it. The generated KASM_RUNTIME_VISION_API_KEY keeps other callers out; llama-server has no other authentication. Running the script again reuses the key. Loading the model can take a few minutes.

If you run llama-server yourself instead, the llama-server.sh script in the same folder starts it with the flags the runtime needs. Optional:

MODEL_GGUF=~/models/gemma-4-12b/gemma-4-12b-it-UD-Q4_K_XL.gguf \
MMPROJ=~/models/gemma-4-12b/mmproj-F16.gguf \
ALIAS=gemma-4-12b \
./llama-server.sh

Set API_KEY= yourself in that case. The loop depends on --jinja, which turns on the server’s tool-calling path; without it the model never makes a tool call and the loop degrades to a deterministic run.

3. Point the endpoint at an address the session can reach

setup.sh writes the endpoint as http://host.docker.internal:8001/v1. A session placed by Kasm runs on a Kasm agent host, where that name does not resolve. Change the line in settings.env to the model machine’s LAN address:

export KASM_RUNTIME_VISION_ENDPOINT=http://<host>:8001/v1

127.0.0.1 would mean the session container itself.

4. Check the endpoint

source ../settings.env
./smoke.sh

smoke.sh sends a synthetic screenshot and checks the three things the runtime asks of the model: a screen judgement, the location of a button, and a tool call. Each prints PASS or FAIL with its time.

If tools fails, the server is not running with --jinja, and the task will not work until it is.

5. Check the model settings

The runtime does not infer the two vision settings from the model name; they must match the model in use. For gemma-4-12b:

export KASM_RUNTIME_VISION_MODEL=gemma-4-12b
export KASM_RUNTIME_VISION_CAPTURE_GEOMETRY=square-padded
export KASM_RUNTIME_VISION_ANSWER_CONVENTION=box2d

setup.sh writes these for you. A mismatch raises no error; it lowers how accurately the model locates things on screen. Vision models explains the two settings.

6. Look at the scenario

Open 07-agentic-discovery/scenario.yaml:

app: chrome
window: "Chrom"
process: "chrome"
cdp: true

steps:
  - action: cdp_navigate "__CATEGORY_URL__"

goal: "This page lists products in one category (e.g. CPUs). Call cdp_list_interactive to see the products, dismiss any cookie banner if present, then click 'Add' on any one suitable product using cdp_click_described. Do not check whether other parts are already selected -- that is not needed. Don't ask follow-up questions; use your best judgement."

agentic_max_iterations: 20

One scripted step opens the category page, with the script filling in __CATEGORY_URL__ for each category; the loop has no navigate tool. The goal is one sentence and the scenario declares no selectors. The model reads the page with cdp_list_interactive and clicks with cdp_click_described, by a handle from that listing, so it can act only on elements the runtime found on the live page. agentic_max_iterations caps the loop at 20 turns.

7. Run one category

The full run covers nine pcpartpicker categories. Start with one, using PCPP_CATEGORIES:

cd ../07-agentic-discovery
PCPP_CATEGORIES="cpu|https://pcpartpicker.com/products/cpu/" ./run.sh

The script checks the endpoint’s health, requests a session, opens the viewer, runs the scenario with the runtime in the image, then downloads the output into out/ and destroys the session.

To run all nine, drop PCPP_CATEGORIES:

./run.sh

The session’s browser loads a public website. What stays local is the model: every screenshot and every prompt goes to your endpoint. Each category gets its own folder, out/<category>/, with its own report and DONE file, and the script logs each category’s verdict as it collects them.

8. Read the summary and the audit

The script prints one row per category. From the nine-category run on 2026-09-03, cpu read:

category         verdict              model verdict  iters  total_s
cpu              pass                 goal-met       6      36.3

verdict is the runtime’s verdict. model verdict is what the model reported when it called record_verdict. The model’s report is not evidence that the click landed, so the script runs audit_build.py, a scripted check outside the loop, which passes a category only if the report has a successful cdp_click_described step on a product. It also looks for each product on pcpartpicker’s parts list and prints FOUND or MISSING; that second check is advisory and does not decide the result.

9. Read a category’s report

Open out/cpu/report.md. Two sections differ from the first tutorial’s report.

The Visual section has a real description of the screen, because a model judged it.
An Agentic transcript section lists each tool call the model made, the backend that ran it, its result and how long it took, followed by the model’s verdict and summary.

From a real run:

#	Tool call	Backend	Result
1	`cdp_navigate "https://pcpartpicker.com/products/cpu/"`	cdp	loaded the page
2	`observe_screen({})`	vision	described the page, including a cookie banner
3	`cdp_list_interactive({"filter":"Allow"})`	cdp	`5bwzm [button] allow cookies`
4	`cdp_click_described({"description":"5bwzm"})`	cdp	clicked allow cookies (button)
5	`cdp_list_interactive({"filter":"Add"})`	cdp	101 of 134 elements, each led by a handle
6	`cdp_click_described({"description":"s9vqg"})`	cdp	clicked AMD Ryzen 7 9800X3D … Add (button)
7	`record_verdict(...)`	none	verdict recorded; run ends

The last section reads deterministic_only=false, and the timings split the run into scripted steps, the loop and time spent waiting on the model. report.json carries agentic.usage, the token counts your endpoint reported on each response, summed across the run; an endpoint that reports no usage leaves the field out.

Cost of the run

Measured on 2026-09-03, one run of all nine categories with gemma-4-12b on one RTX 3090 served by llama-server: 9 of 9 categories passed, with 61 model calls and 202,554 tokens, all local. The loop itself took about 4.4 minutes (266.7 seconds summed across categories); the whole run, with session start and navigation, about 6 minutes.

That is one run of nine samples and gives no reliability figure. During development, runs ranged between 8 and 9 of 9 with the failing category varying, and the site is live and can change under the task.

A hosted OpenAI-compatible endpoint also works in place of a local one, with no code change. The session’s screenshots then leave your network, and grounding accuracy has been measured only for the models on Vision models.

Serve a vision model: the model server, including the CPU option and its limits.
Plans and postconditions: reason once with the loop, then replay the task with no model.
What the model can do: the guardrails the loop runs inside.