Vision models

A model is optional. Kasm Session Runtime runs scripted steps, browser and accessibility actions, and every deterministic check without one, and an outside agent driving the session needs none in the runtime. See When you need a model for what works without one.

Requests the runtime makes

One endpoint, KASM_RUNTIME_VISION_ENDPOINT, serves every model call. It can be any OpenAI-compatible chat-completions server, on your own hardware (for example llama.cpp’s llama-server) or hosted. KASM_RUNTIME_VISION_MODEL is the model name sent with each request, qwen3-vl-4b by default. A hosted API that needs a key gets it from KASM_RUNTIME_VISION_API_KEY, which is never logged or written to a report.

The runtime makes three kinds of request.

Job	What is sent	What comes back	Used by
Judging	A screenshot and a question	Whether the app rendered and shows an error, or whether an expectation holds, with a description	The post-launch check, observe:, expect:, screen postconditions, screen.judge, the loop’s observe_screen and check_expectation
Grounding	A screenshot and a description of an element	Where the element is, or that it is not there	click_grounded, screen.ground
Tool calling	Text only: the goal, the tools, and the judge’s descriptions of the screen	One tool call per turn	goal: scenarios

Judging and grounding need a model that accepts images. Grounding also needs one that places elements accurately, and most models we tried did not. The loop needs tool calling from the server. With llama.cpp that means starting llama-server with --jinja; for a server whose tool calling is poor or absent, KASM_RUNTIME_FUNCTION_CALLING=injected puts the tools in the prompt instead.

Screenshots of the session go to the endpoint, so a hosted model receives them and a model on your own hardware keeps them inside your network.

Capture geometry and answer convention

Two settings describe how a model sees the screen and how it answers a grounding question. Both describe the model rather than a scenario, and the runtime does not infer them from the model’s name.

Setting	Variable	Values	Default
Capture geometry	KASM_RUNTIME_VISION_CAPTURE_GEOMETRY	native sends the screenshot as captured. square-padded adds black bars to make it square, without resizing it	native
Answer convention	KASM_RUNTIME_VISION_ANSWER_CONVENTION	point asks for one point. box2d asks for a bounding box, y first, and clicks its centre	point
Grounding scale	KASM_RUNTIME_VISION_GROUNDING_SCALE	The range of the normalised coordinates the model answers in	1000

Choose them from the model card. A model whose image encoder takes images at their own shape wants native. One that resizes every image to a square squashes a 1024×768 screenshot vertically and loses accuracy on the y axis, unless the capture is padded to a square first. Models trained to output points want point; models trained for detection want box2d.

A wrong setting raises no error. The model still answers, and the runtime clicks in the wrong place. An unknown value is a fatal configuration error, and report.json records the values in force under agent.vision and on every grounding attempt.

A grounded coordinate is only meaningful at the screen size it was produced for. Our examples pin sessions at 1024×768, and report.json records the actual size as screen.

Models we have measured

We have measured two deployments end to end. The Qwen deployment uses the defaults; the Gemma deployment sets both settings:

export KASM_RUNTIME_VISION_MODEL=gemma-4-12b
export KASM_RUNTIME_VISION_CAPTURE_GEOMETRY=square-padded
export KASM_RUNTIME_VISION_ANSWER_CONVENTION=box2d

Model	Hardware	Geometry	Convention	Grounding, bench	What we use it for
Qwen3-VL 4B (qwen3-vl-4b)	CPU	native	point	100% inside the element, median error 3.0 px	Judging and grounding
Gemma 4 12B (gemma-4-12b)	GPU	square-padded	box2d	97.8% inside the element, median error 3.4 px	Judging, grounding and the agentic loop

How these were measured: a bench of 90 labelled UI elements, each answer scored as inside or outside the element’s bounding box, with the median distance from the target. Figures from August 2026.

The settings make a large difference for Gemma. With point held constant, native capture scored 20.6% inside the element (median error 45.0 px) against 96.7% (4.8 px) for square-padded. With square-padded held constant, box2d improved on point from 96.7% to 97.8%. Same bench and date as above.

We also checked grounding on a real session screen: three targets on a 1024×768 Chrome page, each grounded twice at temperature 0, with the true position computed from the page’s DOM. All six answers landed inside the element, median error 2.8 px, and the two trials gave identical answers. Measured 19 August 2026.

For the agentic loop, gemma-4-12b on one RTX 3090 completed 9 of 9 product categories of a PC build on a live public website, with no declared page targets, in one run on 3 September 2026. The loop took 266.7 seconds in total, over 61 model calls and 202,554 tokens as reported by the server. That figure comes from a single run. The loop needs Gemma on a GPU; on CPU a 12B model takes too long per call.

Some models ground badly with every combination of settings. Two small models we tested stayed at or below 10% inside the element across both geometries and every convention, and one of them did not represent vertical position at all.

Serving notes

• Thinking models spend tokens reasoning before they answer. If the reply cap is reached first, llama-server returns HTTP 200 with empty content. The runtime sends a cap of 2000 tokens for judgements and grounding by default (KASM_RUNTIME_MAX_TOKENS_JUDGE).
• If the endpoint fails repeatedly, the runtime stops calling it and finishes the run on deterministic checks, and the report says so. See When you need a model.
• For a model we have not measured, set the geometry and convention its model card implies, then measure it with a grounding audit before relying on grounded clicks. A scenario with grounding_audit: true scores the model’s answers against the control positions an application publishes over AT-SPI, and clicks nothing.

Next

• Serve a vision model
• How an action reaches the app
• Environment variables