Serve a vision model

This guide stands up a vision model and points Kasm Session Runtime at it.

A model is optional. Scripted steps, browser control over the DOM, accessibility clicks and an outside agent driving the session over MCP all run without one. A model judges screenshots (observe:, expect:), locates things on screen (click_grounded, screen.ground) and runs the runtime’s own agent loop (goal: scenarios). See When you need a model.

The runtime talks to one endpoint, KASM_RUNTIME_VISION_ENDPOINT, in the OpenAI chat-completions format, with screenshots sent as base64 image_url parts. Any server that speaks that format works, local or hosted. The SDK’s 00-model-server example uses llama.cpp’s llama-server with one of two models the project has measured, and this guide follows it.

Before you start

• A host the session containers can reach over the network. 127.0.0.1 inside a session means the session container itself.
• Docker, or a llama-server binary built recently enough to know the model’s vision architecture. An older build loads the weights and then fails on the vision tower.
• Disk: about 5 GB for qwen3-vl-4b, about 7 GB for gemma-4-12b.
• For gemma-4-12b: an NVIDIA GPU passed through to Docker with the NVIDIA container toolkit. Docker Desktop on macOS cannot pass a GPU to a container; build llama-server with Metal instead.

1. Choose a model

Model	Hardware	Grounding, measured	Capture geometry	Answer convention
qwen3-vl-4b (Q8_0)	CPU	1.000 in the box, 3.0 px median error	native	point
gemma-4-12b (UD-Q4_K_XL)	GPU required	0.978 in the box, 3.4 px median error	square-padded	box2d

Grounding was measured by the project in August 2026 on 90 labelled UI elements, each answer scored as a hit if it fell inside the element’s box.

• qwen3-vl-4b is the only CPU model measured that grounds reliably. Use it for judging and grounding on a CPU host.
• Use gemma-4-12b for the agent loop. On a CPU a dense 12B model measured about 43 seconds per call.

A different model can work, but its grounding accuracy has not been measured against the runtime.

2. Get the weights

Each model needs two files: the quantised weights and the mmproj vision projector. Without the projector the model is text-only and every judgement fails.

mkdir -p ~/models/qwen3-vl-4b && cd ~/models/qwen3-vl-4b
curl -fSL -O https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct-GGUF/resolve/main/Qwen3VL-4B-Instruct-Q8_0.gguf
curl -fSL -O https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct-GGUF/resolve/main/mmproj-Qwen3VL-4B-Instruct-F16.gguf

For gemma-4-12b the files are gemma-4-12b-it-UD-Q4_K_XL.gguf and mmproj-F16.gguf from unsloth/gemma-4-12b-it-GGUF.

3. Start the server

Generate an API key. llama-server has no authentication beyond --api-key, and it listens on every interface so that session containers can reach it.

API_KEY=$(od -An -N24 -tx1 /dev/urandom | tr -d ' \n')

Start it (CPU, qwen3-vl-4b):

docker run -d --init --name kasm-model-server --restart unless-stopped \
  -p 8001:8001 \
  -v ~/models/qwen3-vl-4b:/models:ro \
  ghcr.io/ggml-org/llama.cpp:server \
  --model /models/Qwen3VL-4B-Instruct-Q8_0.gguf \
  --mmproj /models/mmproj-Qwen3VL-4B-Instruct-F16.gguf \
  --alias qwen3-vl-4b \
  --host 0.0.0.0 --port 8001 --api-key "$API_KEY" \
  --ctx-size 16384 --jinja --cache-reuse 256

For gemma-4-12b, use the ghcr.io/ggml-org/llama.cpp:server-cuda image, add --gpus all before the image name, point --model and --mmproj at its files, set --alias gemma-4-12b, and append --n-gpu-layers 99 --flash-attn on.

Optional: the 00-model-server example’s setup.sh does steps 2 to 4 for you and writes the settings into the SDK’s settings.env:

MODEL=qwen3-vl-4b ./setup.sh     # or MODEL=gemma-4-12b

The runtime depends on these flags.

Flag	Why
--mmproj	The vision projector. Without it every judgement fails
--jinja	Turns on the chat template’s tool-calling path. Required for goal: scenarios
--alias	Makes /v1/models advertise the name the runtime sends
--host 0.0.0.0	The runtime connects from inside a container
--ctx-size 16384 or more	Screenshots are large and runs are multi-turn. An overflow returns HTTP 200 with empty content
--cache-reuse 256	Consecutive screenshots are often identical; re-judging an unchanged screen measured about 4.5 s against 14.0 s fresh

Loading takes a few minutes. Wait until /health answers:

curl -fsS -H "Authorization: Bearer $API_KEY" http://127.0.0.1:8001/health

4. Point the runtime at it

Set these where the runtime starts: in the workspace’s environment for the MCP service (see Enable the runtime on a workspace), or exported in the shell before kasm-session-runtime run.

export KASM_RUNTIME_VISION_ENDPOINT=http://<model host>:8001/v1
export KASM_RUNTIME_VISION_MODEL=qwen3-vl-4b
export KASM_RUNTIME_VISION_API_KEY=<the API key>
export KASM_RUNTIME_VISION_CAPTURE_GEOMETRY=native
export KASM_RUNTIME_VISION_ANSWER_CONVENTION=point

For gemma-4-12b:

export KASM_RUNTIME_VISION_MODEL=gemma-4-12b
export KASM_RUNTIME_VISION_CAPTURE_GEOMETRY=square-padded
export KASM_RUNTIME_VISION_ANSWER_CONVENTION=box2d

KASM_RUNTIME_VISION_API_KEY is sent as a bearer token and never logged or written to a report.

5. Match the vision settings to the model

Three settings describe how the model sees and answers. The runtime does not infer them from the model name. A wrong value produces no error; the runtime clicks in the wrong place or judges less accurately.

Setting	Values	Default	What it controls
KASM_RUNTIME_VISION_CAPTURE_GEOMETRY	native, square-padded	native	The pixels the model is shown. native sends the screen as captured; square-padded letterboxes it into a square with black bars and resamples nothing
KASM_RUNTIME_VISION_ANSWER_CONVENTION	point, box2d	point	How the model is asked to reply when locating something: one point, or a bounding box in [ymin, xmin, ymax, xmax] order
KASM_RUNTIME_VISION_GROUNDING_SCALE	a positive integer	1000	The coordinate grid the model answers in. The runtime asks for 0 to this number on each axis and converts back to pixels

• Geometry follows the model’s vision encoder. Encoders that take images at their own aspect ratio, such as Qwen3-VL, need native. Encoders that resample every image to a fixed square need square-padded, or they squash the screenshot and the vertical answers drift. For gemma-4-12b with point held constant, native measured 0.206 in the box and square-padded 0.967.
• Convention follows how the model was trained to give coordinates. Point-trained models such as Qwen3-VL use point; detection-trained models such as Gemma use box2d.
• Qwen3-VL and Gemma both answer in 0 to 1000. Some models use 0 to 999 or 0 to 1. Check the model card before changing the scale.

An unknown geometry or convention, or a scale that is not a positive integer, stops the runtime with a configuration error.

Use a hosted endpoint instead

Point the same variables at a hosted OpenAI-compatible API and set its key. Nothing else changes. For example, with OpenRouter:

export KASM_RUNTIME_VISION_ENDPOINT=https://openrouter.ai/api/v1
export KASM_RUNTIME_VISION_MODEL=qwen/qwen3-vl-235b-a22b-instruct
export KASM_RUNTIME_VISION_API_KEY=sk-or-v1-...

The grounding measurements above were made with Qwen3-VL and Gemma. Staying in the Qwen3-VL family carries that evidence over; another model’s grounding accuracy is unmeasured. goal: scenarios also need the API to support OpenAI tool calling; that has not been verified against OpenRouter. With a hosted endpoint, screenshots of the session leave your network.

Check it works

1. Check the server advertises the model name you set:

   curl -s -H "Authorization: Bearer $API_KEY" http://<model host>:8001/v1/models

2. Optional: run the 00-model-server example’s smoke.sh from your workstation. It sends a real screenshot through the judgement and grounding requests and prints pass or fail for each; tool calling is reported but does not fail it:

   KASM_RUNTIME_VISION_ENDPOINT=http://<model host>:8001/v1 \
   KASM_RUNTIME_VISION_MODEL=qwen3-vl-4b \
   KASM_RUNTIME_VISION_API_KEY=$API_KEY \
   ./smoke.sh

3. Run a scenario. The agent line in report.md reads deterministic_only=false, and report.json records the settings in use at agent.vision.capture_geometry, agent.vision.answer_convention and agent.vision.grounding_scale.

Troubleshooting

Symptom	Cause	Fix
Report notes “vision endpoint failed repeatedly – degraded to deterministic-only”	The endpoint kept failing during the run. A common cause is an address the session container cannot reach	Check the endpoint from inside the session. Use an address the container can reach; 127.0.0.1 is the container itself
goal: loop degrades with “model replied without a tool call”	The server has no tool-calling path	Start llama-server with --jinja
HTTP 200 with empty content, judgements come back empty	Context too small, or a reply cap too small for a thinking model	Raise --ctx-size to 16384 or more. The runtime’s judgement cap is max_tokens 2000 by default (KASM_RUNTIME_MAX_TOKENS_JUDGE); raise it for a model that thinks at length, and do not put a smaller cap in front of it
Every judgement fails	The mmproj projector is missing and the model is text-only	Pass --mmproj
Clicks land in the wrong place, no error	Geometry, convention or scale does not match the model	Set the pair from the table in step 1 and check the scale on the model card
HTTP 500: Context size has been exceeded with several sessions	The server’s slots share one context pool	Allow about 10k context per concurrent session, or start with -np 1 so requests queue
gemma-4-12b fails to start in Docker	No GPU visible to Docker	Install the NVIDIA container toolkit, or build llama-server with your GPU backend

Next

• Vision models
• Run it with a local model
• What the model can do