What the model can do

This page covers the runtime’s own agentic loop: a scenario with a goal:, where a model chooses the next tool call. It needs a model (KASM_RUNTIME_VISION_ENDPOINT). Scripted scenarios make no tool choices, and an outside agent driving the session over MCP or HTTP is not bound by these rules; see What the runtime can reach for that surface.

The approach comes from Forge (Zambelli), which reports that guardrails around tool calling close most of the reliability gap between small self-hosted models and frontier ones. The runtime took three of its layers: retry nudges (a malformed call gets its exact error back), step enforcement (a tool is blocked until its prerequisites ran) and error recovery (a failed call goes back to the model with the step left open). The runtime enforces each rule below before the tool runs.

Only declared tools

The model sees a list of tools built for the run. A tool is on the list only when the session can support it:

Tool	Present when
`observe_screen`, `press_key`, `click`, `type_text`, `wait`, `check_expectation`, `record_verdict`, `escalate`	Always
`click_element`	The application published named controls over AT-SPI at launch
`click_grounded`	A model endpoint is configured
`cdp_url`, `cdp_list_interactive`, `cdp_click_described`	A browser tab is attached over the Chrome DevTools Protocol
`cdp_wait`	The scenario declares `cdp_waits`
`cdp_query`, `cdp_click`, `cdp_fill`	The scenario declares `cdp_targets`
`wait_while`	The scenario declares `wait_conditions`
`respond`	The operator sets `runner.respond: true`
`ask_human`	The run has an inbox (a messaging or session token is set)
The scenario’s own tools	The scenario declares a `tools:` block

A call naming anything else is rejected with the list of available tools and counted as malformed. An operator can remove built-ins with tools.disabled in the operator file; the list is checked before the app launches, and an unknown name or one that would strand a dependent tool stops the run.

Limits:

The scenario decides the vocabulary. A declared tool with run: shell runs a literal command the author wrote for each allowed parameter value. The model picks the value; it cannot write the command. If the scenario declares no tools: block, the model has no shell.
tools.disabled affects built-ins only. It cannot remove a tool the scenario declared.

Closed parameters

Parameters are checked against each tool’s schema: required keys, types, enums and numeric bounds. A violation is a malformed call.

click takes center or a name from the scenario’s click_targets. The model never supplies a coordinate.
cdp_query, cdp_click and cdp_fill take a name from the scenario’s cdp_targets. The model never writes a CSS selector, and no built-in tool evaluates page script the model wrote.
type_text and cdp_fill take a ref, the key of a value the scenario provides:. The model never composes the text typed or filled, and never sees a secret’s value; the result reports only the number of characters.
A scenario-declared tool’s parameters are all enums. There is no free-string parameter kind.

Limits, where the model does supply free text:

press_key takes a key name or chord ("Return", "ctrl+l"): any key the session’s keyboard layout maps, with ctrl, alt, shift or super as modifiers. The model cannot type a string in one call, but it can press single letter keys, one call per key, with a fresh observe_screen before each. Disable press_key in the operator file where stray keystrokes are a risk.
cdp_list_interactive takes an optional text filter, and cdp_click_described takes a handle or an element’s text. The click goes to an element that exists on the page; the model still writes no selector or script.
check_expectation and click_grounded take a description and an expectation in the model’s words. That text goes to the vision model as a question. It never reaches an input device.
escalate, record_verdict and respond carry prose that ends up in the report, redacted like everything else the runtime writes.

Observe before a model-chosen click

press_key, click, type_text and click_grounded require observe_screen (or check_expectation) to have completed since the screen last changed. The runtime tracks this: every tool that changes the screen clears the record, so the next input action needs a new observation. A call without one is refused with a message naming the missing step.

click_element, the cdp_* tools and wait_while do not require a prior observation, because the target comes from the application or from the scenario. They still clear the observation record when they change the screen, and each step is screenshotted for the report.

Grounded clicks must verify

click_grounded is the one tool where a model locates something on screen. It exists only when a model endpoint is configured.

The model gives a description and the change it expects. It never sees pixel coordinates; the runtime asks the vision model to locate the description on a fresh capture and converts the answer to pixels.
After the click, the screen is judged against the expectation. The outcome is confirmed, contradicted or unavailable, recorded per attempt in report.json under grounding[].outcome. Only confirmed counts as verified.
In the loop, a contradicted click is reported to the model as “the expected change was NOT observed”, and the runtime does not click again on its own. Because the click changed the screen, the model must observe again before any further pointer action.
In a scripted click_grounded step, a contradicted click is re-located once, and the runtime refuses a second click if the new answer is the same coordinates.
An unavailable judge records the click as unverified, adds a concern, and tells the model the change is unverified. It does not re-click.
In a scripted scenario, a click_grounded step without expect: is rejected when the scenario loads.
Locating is bounded by runner.grounding_attempts (default 2) per call.

Budgets

Everything the model influences has a bound, and each bound ends in a defined state:

Bound	Default	Setting	When reached
Model calls in one loop	25	`runner.max_iterations`, or the scenario’s `agentic_max_iterations`	The loop ends; the report notes it ended without a model verdict
Malformed replies, whole run	3	`runner.max_retries_per_step`	The loop degrades; deterministic checks decide the verdict
Consecutive declared-tool failures	unbounded	`runner.max_tool_errors`	The loop degrades
Calls to one declared tool	set per tool	the tool’s `budget`	Further calls fail as a resolution error
Time in waiting tools, whole run	unbounded	`runner.max_hold_s`	The loop degrades
One `wait` call	30 s	`runner.wait_tool_max_s`	The schema refuses a larger value
Whole run	600 s	`KASM_RUNTIME_DEADLINE_S`	The last 30 s are reserved so diagnosis and the report always complete
Consecutive failed model calls	2	`model.disable_after`	The model is off for the rest of the run; judgements are recorded as not judged

Resolution failures (an element not found, a window not there) go back to the model with the step open. The deadline bounds them; they do not use up the malformed-reply allowance.

A scripted scenario may have at most 40 steps, and a plan: at most 40 steps.

The verdict

The runtime decides the verdict:

A run passes only if the deterministic checks pass: the process is alive, the application window is present, and no fatal log pattern matched.
The model’s record_verdict of goal-met never upgrades a failure.
A model’s goal-not-met on its own becomes a concern (pass-with-concerns). It becomes app-failed only when a deterministic failure already exists or a check_expectation the model ran during the loop came back false.
escalate locks the verdict to test-error with outcome escalated at the moment it is called, so later checks cannot change it.