How an action reaches the app

Kasm Session Runtime ranks the ways it can reach an element by where the target coordinate comes from. Scenarios should use the highest rung that reaches the element. The ranking is called the coordinate-trust ladder, and only the bottom rung needs a model.

The ladder

Rung	Where the target comes from	Scenario form	Needs a model
0. Page element	The browser’s DOM, over the Chrome DevTools Protocol. The action goes to the element itself, with no coordinate	cdp_click, cdp_fill and the other cdp_* actions	No
1. Declared position	A fraction of the window’s width and height, written by the scenario author with a screenshot in hand and reviewed like code	click center, click "<declared-target>", click <fx> <fy>	No
2. Accessibility	The application’s own widget tree over AT-SPI: the app names the control and performs the action	click_element "<name>"	No
3. Vision-grounded	The model looks at a fresh screenshot and says where the described element is	click_grounded "<description>" with a required expect:	Yes

Key chords sit beside rung 0. Where an application has a keyboard shortcut, key "<chord>" does not depend on layout, theme or screen size, and should be preferred to any click.

Coverage of each rung

Rung 0 works only in Chromium-family browsers (Chrome, Chromium, Brave, Edge, Vivaldi), and only inside a page. The browser’s own toolbar, extension popups and native file dialogs are not in any page’s DOM. The browser must be started with a debugging port and its own --user-data-dir. CDP is opt-in per scenario (cdp: true, or any cdp_* step). Firefox and Tor Browser are not supported at this rung.

Rung 1 reads the window’s geometry at the moment of the click, so a window that has moved or resized still gets the right pixel. A fraction outside 0 to 1 is rejected when the scenario loads. target: window "..." makes the fraction relative to another window, such as a native dialog.

Rung 2 works for GTK, Qt and Electron applications that publish their controls. When the control offers an action, the runtime asks the application to perform it, and no coordinate is involved. When it offers only a position, the runtime clicks the centre of the rectangle the application reports. That second path is less reliable: on LibreOffice we have seen reported positions about 24 pixels off. expect_state asks the application what state the control is in afterwards (Bold checked), without a screenshot or a model.

Rung 3 is the last resort. The runtime captures the screen, asks the model for the location of the description, clicks there, and asks the model to confirm the expected change. A step without expect: is rejected when the scenario loads. If the change is absent, the runtime grounds again, and clicks again only if the new location is different; it never repeats a click at the same place. If the model cannot be asked to confirm, the click is recorded as unverified with a concern. With no model configured, a scripted click_grounded step ends the run as test-error, and the agentic loop is not offered the tool. Accuracy depends on the model and its settings; see Vision models.

The model’s choices

In the runtime’s own agentic loop, the model never supplies a coordinate or a selector. It picks a declared target name, a CDP target name from the scenario, an accessibility name, or a page element from the list the runtime gave it (cdp_list_interactive), and for rung 3 it describes the element in words. The loop’s model does not see the screenshot; it reads the judge’s description of the screen, and the runtime makes the grounding call. See What the model can do.

An outside agent over MCP gets more: input.click takes a raw x and y, and screen.ground returns a location for the caller to click.

Contents of the report

Every step row in report.json records the backend that ran it: cdp, atspi, xtest, vision, shell or none. A click also records its rung (center, declared, literal, element, grounded) and the pixels it resolved to. Declared and grounded clicks are both delivered by xtest, and the rung says where to look when one misses: a wrong declared fraction means the scenario needs fixing, and a wrong grounded click means the model placed it wrongly. Runs that ground add a grounding block with what was asked for, where the model put it, and whether the click was verified.

Next

• Vision models
• Plans and postconditions
• Scenario YAML