Accessibility

This guide drives native applications, such as file dialogs, settings panels, and installers, by the names of their widgets. For when to use this mode and how it compares with the others, see Control modes.

1. Accessibility is on by default

Every box has accessibility unless it is launched without it. It starts the AT-SPI bus before the desktop, so applications publish their widgets.

You cannot enable it on a running box. An application joins the tree only if the bus was running before the application started. On a box with no accessibility, the widget tools fail with a message that says so.

Accessibility works on the X11 and Wayland images.

2. Read the tree

holm widget "$BOX" tree --depth 4
holm widget "$BOX" tree --app gimp --depth 6

Each node has:

A tree read stops at depth 4 unless you give --depth. A full tree can have thousands of nodes, so narrow it with --app.

3. Find a widget

holm widget "$BOX" find "Street"
holm widget "$BOX" find "Street" --role text
holm widget "$BOX" find "OK" --role "push button" --exact

A query matches:

• The widget's own name.
• A label next to the widget. A form field usually has no name of its own, so the search also finds the nearest actionable widget to the right of a label with that text on the same row, or below it in the same column, within 220 pixels.

The best match is first. An exact match ranks above a partial match, and a field ranks above the label that names it. Applications with an active window are searched first, so when two windows each have an OK button, the active one wins.

4. Fill a field

holm widget "$BOX" fill "Street" "12 Bishop Street" --role text

fill writes the value into the widget's text directly. It does not send key presses. If an application acts on each key press, focus the widget and type instead:

holm widget "$BOX" focus "Street" --role text
holm keyboard "$BOX" type "12 Bishop Street"

A widget that is not editable refuses fill, and the error names its role.

5. Press a button

holm widget "$BOX" press "OK" --role "push button"

press runs the widget's first action. Give --action to run a different one:

holm widget "$BOX" find "Remember me"
holm widget "$BOX" press "Remember me" --action NAME

Use a name from the actions list that find returns.

Toolkits name actions differently. GTK calls the main action of a button click, and Qt calls it Press. find lists the actions of each widget, and the --action match ignores case.

6. Choose between press and a real click

press sends no pointer event. This has two effects:

• It works on a widget that is covered by another window or scrolled out of view.
• An application that watches the pointer, such as for hover effects or drag and drop, sees nothing.

When the application must see a real click, use the widget's at point with a screen click:

holm widget "$BOX" find "Upload" --role "push button"
holm mouse "$BOX" click 640 412 left

A widget with no actions cannot be pressed. The error says so and tells you to click its at point.

The same steps in other interfaces

MCP

The widget tool takes op with find, tree, press, fill, or focus:

{ "box_id": "box_…", "op": "fill", "query": "Street", "role": "text", "value": "12 Bishop Street" }

See widget.

REST

POST /v1/boxes/{id}/screens/{screen}/desktop/node:

{ "op": "find", "node": { "query": "Street", "role": "text" }, "limit": 10 }
{ "op": "set", "node": { "query": "Street", "role": "text" }, "value": "12 Bishop Street" }
{ "op": "invoke", "node": { "query": "OK", "exact": true }, "action": "click" }

The response has nodes for tree and find, and node for the other operations. In a batch, use the on_node action with the same body under what. See Native widgets.

Rust

let computer = Computer::builder().launch().await?;
let screen = computer.primary();

let street = NodeQuery {
    query: "Street".to_string(),
    role: Some("text".to_string()),
    ..NodeQuery::default()
};
screen.set_node(&street, "12 Bishop Street").await?;

let ok = NodeQuery {
    query: "OK".to_string(),
    exact: true,
    ..NodeQuery::default()
};
screen.invoke_node(&ok, None).await?;

if let Some(at) = screen.find_nodes(&street, Some(1)).await?.first().and_then(|node| node.at) {
    screen.click(at, Button::Left).await?;
}

During a takeover

While a person has exclusive control, focus, fill, and press are refused. tree and find continue to work. See Human control.

Troubleshooting