Desktop input

This guide sends mouse and keyboard input to a box by screen coordinates, captures the screen, and uses the clipboard. It works with any application. On a web page, act on elements by name instead: see Control modes.

Coordinates

Coordinates are device pixels on one screen. (0, 0) is the top-left corner.

• Get a point from the most recent full-size screenshot.
• Do not calculate a point from a scaled or cropped capture.
• Take a new screenshot after a person, another process, or a new browser tab changes the screen.
• A screenshot does not show the pointer. Use cursor to get its position.

Mouse

button is left (default), right, or middle.

In Rust, move_along, drag_along, button_down, button_up, key_down, and key_up are methods of the Desktop trait. Import it with use holm::Desktop as _;.

holm mouse "$BOX" move 640 400
holm mouse "$BOX" click 640 400
holm mouse "$BOX" click 640 400 right
holm mouse "$BOX" click 640 400 left --double
holm mouse "$BOX" drag 100 100 400 300

computer.move_to((640, 400)).await?;
computer.click((640, 400), Button::Left).await?;
computer.double_click((640, 400), Button::Left).await?;
computer.drag((100, 100), (400, 300), Button::Left).await?;

A drag is one press, one move, and one release. A path is one press, a move through each point, and one release, so it draws one stroke with corners. A drag for each part of the line draws separate strokes.

holm mouse "$BOX" path 100 100 300 100 300 300 100 300

Modifier keys

Hold modifier keys during a click, drag, or path with --held. Pressing a key in a separate command does not work, because that key is released when its command ends.

holm mouse "$BOX" click 640 400 left --held shift
holm mouse "$BOX" drag 100 100 400 300 left --held ctrl

computer.click_with((640, 400), Button::Left, &[Held::Shift]).await?;
computer.drag_with((100, 100), (400, 300), Button::Left, &[Held::Ctrl]).await?;

MCP: held: ["shift"] on click or drag. The modifiers are shift, ctrl, alt, and super.

Motion

The pointer jumps to a point unless you ask for motion:

Use human for a page that watches pointer movement, or for a person who watches the screen.

holm mouse "$BOX" move 640 400 --smooth
holm mouse "$BOX" click 640 400 left --human --seed 42

MCP: motion: "human" and seed: 42 on click, drag, mouse_down, and the page tools.

use holm::{Desktop as _, Motion, motion};

let steps = motion::path(Point::new(100, 100), Point::new(400, 300), Motion::Human, 42);
computer.move_along(&steps).await?;

Hold a button

Use down and up only when a drag cannot do the task, for example when you must wait for something before you release:

holm mouse "$BOX" down 400 400 left --hold 10
holm mouse "$BOX" move 900 600 --smooth
holm mouse "$BOX" up 900 600 left

With no point, down and up act at the pointer. A move between them drags.

The server releases a button that stays down:

• After --hold seconds (hold_seconds in MCP, hold_ms in REST). Default 10 seconds, maximum 60.
• When a person takes control.
• At the end of a batch, unless the step gave a hold time. The result names each released button.

down and up need a server. They do not work with --local.

Keyboard

Typing goes to the element that has the keyboard focus. Click the field, or focus it, first.

holm keyboard "$BOX" type "hello"
holm keyboard "$BOX" press enter
holm keyboard "$BOX" press ctrl+l

computer.type_text("hello").await?;
computer.press("enter").await?;
computer.press("ctrl+l").await?;

Slow typing

Some inputs act on each key and lose characters that arrive at full speed. Add a delay between keys. 20 to 50 ms is usually enough.

holm keyboard "$BOX" type "one key at a time" --delay 30

computer.type_text("one key at a time").every(Duration::from_millis(30)).await?;

MCP: delay_ms: 30 on type_text. REST: delay_ms on a type action.

Key names

A key is a name or a combination: enter, tab, escape, up, pagedown, ctrl+a, cmd+shift+p. Common other names also work, such as esc, return, pgdn, cmd, and win.

Hold a modifier across keys

A combination releases its modifiers at the end. To keep a modifier down across several keys, give all the keys in one command with --held:

holm keyboard "$BOX" press tab tab tab --held alt

computer.press(["tab", "tab", "tab"]).holding([Held::Alt]).await?;

MCP: press_key with chord: "tab", then: ["tab", "tab"], and held: ["alt"].

This goes to the third window. Three separate alt+tab presses go only to the second.

Hold a key

down keeps one key down until up. Everything typed or clicked while it is down carries it:

holm keyboard "$BOX" down shift --hold 20
holm mouse "$BOX" click 300 200
holm mouse "$BOX" click 300 400
holm keyboard "$BOX" up shift

down takes one key, not a combination. The server releases the key at the same times as a button.

Scroll

scroll turns the mouse wheel at a point, in notches. It moves what is under the point, so it reaches a list or a sidebar without a click.

holm mouse "$BOX" scroll 640 400 down 3
holm mouse "$BOX" scroll down
holm mouse "$BOX" scroll 640 400 5 -2

With no point, the CLI scrolls at the center of the screen. With no count, it scrolls 3 notches. The signed form takes DY then DX: positive is down and right.

computer.scroll((640, 400), Delta::down(3)).await?;
computer.scroll((640, 400), Delta::right(3)).await?;

MCP: scroll with x, y, dy, and dx.

On a web page, scroll_page (MCP) moves the page by pixels and returns the new position. See Control modes.

Wait for the screen

After an action that draws, such as opening a menu, wait until the screen stops changing before you take a screenshot:

holm wait "$BOX" --settle 400 --within 10000

computer.wait_until_still(Duration::from_millis(400), Duration::from_secs(10)).await?;

MCP: wait_until_still. REST: a wait_still action.

The screen must not change for settle (default 400 ms). The wait stops at within (default 10 seconds). A screen with an animation reaches the limit. On a web page, wait_for is better: it waits for the element you need.

Screenshots

holm screenshot "$BOX" full.png
holm screenshot "$BOX" region.png --at 100,80 --size 400x300
holm screenshot "$BOX" window.png --window 42
holm screenshot "$BOX" small.png --scale 50

let full = computer.screenshot().await?;
let region = computer
    .capture(&Shot::region(Rect::new(Point::new(100, 80), 400, 300)))
    .await?;
let small = computer.capture(&Shot::of(Of::Screen).scaled(50)).await?;

Get window IDs from holm window <box> list or the MCP window tool with op: list. A window capture uses the window's position when the capture runs.

A scaled image uses fewer bytes, but do not calculate click points from it.

MCP tools that change the screen return a frame. Give have_frame with the last frame's hash: if the screen did not change, the result says unchanged and sends no image.

Clipboard

Each screen has two selections:

holm clip "$BOX" "ready to paste"
holm clip "$BOX"
holm clip "$BOX" "middle-click paste" --primary
holm clip "$BOX" --primary

computer.set_clipboard("ready to paste").await?;
let text = computer.clipboard().await?;
computer.set_selection(Selection::Primary, "middle-click paste").await?;

MCP: clipboard with text to set it, or without text to read it, and selection. REST: GET and PUT /v1/boxes/{id}/screens/{screen}/clipboard.

Setting the clipboard and pasting is faster than typing long text. From Rust, clipboard_bytes and set_clipboard_bytes move other types, such as image/png.

Batches

A batch runs several steps in one call. The box holds the screen for all the steps, so no other client can act between them, and one result comes back at the end. Use a batch when you know the steps in advance: a form, a drawing, or a key held across clicks.

CLI

holm batch reads a file, or standard input, with a list of REST actions:

[
  { "type": "click", "at": { "x": 200, "y": 150 } },
  { "type": "type", "text": "hello", "delay_ms": 20 },
  { "type": "press", "chord": "enter" },
  { "type": "wait_still", "settle_ms": 400 }
]

holm batch "$BOX" steps.json --settle 400

The file can also be an object with an actions list. --keep-going runs the remaining steps after a step is refused.

MCP

batch takes actions, each with a tool name and that tool's arguments:

{
  "box_id": "box_…",
  "actions": [
    { "tool": "mouse_down", "arguments": { "x": 100, "y": 100 } },
    { "tool": "move", "arguments": { "x": 200, "y": 150, "pause_ms": 40 } },
    { "tool": "move", "arguments": { "x": 300, "y": 100, "pause_ms": 40 } },
    { "tool": "mouse_up", "arguments": {} }
  ],
  "keep_going": true
}

move and draw are steps only in a batch. draw takes through, a list of {x, y}. pause_ms on move waits after the move. A drawing program needs about 40 ms to see each point.

REST

POST /v1/boxes/{id}/screens/{screen}/actions takes the same list as the CLI, under actions. See Actions for every action type.

Buttons and keys at the end

A button or key that is still down when the batch ends is released, and the result names it in released or released_keys. A step that gives a hold time keeps it down until that time, and the result lists it in holding or holding_keys.