guide
Discover, capture, act, verify
Every Tendril workflow is the same four steps. Each is its own command, with its own JSON envelope, so an agent can stop, think and resume at any point.
Discover targets
tendril listplatform: Linux / X11permissions: 4targets:- Display 1 screen 1920x1080+0+0 scale=1/1 capture=true, input=true - Window 0x600003 Chromium-browser 1918x1053+1+22 scale=1/1 capture=true, input=true title="Tendril Smoke Browser" app=Chromium-browser - Window 0xe00003 Chromium-browser 797x587+125+94 scale=1/1 capture=true, input=true title="Trail log" app=Chromium-browser - Window 0x40000c XTerm 604x316+41+102 scale=1/1 capture=true, input=true title="Tendril Headless Shell" app=XTerm Targets are windows, displays and cameras. Each has an id, bounds, a scale factor and what it supports: capture, input and, for some, audio. With --json each target also carries its kind, title, application name and process id. Commands that act on a target take exactly one of --window ID or --display ID; capture also accepts --camera ID.
Capture
tendril --window 0xe00003 capture -o before.pngcapture target: Window 0xe00003platform: Linux / X11original: 797x587output: 512x377resized: truesource_to_output: 512/797 x 377/587output_to_source: 797/512 x 587/377format: PngBy default a capture fits a 512 × 512 box, keeping its aspect ratio, so screenshots do not overwhelm an agent's context. --max-width and --max-height raise the bound; capturing a single window rather than a display is usually better than a bigger image. --format png|jpeg and --compression 0–100 trade size for detail. With --json the image comes back as base64 in the envelope; with -o it is written to a file.
run always takes source coordinates. A capture reports the ratio to convert from what the agent saw.A camera works the same way: tendril --camera ID capture takes one frame, through AVFoundation on macOS, V4L2 on Linux or DirectShow on Windows.
Semantic elements
Pixels are not the only way in. list-elements reads the platform's accessibility tree (AX on macOS, AT-SPI on Linux, native Win32 controls on Windows) and returns buttons, fields and text with ids and bounds:
tendril --window ID list-elements --jsontendril --window ID run 'click(element-id),send("Ridge loop")' Element ids belong to one snapshot. They are stable while the interface is, and click(id), press(id) and element(id) list the elements again, find the match and act at the centre of its bounds. When there is no accessibility service, the result says so and falls back to the window tree:
tendril --window 0xe00003 list-elementsplatform: Linux / X11elements: 1notes: X11 AT-SPI element listing failed: org.a11y.Bus.GetAddress failed: … falling back to the X11 window tree.- 1: window<Chromium-browser/Chromium-browser> 797x587+125+94 actions=clickThe run language
tendril run takes one argument: plain text, a comma-separated sequence of actions, or an action list from MCP. All three become the same typed model.
tendril --window ID run 'hello world' # types the texttendril --window ID run 'hold(ctrl),c,release(ctrl),wait(250ms),send("done")' tendril --window ID run 'hover(320,240),wait(250ms)' tendril --window ID run 'drag(100,200,400,200)' tendril --window ID run 'scroll(220,420,8)' | Action | Does |
|---|---|
send("…") | Type text |
Return, Tab, c, … | Tap a key |
hold(ctrl), release(ctrl) | Press and release a modifier around other actions |
wait(250ms), wait(1s) | Pause; a bare number means milliseconds |
lclick(x,y), rclick, mclick | Click a button at source coordinates |
dblclick(x,y) | A calibrated double-click (alias doubleclick) |
move(x,y) | Move the pointer without clicking, for hovers and tooltips (alias hover) |
drag(x0,y0,x1,y1) | Press, move and release |
scroll(x,y,dy) | Wheel ticks under a point; positive scrolls down |
click(id), press(id) | Act on a semantic element |
Android adds its own verbs (launch, tap_text, scroll_until, assert_visible, back, notifications and more); see Platforms.
Guards
Some inputs are refused because typing them would almost certainly be a mistake:
- A single bare word that looks like an action, such as
Returnortype "hi", is rejected rather than typed into the focused window (see the error it returns). Usewait(1ms),Returnto press a key, orsend("Return")to type the word. - On X11,
Ctrl+Lfollowed by a URL into a browser window is refused: browsers decide where synthetic key chords land, and the URL can end up typed into the page. Click the address bar from a capture instead, then verify.
On X11, text that the keyboard map cannot type directly, such as most non-ASCII text, is pasted through a short-lived clipboard selection instead. Terminals get a terminal-compatible paste shortcut, and the envelope's notes say what was done.
Focus and the shared desktop
An agent often works on a desktop someone else is using. run therefore:
Waits its turn
A host-wide lock serialises every
run, from any process or MCP server (see Agents).Records where you were
On X11, the focused window and the pointer position.
Focuses the target and acts
So keys and clicks reach the requested window, not whatever was on top.
Puts things back
Focus and pointer are restored unless
--no-restore-focusis given. Where an adapter cannot restore reliably yet, the run still succeeds and reportsfocus_restored: falsewith a note.
Audio
tendril listen records WAV audio with PipeWire or PulseAudio on Linux and CoreAudio on macOS, writing a temporary file or --output PATH; the envelope reports where. Elsewhere it reports capability and permission state only, as probe_only.
Aliases instead of sessions
Tendril keeps no session, so it cannot remember a target. It can write a shell function that does:
eval "$(tendril --window 0xe00003 alias --name trail)"trail capture --jsontrail run 'send("hello")'--shell bash|zsh|fish|powershell chooses the syntax.