tendril0.0.4

overview

Eyes and hands on a desktop.

Tendril lets an agent see and drive a computer the way a person does: find a window, take a screenshot, read its controls, then type, click and scroll. It is one stateless Rust binary for Linux, macOS, Windows and Android. There is no daemon and no session, and every command stands alone.

01 · listtendril listWindows, displays and cameras, with ids and capabilities.

02 · capture--window ID captureA screenshot, plus the scale to map it back.

03 · act--window ID run '…'Text, keys, clicks, drags and scrolls, serialised by a lock.

04 · verify--window ID captureLook again before the next step.

The loop on a strip of filmOn the left, an agent, either through the command line or the MCP server. On the right, a strip of film with four frames. Frame one, list: discover windows, displays and cameras. Frame two, capture: take a screenshot and its coordinate scale. Frame three, act: run typed input, clicks and scrolls, holding the execution lock and restoring focus afterwards. Frame four, verify: capture again and compare. An arrow returns from frame four to frame two when the result is not yet right. Below the strip, the places a frame can come from: X11, Wayland, macOS, Windows, Android over ADB, any of them over SSH, and the Windows host from WSL. agent CLI or MCP stateless calls 01 list tendril list 02 capture … capture 03 act … run '…' 04 verify … capture lock held focus restored not right yet: capture again X11 Wayland macOS Windows Android SSH WSL
The loop.Each frame is a separate, stateless invocation; an agent repeats capture and act until the screen says it is done.

One turn of the loop

A form in a disposable X11 desktop, driven by one run call between two captures. These are the actual captures:

A form titled Trail log with an empty Title field, an empty Notes box and a Save button.
14Acapture, before
The same form with the title Ridge loop, the note Fog lifted at the saddle, and the status Saved: Ridge loop under the Save button.
15Acapture, after one run
tendril listplatform: Linux / X11targets:- Display 1 screen 1920x1080+0+0 scale=1/1 capture=true, input=true- Window 0xe00003 Chromium-browser 797x587+125+94 scale=1/1 capture=true, input=true title="Trail log" app=Chromium-browser… and two more windows: the helper's browser and shelltendril --window 0xe00003 capture -o before.pngtendril --window 0xe00003 run 'send("Ridge loop"),Tab,send("Fog lifted at the saddle."),lclick(72,354),wait(300ms)'tendril --window 0xe00003 capture -o after.png

The run's JSON envelope says what happened, including the parts a person would not see: it waited for no one in the lock queue, moved focus to the target to deliver input, and put focus and the pointer back afterwards.

"action_count": 5,"execution_lock": { "acquired": true, "queue_position_at_join": 1, "wait_ms": 0, "stale_ms": 30000, "timeout_ms": 60000 },"focus_transferred": true, "focus_restored": true, "pointer_restored": true

Why it is built this way

Stateless
No daemon, session or hidden store. An agent can call it from anywhere, in parallel with others, and a crash leaves nothing behind. Where a stored target would help, tendril alias writes a shell function instead.
Typed, everywhere
The command line and the MCP server share one typed surface and the same JSON envelopes, so an agent gets identical results either way.
Careful with a shared desktop
Input is serialised by a host-wide lock, focus is restored after each run, and inputs that look like commands are refused rather than typed into the focused window.
Honest about platforms
Where a backend is missing, results say so: a probe-only audio capture, a skipped focus restore or a fallback element tree is reported in the envelope, not hidden.
Anywhere
The same invocation runs on a remote host over SSH, on the Windows host from WSL, or against an Android phone over ADB.

Where to go