Driving it
The loop is look, decide, act, look again. Everything the agent knows comes from a frame, and everything it does arrives as a keystroke or a pointer report.
Looking
A look returns a JPEG of the screen, or a region of it. Regions are the cheaper habit: a full screen is thousands of visual tokens and a cropped dialog is a fraction of that, so a run that crops where it can costs a fraction of one that does not.
Waiting is better than polling. Ask whether a region changed, with a pixel threshold, rather than fetching frames in a loop and comparing them by eye.
Acting
The HTTP surface is small. Every endpoint is a POST, takes JSON, and needs
Authorization: Bearer <key>. There is no localhost exemption, because
code running on the device is not authority to type on the target.
/v1/mouse- absolute position,0..32767per axis, origin top left, with the button mask./v1/wheel- one signed scroll report./v1/keys- a raw keyboard report, up to six usages. An empty body releases./v1/tap- press and release, played on the device, hold capped at 200 ms./v1/release- everything up. The pointer stays where it is./v1/screenshot- a frame, where the firmware offers one.
The Python client wraps these as move, click,
scroll, tap, type and release, with a
separate eyes object for screenshot, roi, changed
and wait.
Why typing has timing in it
A keystroke is a press report and a release report, and the gap between them decides whether the target sees a character at all.
- Held for less than one USB polling interval, the key is never collected: the press and the release land in the same window and only the release is seen.
- Held past the target's auto-repeat delay, roughly 250 ms on Windows, the character duplicates. Sent as two round trips from a loaded host, press and release have been measured 350 to 680 ms apart, which is squarely inside that.
So the tap happens on the device, where the hold is what was asked for regardless of what the caller is doing. Anything still held is released after five seconds of silence, because a stuck key repeats forever and nothing on screen says why.
Look before you act
Actions that move or type refuse to run unless a look returned a frame recently. Releasing a stuck key is exempt, since needing a screenshot to let go of a key would be backwards. This is a guardrail on the agent, not an access control: anyone holding the key bypasses it. What it buys is that an agent cannot act on an assumption about a screen it has not seen.
Release everything at the end of a run, and after any abort. A pointer button left down selects whatever the cursor passes afterwards.