What to expect · Agent Machine Interface

What it does well.

Any screen a person can operate. BIOS, firmware menus, GUI installers, UAC prompts, appliance web UIs, a machine that has not finished installing an operating system yet.

Real keyboard behaviour. Chords, hold, ctrl-drag and shift-click, because reports go straight to the USB gadget rather than through a tap-one-key-at-a-time control API.

Nothing on the target. No agent, no credentials, no package to maintain, nothing to uninstall afterwards.

A loop quick enough to leave running unattended, and an acceptance measurement on your own hardware that says how quick.

Sound, on the paths that carry it. The target's audio arrives alongside the picture on a device of its own, independent of the video path: ours kept working through a video freeze that needed a physical replug.

What it cannot do.

It sees pixels. There is no DOM, no accessibility tree, no process list and no file system. If a value is not drawn on the screen, the interface cannot read it.

Nothing comes back except what the machine puts on the wire: picture and sound. No clipboard, no file transfer, no process list, no state the screen does not show. That narrowness is what makes it safe to point at a machine you do not trust.

The keyboard layout is US-only today. A target expecting a different layout will receive the wrong characters.

Capture has to run on the machine that physically owns the hardware. USB-over-IP cannot carry the isochronous transfers video capture needs; it attaches, binds, and drops about a second later, every time.

What surprises people

The screen is not a passive canvas.

These are the ones that cost us the most time. They matter to you because they are the difference between a loop that works and a loop that reports success while doing nothing.

A newer frame is not your frame Capture, encode and transport together run deeper than one frame interval, so the first frame after an action still shows the screen before it. It never looks like a timing bug. It looks like an actuator moving to the wrong place, and it will fit a confident straight line through coordinates nobody observed.

The screen reacts to being pointed at Moving a cursor changes the region it arrives in and the region it leaves, and anything that highlights on hover repaints under it. Three attempts at measuring our own reflex loop were killed by this before we stopped assuming any pixel is inert.

Reading too early gives confident wrong answers Text read from the first changed frame catches a glyph mid-render: A comes back as 4, s as 5. Not an error, not a low-confidence score, but a wrong value delivered cleanly. Timing is measured from the first changed frame; values are read from a settled one.

A stuck key is worse than a dropped one Push the input path past what it will carry and the more damaging failure is not a missing character but an extra one: a lost release report leaves a key held and the operating system auto-repeats it.

Sound is an event, not a direction Audio tells you something happened, often before anything is drawn. It does not tell you where: a bearing from stereo would be a second claim about position, and position needs exactly one source. It is a trigger that makes the agent look, never an aimer. And getting a usable event out of it is its own problem: ours matches its templates cleanly on a quiet background and does not survive a live soundtrack, so we run it where we have shown it works rather than everywhere.

Real screens are not pristine A modal dialog sitting over the middle of a screen biased three of our calibration points by about 17 px while the other 22 were within 5, which is enough to fail an otherwise perfect fit. Calibration has to reject on deviation from the fit, not on where a point sits.

Instruments validate their own assumptions A tracker once reported 5,995 frames with the ball detected across 100 seconds, a figure entirely satisfiable by staring at a stationary score digit, which is what it was doing. We report numbers that are impossible to satisfy on a dead screen.

What we will tell you.

If the job does not need this, we will say so. An API you already have is cheaper than an interface, and most of the value here is in the cases where no API exists.

If a target turns out to be a bad fit, that comes back as a finding rather than a slower loop. Hardware that re-enumerates, a display that will not hold a mode, a host that cannot own the hardware: all of it is better known in week one.

What we do not experiment with

Serving a hand-built EDID to a machine in use wedged its display output, and it never came back across two reboots, a restore of the known-good blob and a physical replug. There is no undo from the source side, because the sink has to re-read to notice and a wedged sink never re-reads. We parse any candidate blob with an independent tool first, and we only ever serve one to a bench target, never to something you depend on.

How we know it holds

We proved it on a harder target than a business screen.

The most demanding thing we have driven this way is a real-time game: a closed loop against nothing but pixels and a keyboard, with no access to the program's state, finishing levels on the clock. Two of them, on the same day, each after several failed attempts that were worth more than the successes.

A dialog box does not move while you think about it. Almost every failure we catalogued there, a detector fooled by scenery it was told could not fool it, a recovery ladder that could not run from the one state it had never considered, an instrument that answered when it could not see, is the same failure that turns up on ordinary screens, only slower and easier to miss.

What that was and was not

A demonstration on our own bench, not a customer deployment, and the runs used the game's own invulnerability so the loop was being tested rather than the marksmanship. What it establishes is narrow and useful: a control loop that reads only the screen, acts only through a keyboard, recovers from its own mistakes, and keeps going long enough to finish something. That is the part that transfers.

Where it is the wrong tool.

If a documented API exists and you are allowed to use it, use it. It will be faster, cheaper and easier to reason about than any screen, and we will tell you so in the first conversation rather than the last.

The same goes for anything needing data the screen never shows, bulk file movement, or a rate no keyboard could sustain. Those are jobs for a different instrument.

A useful test