What to expect · Agent Machine Interface

What it does well.

Any screen a person can operate. BIOS, firmware menus, GUI installers, UAC prompts, appliance web UIs, a machine that has not finished installing an operating system yet.

Real keyboard behavior. Chords, hold, ctrl-drag and shift-click, rather than one key at a time.

Nothing on the target. No agent, no credentials, no package to maintain, nothing to uninstall afterwards.

A loop quick enough to leave running unattended, and an acceptance measurement on the operator's own hardware that says how quick.

Sound, on the paths that carry it. The target's audio arrives alongside the picture on a device of its own, independent of the video path: ours kept working through a video freeze that needed a physical replug.

What it cannot do.

It sees only what is drawn. Nothing underneath the picture, no list of what is running, no files. If a value is not on the screen, AMI cannot read it.

Nothing comes back except what the machine shows and plays. No clipboard, no file transfer, no process list, no state the screen does not show. That narrowness is what makes it safe to point at a machine that is not trusted.

It cannot distinguish a program that has stopped from one that is merely slow. Both put the same unchanging picture on the wire, so liveness is a question the screen cannot answer on its own.

The keyboard layout is US-only today. A target expecting a different layout will receive the wrong characters.

The equipment has to be physically attached to the machine driving it. Putting it at the far end of a network connection does not work, and fails a second after it appears to.

What surprises people

The screen is not a passive canvas.

These are the ones that cost us the most time. They matter because they are the difference between a loop that works and a loop that reports success while doing nothing.

A newer frame is not the frame after the action Capture, encode and transport together run deeper than one frame interval, so the first frame after an action still shows the screen before it. It never looks like a timing bug. It looks like an actuator moving to the wrong place, and it will fit a confident straight line through coordinates nobody observed.

The screen reacts to being pointed at Moving a cursor changes the region it arrives in and the region it leaves, and anything that highlights on hover repaints under it. Three attempts at measuring our own reflex loop were killed by this before we stopped assuming any pixel is inert.

Reading too early gives confident wrong answers Text read from the first changed frame catches a glyph mid-render: A comes back as 4, s as 5. Not an error, not a low-confidence score, but a wrong value delivered cleanly. Timing is measured from the first changed frame; values are read from a settled one.

A stuck key is worse than a dropped one Push the input path past what it will carry and the more damaging failure is not a missing character but an extra one: a lost release report leaves a key held and the operating system auto-repeats it.

Sound is an event, not a direction Audio says something happened, often before anything is drawn. It does not say where: a bearing from stereo would be a second claim about position, and position needs exactly one source. It is a trigger that makes the agent look, never an aimer. Getting a usable event out of it is its own problem: ours matches its templates cleanly on a quiet background and does not survive a live soundtrack, so we run it only where it has been shown to work.

Real screens are not pristine A modal dialog sitting over the middle of a screen biased three of our calibration points by about 17 px while the other 22 were within 5, which is enough to fail an otherwise perfect fit. Calibration has to reject on deviation from the fit, not on where a point sits.

Invented data beats missing data, and that is the danger A calibration that could not collect enough samples once made a curve up and returned it looking exactly like a measured one. Everything downstream under-shot by the real factor with nothing in the result saying anything had been guessed. An instrument that will not admit it has no answer is worse than one that fails.

Instruments validate their own assumptions A tracker once reported 5,995 frames with the ball detected across 100 seconds, a figure entirely satisfiable by staring at a stationary score digit, which is what it was doing. We report numbers that are impossible to satisfy on a dead screen.

A stopped program looks exactly like a slow one A still picture with something caught mid-movement is what a crashed program looks like. It is also what a busy one looks like, and a throttled one, and a machine waiting on a lock. The screen carries no way to tell them apart, and the wrong reading leads to the wrong fix: we diagnosed one of our own stalls twice over before finding it, because each wrong theory had a plausible remedy that changed the symptom slightly. Where we control the software we make it say what happened, on the screen, where the camera can read it. Where we do not, the answer has to come from a second channel rather than from a better look at the same frame.

What we will say.

If the job does not need this, we will say so. An API already in hand is cheaper than AMI, and most of the value here is in the cases where no API exists.

If a target turns out to be a bad fit, that comes back as a finding rather than a slower loop. Equipment that will not stay put, a display that will not settle, a machine that cannot host the kit: all of it is better known in week one.

What we do not experiment with

Lying to a machine in use about the display attached to it killed its picture, and it never came back across two reboots, a restore and a physical replug. There is no undo, because the machine has to look again to notice and it never looks again. We only ever try it on a bench target, never on anything in production.

How we know it holds

We proved it on a harder target than a business screen.

The most demanding thing we have driven this way is a real-time game: a closed loop against nothing but pixels and a keyboard, with no access to the program's state, finishing levels on the clock. Two of them, on the same day, each after several failed attempts that were worth more than the successes.

A dialog box does not move while the agent thinks. Almost every failure we cataloged there, a detector fooled by scenery it was told could not fool it, a recovery ladder that could not run from the one state it had never considered, an instrument that answered when it could not see, is the same failure that turns up on ordinary screens, only slower and easier to miss.

What that was and was not

A demonstration on our own bench, not a customer deployment, and the runs used the game's own invulnerability so the loop was being tested rather than the marksmanship. What it establishes is narrow and useful: a control loop that reads only the screen, acts only through a keyboard, recovers from its own mistakes, and keeps going long enough to finish something. That is the part that transfers.

Where it is the wrong tool.

If a documented API exists and its use is permitted, use it. It will be faster, cheaper and easier to reason about than any screen, and we say so in the first conversation rather than the last.

The same goes for anything needing data the screen never shows, bulk file movement, or a rate no keyboard could sustain. Those are jobs for a different instrument.

A useful test