Examples · Agent Machine Interface

Bare-metal provisioning A machine with no operating system has no API, no agent and no remote shell. Installer screens, partitioning, first boot and the license page are reachable by a keyboard and nothing else. This is the case AMI exists for, and the one that scales: the same sequence runs on the tenth machine as on the first.

Firmware and BIOS across a fleet Pre-boot settings exist before the network stack does. Vendor menus differ, change between revisions, and the tools that exist cover one manufacturer. A keyboard and a view of the screen work on all of them.

Appliances with a web UI and no usable API A storage shelf, a UPS, a camera recorder, a building controller. The interface was designed for a person, and either the API does not exist or the credentials are not worth scripting against. The screen is the supported surface, so use it.

Software that cannot be modified GUI-only applications, licensing that forbids automation hooks, or a support contract that lapses the moment anything is installed alongside the product. Driving the machine from outside changes nothing on it, so nothing is voided.

Estates where nothing may be installed Regulated environments, warranty-bound hardware, or a network segment that will never accept an agent. The constraint that blocks every other approach is the premise this one starts from.

Recovery when the usual path is gone The machine has dropped off the network, the agent is dead, or the system will not boot far enough to answer. None of that matters to a video output and a keyboard input, which is exactly when they matter most.

The seventh shape

Where recording the mouse and keyboard is not enough.

A macro recorder replays what happened once. It cannot look at the screen, so it cannot tell whether any of it worked. That holds until the machine does something slightly different, which it will, and the recorder keeps going regardless.

Coordinates drift. A window opens somewhere else, a resolution changes, a dialog sits twenty pixels lower after an update. The recorder clicks the old position and hits whatever is there now.

Fixed waits are always wrong. Too short and it races the machine; too long and a twenty-minute job takes two hours. Real durations vary with load, content and the day.

Nothing branches. An unexpected update prompt, a certificate warning, a retry dialog. A recorded sequence has no answer for any of them and no way to notice one appeared.

Failure is silent and compounding. Once one step lands wrong, every later step runs against a state nobody predicted, and the run still reports that it finished.

What we replace each with

Find the thing on screen and act on where it actually is, against a calibrated model of where the pointer lands.

Wait for the screen to change and settle, rather than sleeping for a guess. Waiting costs nothing and takes exactly as long as the machine takes.

Read the screen to decide what to do next, which is what makes an unexpected dialog a branch rather than the end of the run.

Check the precondition before acting and stop when it cannot be checked, so a run that has lost its footing says so instead of continuing.

Why this one is personal

We ran a typing benchmark across five trials and it produced a clean-looking table. Every keystroke had gone into the wrong window. The target had lost focus, and nothing in the numbers said so. That is the macro failure mode exactly: the work did not happen, the run reported success, and only a check against the screen could have told the difference. It is why the loop verifies focus and state before it acts, and aborts when it cannot.

The eighth shape

Testing what the user sees, not what the application says it drew.

A test framework reaches into the program: an accessibility tree, a document model, an automation interface. That is a privileged channel, and what it tests is the program's own account of itself. AMI reads the screen from outside, on the hardware the software will ship on.

What it catches that an instrumented test cannot

A control the application believes it drew, rendered behind a modal, clipped off the display, or painted in a color nobody can read.

Faults that live below the application: display scaling, graphics drivers, compositing, a second monitor, a resolution the layout was never tried at.

The shipped binary on the shipping machine, with no debug build, no instrumentation flag and no test hook compiled in. What is tested is what the customer receives.

Everything the framework cannot attach to at all: installers, firmware menus, kiosk mode, a machine part-way through booting, a protected video path.

And one measurement nothing else gives

How long after an input the user can see the result. We take it the way we take our own figures: a keyboard or pointer action, then the first frame in which the region changes. Measured at the glass, it includes the parts a code-level timer never sees, and it is the number a person experiences as the application feeling quick or slow.

The same rig gives a regression test for it. A build that gets slower to respond fails on a number rather than on somebody's impression.

The check has to be checked first

A visual test is worth what its comparison is worth, and a comparison that cannot fail cannot pass. A pixel diff against a stored reference is honest about this: it breaks loudly the moment anything moves. A model asked whether a screen looks right is not, because a model that has understood nothing still answers, and usually answers that everything is fine.

That reads exactly like a clean result. We test for it by showing the checker matched pairs, a real capture and the same capture with an obvious defect painted into it, and requiring that it tell them apart before any verdict from it counts. On our own bench a small local vision model returned an identical verdict on six of six pairs when half of the image had been blanked out. It reported a clean site because it could not see the site at all.

So an acceptance run we hand over reports what it inspected as a number, and refuses to report a pass when the coverage behind it was not earned. A test that has quietly stopped looking is worse than no test, because it still produces green.

What it is not

It is slower than an instrumented test and it does not replace unit or integration testing. It belongs on the last mile: acceptance on real hardware, and the cases a framework cannot see. It reads pixels, so a deliberate change to the user interface fails the test until the expectation is updated. That is a real cost, and the same property makes it notice a change nobody intended.

Before planning around any of these

Several of the shapes above touch industrial equipment. AMI must not be relied on where a failure could injure someone or cause serious loss, and it is not a safety instrumented system. The full statement, together with what AMI cannot do at all, is on the safety page.

The ninth shape

Buying things, without the payment details ever reaching the agent.

Ordering parts, renewing a licence, booking a shipment. Checkout was built for a person, and for most suppliers there is no purchasing interface worth integrating against. The agent works the page the way a buyer does: finds the item, reads it carefully enough to know it is the right one, fills the form, and stops where the money moves.

The card is never typed. A vault the operator controls fills it, so the number never becomes keystrokes, never enters the agent's context, and is not in anything derived from it. The same arrangement used for passwords.

Where it stops is yours to set. Take it to the confirmation and press that yourself, or let it complete within limits you have drawn. That line belongs to whoever owns the card.

The limit is the account's. A card with a low ceiling, a single-use number, or a supplier account that can only ship to one address. The same controls that would bound a member of staff, doing the same job.

When a listing is unclear, it asks. The question box and the message thread are on the page like everything else, so an ambiguity gets resolved the way a buyer resolves it rather than guessed at.

A verification in the way is just another screen. Where a check appears part-way through, on an account you hold, doing a thing you could have done yourself, it gets read and answered rather than the job stopping to wait for you. It is the same reading that gets it through every other screen.

What it is not

Not unattended spending. Nothing here removes the need for a person to be accountable for a purchase, and the design deliberately keeps one in the loop at the point it matters.

Not a way around a site that has told you no. Where you hold the account and the task is one you could do yourself, an occasional verification is a speed bump. Where the check is there because the operator does not want automated use, or the volume is one no person would transact, that is a no, and it does not stop being a no because the keystrokes look human.

Not a substitute for the account limits. Where a mistake would be expensive, what bounds it is the ceiling on the card and the address the goods can reach, not how carefully anything read the page.

Wherever the usual way in does not exist.

Where recording the mouse and keyboard is not enough.

Testing what the user sees, not what the application says it drew.

Buying things, without the payment details ever reaching the agent.