Use cases · Agent Machine Interface

Bare-metal provisioning A machine with no operating system has no API, no agent and no remote shell. Installer screens, partitioning, first boot and the license page are reachable by a keyboard and nothing else. This is the case the interface exists for, and it is the one that scales: the same sequence runs on the tenth machine as the first.

Firmware and BIOS across a fleet Pre-boot settings live before the network stack does. Vendor menus differ, they change between revisions, and the tools that exist tend to cover one manufacturer. A keyboard and a view of the screen work on all of them.

Appliances with a web UI and no usable API A storage shelf, a UPS, a camera recorder, a building controller. There is an interface, it was designed for a person, and either the API does not exist or the credentials are not worth scripting against. The screen is the supported surface, so use it.

Software that cannot be modified GUI-only applications, licensing that forbids automation hooks, or a support contract that lapses the moment anything is installed alongside the product. Driving the machine from outside changes nothing on it, so nothing is voided.

Estates where you may not install Regulated environments, warranty-bound hardware, or a network segment that will never accept an agent. The constraint that blocks every other approach is the premise this one starts from.

Recovery when the usual path is gone The machine has dropped off the network, the agent is dead, or the system will not boot far enough to answer. None of that matters to a video output and a keyboard input, which is exactly when you need them most.

The seventh shape

Where recording the mouse and keyboard is not enough.

A macro recorder replays what happened once. It cannot look at the screen, so it cannot tell whether any of it worked. That is fine until the machine does something slightly different, which it will, and the recorder keeps going regardless.

Coordinates drift. A window opens somewhere else, a resolution changes, a dialog sits twenty pixels lower after an update. The recorder clicks the old position and hits whatever is there now.

Fixed waits are always wrong. Too short and it races the machine; too long and a twenty-minute job takes two hours. Real durations vary with load, content and the day.

Nothing branches. An unexpected update prompt, a certificate warning, a retry dialog. A recorded sequence has no answer for any of them and no way to notice one appeared.

Failure is silent and compounding. Once one step lands wrong, every later step runs against a state nobody predicted, and the run still reports that it finished.

What we replace each with

Find the thing on screen and act on where it actually is, against a calibrated model of where the pointer lands.

Wait for the screen to change and settle, rather than sleeping for a guess. Waiting costs nothing and takes exactly as long as the machine takes.

Read the screen to decide what to do next, which is what makes an unexpected dialog a branch rather than the end of the run.

Check the precondition before acting and stop when it cannot be checked, so a run that has lost its footing says so instead of continuing.

Why this one is personal

We ran a typing benchmark across five trials and it produced a clean-looking table. Every keystroke had gone into the wrong window. The intended target had lost focus, and nothing in the numbers said so. That is the macro failure mode exactly: the work did not happen, the run reported success, and only a check against the screen could have told the difference. It is why the loop verifies focus and state before it acts, and aborts rather than proceeding when it cannot.

The eighth shape

Testing what the user sees, not what the application says it drew.

A test framework reaches into the program: an accessibility tree, a document model, an automation interface. That is a privileged channel, and what it tests is the program's own account of itself. This reads the screen, from outside, on the hardware the software will actually ship on.

What it catches that an instrumented test cannot

A control the application believes it drew, rendered behind a modal, clipped off the display, or painted in a colour nobody can read.

Faults that live below the application: display scaling, graphics drivers, compositing, a second monitor, a resolution the layout was never tried at.

The shipped binary on the shipping machine, with no debug build, no instrumentation flag and no test hook compiled in. What you test is what the customer receives.

Everything the framework cannot attach to at all: installers, firmware menus, kiosk mode, a machine part-way through booting, a protected video path.

And one measurement nothing else gives you

How long after an input the user can see the result. We take it the same way we take our own figures: a keyboard or pointer action, then the first frame in which the region changes. It is measured at the glass, so it includes the parts a code-level timer never sees, and it is the number a person actually experiences as the interface feeling quick or slow.

The same rig gives you a regression test for it. A build that gets slower to respond fails on a number rather than on somebody's impression.

What it is not

It is slower than an instrumented test and it is not a replacement for unit or integration testing. Use it for the last mile: acceptance on real hardware, and the cases a framework cannot see. It also reads pixels, so a deliberate change to the interface fails the test until the expectation is updated. That is a real cost, and it is the same property that makes it notice a change nobody intended.

Before you plan around any of these

Several of the shapes above touch industrial equipment. The interface must not be relied on where a failure could injure someone or cause serious loss, and it is not a safety instrumented system. The full statement, together with what the interface cannot do at all, is on the safety page.

Wherever the usual way in does not exist.

Where recording the mouse and keyboard is not enough.

Testing what the user sees, not what the application says it drew.