Why My Game Agent Doesn't Let an LLM Control Every Frame

An AI game agent can press the correct key and still be wrong.

In Elite Dangerous, sending the zero-throttle binding proves only that the command left the agent. It does not prove that the ship stopped. If the next action assumes that it did, the rest of the plan is already operating on a false state.

I first tried the obvious architecture: capture a frame, ask a vision model what to do, send a key, and repeat. The problem was not simply model latency. I had placed three different clocks, failure modes, and standards of evidence into one loop.

WindowsAgent grew out of a harder question than automating one game: can a high-level AI understand which game capabilities it already has, discover what is missing, and then develop a new skill that can be invoked, verified, stopped, replayed, and repaired?

In this system, an Action is a game capability the AI can invoke. A Gate is a precondition that current evidence must prove before the Action is allowed to continue. A Streaming Action repeatedly observes, acts, and verifies inside a bounded control loop. Raw frames and events are retained as Evidence, separately from the model's interpretation, so later reasoning can still be challenged.

The engineering conclusion I reached is simple: a high-level model can own goals, planning, exceptions, and skill development, but it should not own every time-sensitive control pulse.

The first low-return design: ask the model every frame

The most direct AI game demo continuously takes screenshots, sends them to a vision model, and lets the model choose the next keypress.

That looks convincing in a demo. Under real-time control, one screenshot, transfer, queue, inference, and response can already exceed the game's useful reaction window. Lower average latency does not eliminate network jitter, service queues, or occasional outages, and none of them has a hard upper bound.

Delay is only part of the problem. Asking again on every frame means reconstructing the state on every frame. The previous answer does not automatically become a reliable contract for the next one. A model can give a coherent explanation without proving that the game changed after its command.

I eventually realized that choosing a faster model would not fix the architecture. Ship attitude and throttle control operate on tens to hundreds of milliseconds. Evidence recording and event indexing operate on seconds. High-level planning, development, and review operate on seconds to minutes. Speed exposed the problem, but responsibility was what actually needed to be separated.

The fast loop: Actions own deadline-bound reflexes

The first WindowsAgent loop handles fast reflexes.

A Streaming Action observes the current state, checks its Gates, emits a short input, records an event, and then verifies the new state. It does not wait for a high-level model to decide every pulse, and uncertain evidence never grants it more authority.

• Every Action has explicit time and sample limits.
• An unclear observation returns UNKNOWN; it never reuses a convenient earlier frame.
• A failed control sequence can stop and run bounded compensation, such as returning throttle to zero.
• Command delivery and observed outcome are recorded as different events.

The station-departure flow makes this distinction concrete. The Action first uses fresh HUD evidence to establish that the ship has really started moving. It then waits for the mass-lock state to clear. After sending zero throttle, it still requires three later visual confirmations of zero speed before reporting completion.

zero-throttle command sent
→ key delivery confirmed
→ ship state still unproven
→ later HUD frames sampled
→ STOPPED confirmed three times
→ Action completed

The Action verifies a state transition, not a keypress: a command was emitted, the game produced a new state, and the postcondition became observable. Without that final step, the system can prove only what it attempted, not what happened.

The index loop: models can help find evidence, but cannot create it

The second loop records events and helps locate failures.

An append-only Event Journal retains Action starts, stages, child Actions, failures, and completions. An Evidence Recorder samples the actual game so that a long journey can still be reconstructed later.

A small vision model may write timestamped scene descriptions to help a high-level agent find the relevant part of a long recording. Those descriptions are an untrusted index. They are allowed to be incomplete, but they can never replace the original frames.

This boundary came from a real debugging case. The game had reached the correct page, but the button classifier returned UNKNOWN, so the whole flow stopped. The system did not silently switch recognizers or infer that a visible page implied an actionable button. The high-level agent used the event time to retrieve the corresponding frame, inspected the OCR region and classification conditions, repaired the Action that owned the capability, and ran it again.

The index loop does not let a model decide what happened. It helps the model find the evidence that deserves a new question.

The slow loop: the AI is both player and skill developer

The third loop is where high-level intelligence belongs.

The agent first identifies the foreground game and reads its Rule: a plugin directory containing game guidance, Action contracts, and Runtime declarations. If an existing capability is sufficient, it invokes the Action. If evidence reveals a missing capability, it returns to the package that owns that skill, changes the implementation, verifies the contract, and updates the Rule.

The model is therefore not just a remote pilot. It uses skills to play while also developing the skills needed to continue.

The current Elite Dangerous Rule has grown to 100 Actions. That number is not a feature checklist; it is a pressure test. As capabilities expand from reading one HUD state to menu navigation, attitude control, and multi-stage travel, can every Action retain its own contract, failure semantics, and evidence?

This is why a skill tree is a better description than a macro library. A macro remembers an input sequence. An Action must also know when it is authorized to run, how completion will be proven, and who regains control after failure.

Three loops are three responsibilities, not three modules

It is easy to interpret the design as ordinary performance tiering: keep fast work local and send slow work to the cloud. The actual boundary is closer to a control system's division of responsibility.

The high-level AI can decide what the ship should do and develop a missing capability. It does not personally choose every short key-down or key-release pulse.

The fast loop cannot quietly acquire high-level authority either. An Action operates only inside its schema, Rule, and Gates. An undefined screen or infrastructure failure must remain an explicit failure. It cannot change models, switch data sources, or guess an alternative route just to keep the workflow moving.

Why faster models do not remove the boundary

This design does not depend on 2026 models being slow.

Inference will continue to improve, but more model throughput does not remove network round trips, queues, jitter, or service interruptions. A low average latency is not a deadline guarantee.

Moving the high-level model onto the PC changes latency but not automatically responsibility. Inference may still be variable, context-dependent, and probabilistic. High-level reasoning is useful for goals, exceptions, and capability development. Deterministic control should remain deterministic where the domain allows it.

A sufficiently fast local model might move some work across the boundary. That is the part of this design I am least certain about. My current position is that faster inference can reduce latency inside a responsibility, but does not by itself prove that the model should own the responsibility.

Engineering rules that survived the experiment

• Separate work by clock, consequence, and trust level, not only by technology name.
• Let the high-level model own goals, not every control pulse.
• Record command delivery and postcondition evidence separately.
• UNKNOWN is a valid result, not an empty field waiting to be guessed.
• A vision model may index evidence, but the index never becomes the evidence.
• Repair the Action that owns a capability instead of hiding a workaround in a higher-level prompt.
• A reusable skill must be verifiable, stoppable, replayable, and repairable.

The next problem: how do you verify a growing skill tree?

The three loops separate responsibilities across time scales. They do not automatically solve governance as the capability graph grows.

With 100 Actions, a change to one low-level output, schema, or failure meaning may affect multiple parent workflows. Adding the next skill is no longer the hardest problem. The harder problem is proving that it did not invalidate a parent's precondition or turn a visible failure into a hidden alternative path.

The next stage is therefore not simply teaching the agent more moves. The whole skill tree needs static checks, black-box acceptance, and evidence from the real Windows runtime. A high-level AI can participate in the repair, but every proposed change must return to the same questions: are the inputs explicit, is the authority bounded, is failure visible, and was the postcondition actually observed?

This is the Agent Runtime question I care about most. Models will become more capable and skill graphs will grow. Long-term maintainability will depend less on how many actions the AI can perform than on whether each composed capability still preserves a boundary that can be verified.

Project: WindowsAgent website · GitHub repository