WindowsAgent
GitHub

Agent-facing game runtime for Windows

Fast reflexes.
Slow reasoning.
Your agent plays the game.

WindowsAgent is a runtime where AI agents author and execute actions for Windows games — perception, sandboxed execution, and input control under one strict contract.

View on GitHub Read the docs
  • Go 1.23+
  • Windows 10 1903+ amd64
  • MIT
  • Codex · OpenCode

How it works

The agent is the developer — and the pilot

One capture is the agent's single perception entry point. From there the agent reads the rule, authors or invokes an action, and receives durable evidence back.

  1. 01

    Perceive

    WGC captures the primary monitor; every artifact carries foreground process metadata and a SHA-256 hash.

  2. 02

    Navigate

    The capture resolves the foreground game to its Rule plugin — AGENTS.md guidance plus a live action catalog.

  3. 03

    Author

    The agent writes Starlark script packages and declares actions; manifests are validated against strict schemas.

  4. 04

    Execute

    Finite actions return one validated result; streaming actions run reflex loops with explicit evidence gates.

  5. 05

    Feedback

    Every step lands in an append-only event journal the agent can replay — the shared timeline for planning.

What the runtime provides

Ground truth for your agent

EYEPerception

  • Windows Graphics Capture + D3D11, HDR tone-mapped to SDR
  • Resident PP-OCRv6 DirectML workers, declared per Rule
  • ScreenParser v2 ONNX detector on hash-pinned frames
  • Reference-density screen regions mapped through a 16:9 viewport

COREExecution

  • Sandboxed Starlark script packages with strict limits
  • Package-owned native DLLs via a generic Windows amd64 FFI
  • Finite and streaming actions with durable invocation IDs
  • Append-only event journal with authenticated loopback replay

HANDControl

  • Foreground-bound scan-code key injection, one press at a time
  • Game key bindings resolved live from the active .binds preset
  • Logical UI moves selected by the supervising model
  • Action OSD overlay shows the live action without stealing focus

LAWContracts

  • Manifest, input, and output schemas validated on every call
  • UNKNOWN is a first-class result — evidence is never invented
  • SHA-256 pinned models and artifacts, verified before use
  • Bad requests fail explicitly; nothing degrades silently

Two-speed architecture

Slow planning meets fast reflexes

The high-level model never sits in the fast path. It plans and supervises; streaming actions react in real time behind hard evidence gates. The event journal joins the two.

~10sis one raw LLM screenshot-think-act turn. Games don't wait — so the fast path belongs to streaming actions, while the model plans and authors new actions.

Slow · The Planner

High-level model · Codex / OpenCode

  • Captures the screen and reasons about the goal
  • Arranges UI one logical key at a time
  • Starts, watches, and stops streaming actions
  • Replays the event timeline to plan the next move
event journal append-only · durable · replayable

Fast · The Reflex

Streaming Starlark actions

  • Observe → gate → key pulse → emit event
  • Hard gates: consecutive frames, confidence margins
  • Fails explicitly when evidence runs out
  • Interruptible, always bounded, never guessing

Proof, not promise

Shipped Rules

Every capability below was built on the same runtime your agent gets.

Elite Dangerous

Full docking workflow

A supervised departure: the planner arranges AUTO LAUNCH key-by-key, then the leave-station streaming action takes over — throttle gates, mass-lock checks, and three consecutive zero-speed OCR confirmations before it reports completion.

  • compass
  • ship-status
  • ship-speed
  • flight-status
  • leave-station
  • align-station-target
  • ui-control
  • set-throttle

Crimson Desert

Memory + save decode

The inventory action reads game memory first; only when that fails does it locate the newest save file and decode it through a package-declared native DLL over the sandboxed FFI.

  • inventory
  • native FFI

Palworld

On-device vision

The screenparser action runs a pinned FP16 ScreenParser v2 ONNX model over DirectML on one caller-supplied, hash-pinned frame — then exits. No loop, no fallback, no residue.

  • ui-elements
  • ONNX · DirectML

The meta-game

Making a game agent-playable is itself a game

Anyone can contribute a Rule for any game. Each new capability is a cleared level — judged by strict contracts, recorded in replays, and unlocked for every agent.

QUESTA capability map per game

  • Level one: read a single HUD value with evidence
  • Mid game: compose classifiers and gate actions
  • Boss fight: a full streaming reflex workflow

REFEREEContracts are the scoring system

  • Strict schemas judge every contribution on every call
  • Black-box acceptance contracts decide what counts as cleared
  • Guessing never passes — UNKNOWN is honest, not a win

REPLAYEvery run is a record

  • The durable event journal captures the whole clear
  • Export, share, and compare your agent's runs
  • A four-minute station departure becomes a time to beat

OPENAny game can enter the map

  • A Rule is one plugin folder: guidance, actions, runtimes
  • Hot-swap plugins with no agent restart or rebuild
  • Your contribution becomes every agent's new ability

Quick start

From clone to first capture

1 · Build

go test ./...
./scripts/build-windows-capture-agent.sh

2 · Run in the signed-in session

.\.build\windows-capture-agent-console.exe `
  --rules-dir (Resolve-Path .\.build\Rules)

3 · Capture & invoke

curl.exe --data-binary '{"include_cursor":true}' `
  http://127.0.0.1:8787/v1/captures

curl.exe --data-binary '{"actionId":"elite-dangerous/ship-status","inputs":{}}' `
  http://127.0.0.1:8787/v1/actions/invoke
Trusted networks only. The HTTP server listens on 0.0.0.0:8787 without authentication, TLS, or CORS by default. Anyone who can reach that port can trigger captures and read foreground process metadata. Use it only on a trusted LAN or private overlay network.