WindowsAgent
GitHub

A structured Windows runtime for AI agents

Give your AI agent
a Windows computer.

Install WindowsAgent with AssistGUI, connect over a trusted LAN or optional Tailscale network, and give your agent one structured runtime for seeing, acting, and verifying.

Download AssistGUI View on GitHub
  • AssistGUI entry point
  • Windows 10 1903+ · amd64
  • Trusted LAN · Tailscale
  • MIT
01
Open AssistGUIThe runnable Windows setup app
02
Install and startManage the WindowsAgent runtime
03
Connect your agentUse the reported LAN or Tailscale endpoint

Start here

AssistGUI gets WindowsAgent running

Download the release ZIP, launch AssistGUI in your signed-in Windows session, and manage the installed runtime without beginning at a terminal.

Download windows-assist-gui.zip
  1. 01

    Download and launch

    Extract the runnable AssistGUI distribution from the latest GitHub Release, then open it on Windows.

  2. 02

    Install and manage

    Install, update, repair, uninstall, start, or stop WindowsAgent. Inspect the installed version and WindowsAgent running state.

  3. 03

    Choose a trusted route

    Use the LAN endpoint AssistGUI reports, or provide your own Tailscale auth key to enable the optional private-overlay connection.

  4. 04

    Give the endpoint to your agent

    AssistGUI reports connection status and address. Share the endpoint only with an agent on the same trusted network.

AssistGUI reports Tailscale status and IP, but it does not create credentials. Runtime endpoints are unauthenticated by default: keep them off the public Internet.

What WindowsAgent provides

One runtime. Six capability groups.

WindowsAgent is more than a screenshot server, shell wrapper, remote desktop, or game bot. It exposes Windows capabilities through explicit, inspectable contracts.

Available

See

Capture the primary display through WGC, identify the foreground executable, resolve its owning Rule, and return a verified artifact with capture provenance.

Partially available

Act

Send bounded, foreground-bound keyboard and pointer operations. Completion proves the input operation—not that the application accepted the intended result.

Partially available

Run

Execute structured process operations, uploaded PowerShell files, or bounded Starlark automation with durable results and explicit cancellation contracts.

Available · optional

Move data

Transfer files through the independent SFTP-only filesystem runtime. It is a data plane—not an SSH shell or hidden execution fallback.

Available + partial

Observe

Inspect processes and services today. Optional event, invocation, Evidence, and Visual Log paths add durable history at their documented maturity levels.

Available + partial

Connect

Reach the runtime over a trusted LAN. AssistGUI can report endpoints and manage an optional Tailscale route; that integration remains partially available.

Maturity is part of the contract. “Partial” means a real path exists with documented work still deferred. Declaration-only registrations and draft designs are not presented here as running features.

Normal Windows control

One route from your agent to a Windows PC

WindowsAgent exposes the Windows capabilities an agent needs through one runtime instead of requiring a custom stack of unrelated capture, file-transfer, execution, and input bridges.

01

Understand the current state

Capture the desktop, identify the foreground application, and inspect the process and service snapshot before deciding what to do.

02

Perform a bounded task

Transfer files, run a structured command or uploaded PowerShell workflow, then use foreground-bound keyboard or pointer input where supported.

03

Check what happened

Read the operation result and, where the capability provides it, inspect durable events or Evidence instead of assuming a sent command achieved the goal.

Private network boundary. The runtime is not a public remote-desktop service. Its HTTP and SFTP endpoints are unauthenticated by default; use them only on a trusted LAN or private overlay and keep them off the public Internet.

Built for agents

Observable operations, not an opaque shell and a guess

An AI agent needs to know what it addressed, what was accepted, what finished, and what still needs independent verification. WindowsAgent makes those boundaries explicit.

  1. 01

    Know the owner

    Foreground executable identity selects the owning capability boundary. A window title or visible text does not silently choose a Rule.

  2. 02

    Bound the operation

    Structured inputs, declared permissions, limits, deadlines, and result schemas keep each operation reviewable.

  3. 03

    Fail explicitly

    Protocol, permission, schema, process, and artifact failures stay failures. The runtime does not hide them behind guessed state or silent provider changes.

  4. 04

    Keep durable state

    Where a capability supports it, invocation IDs and ordered events let an agent resume observation instead of inferring progress from timing.

  5. 05

    Verify the postcondition

    A sent command proves only the command. When the goal matters, inspect the declared result or independent Evidence instead of assuming application success.

transport accepted runtime completed execution result application accepted user goal verified

These are distinct claims. Not every operation supplies every layer of evidence; its own contract defines what can be proven.

Advanced game control

When one model turn is too slow

For games, an executable-scoped Rule can add finite Actions, streaming workflows, OCR, vision, and game bindings. The agent plans at a high level while a bounded local Action owns the fast feedback loop.

NORMAL WINDOWS CONTROL agent → WindowsAgent operation
ADVANCED GAME CONTROL agent → high-level plan → local Rule / Action → bounded fast loop → game

0 model callsin the shipped fast path. Deterministic Starlark Actions own the bounded feedback loop while the model plans and supervises outside it.

Fast · The Reflex

Streaming Starlark actions

  • Observe → gate → key pulse → emit, on a 250 ms cadence
  • Hard gates: consecutive frames, confidence margins, sample budgets
  • 40–160 ms scan-code pulses; critical compensation on exit
  • Interruptible, always bounded, never guessing

Index · The Visual Log

1 FPS Evidence + gemma-4-e4b-it-8bit

  • Bounded 1 FPS 1080p recordings — MP4 segments, SHA-256 manifest, contact sheets
  • Passive Gemma scene index, one fresh Evidence frame per tick, over a LAN oMLX endpoint
  • An untrusted locator: narrows the interval, never authoritative evidence
  • Failure-isolated — a bad description drops one sample; recording never stops

Slow · The Planner-Developer

High-level model · Codex / OpenCode

  • Plans the goal; starts, watches, and stops streaming actions
  • Uses the index to find the suspect interval, verifies against authoritative Evidence
  • Edits the owning Action package when evidence exposes a fault
  • Hot-syncs the Rule — no rebuild, no agent restart — and re-runs acceptance

event journal + evidence timelineappend-only · durable · replayable

Advanced packages · real implementations

Shipped game Rules

These executable-scoped packages extend the general runtime with game-owned observation, decisions, bindings, and Actions.

Elite Dangerous

Full docking workflow

A supervised departure: the leave-station streaming action owns the whole flow — AUTO LAUNCH menu preparation, throttle gates, mass-lock checks, and three consecutive zero-speed OCR confirmations. The planner confirms the station, then just watches.

  • compass
  • ship-status
  • ship-speed
  • flight-status
  • leave-station
  • align-station-target
  • ui-control
  • set-throttle

Crimson Desert

Memory + save decode

The inventory action reads game memory first; only when that fails does it locate the newest save file and decode it through a package-declared native DLL over the sandboxed FFI.

  • inventory
  • native FFI

Palworld

On-device vision

The screenparser action runs a pinned FP16 ScreenParser v2 ONNX model over DirectML on one caller-supplied, hash-pinned frame — then exits. No loop, no fallback, no residue.

  • ui-elements
  • ONNX · DirectML

Rules may declare Monitor or Reaction eligibility, but WindowsAgent does not ship a scheduler or subscription dispatcher. Catalog eligibility is not a running automation.

Real game evidence

Elite Dangerous proves the advanced loop

The first shipped Rule uses local deterministic Actions for flight phases where a model-turn control loop is too slow. The media below is real runtime evidence, not a product mockup.

GAMEAbout the game

  • A 1:1-scale Milky Way — hundreds of billions of star systems, flown in real time
  • No pause button: docking, supercruise, and hyperspace jumps are manual flying skills
  • Long-haul trade runs mix minutes of routine flight with seconds that decide everything

FITWhy it fits an agent runtime

  • High-contrast orange HUD — friendly to OCR and region classifiers
  • Hard failure gates (mass-lock, throttle zones, station rotation) make honesty testable
  • Routine flight phases hand over cleanly to deterministic streaming reflexes
  • Every flight ends with observable postconditions — docked, departed, jumped — not vibes
Filmed off the living-room TV: the agent owns flight controls through supercruise while the Action OSD narrates every gate — zero model calls in the fast path.
Close-up of the Action OSD overlay on Elite Dangerous during supercruise assist
Up close: the supercruise-assist-to-destination OSD — preflight gates, destination alignment, flight-control ownership.

Extend the advanced layer

A Rule keeps game semantics with the game

Each executable-scoped Rule owns its guidance, Action packages, schemas, bindings, coordinates, classifiers, and postconditions. Core stays game-neutral.

  • Start with one evidence-backed observation
  • Add a finite Action for one bounded operation
  • Use a streaming Action only when the workflow must retain state, wait, compensate, or verify over time

Developers and operators

Build, inspect, and extend WindowsAgent

AssistGUI is the normal installation path. Use the repository path when you need to build the runtime, inspect its contracts, develop a Rule, or contribute code.

1 · Build from source

go test ./...
./scripts/build-windows-capture-agent.sh \
  --skip-assist-gui --skip-catalog

2 · Run in the signed-in session

.\.build\windows-capture-agent-console.exe `
  --rules-dir (Resolve-Path .\.build\Rules)

3 · Create a capture

curl.exe --data-binary '{"include_cursor":true}' `
  http://127.0.0.1:8787/v1/captures
GitHub repository Source, releases, and project README API and runtime guide Current commands, HTTP surface, and project map Design registry Canonical maturity for maintained designs Agent Skills Repository-owned development and operating workflows Security Trust boundaries and vulnerability reporting Contributing How to propose and validate a change
Trusted networks only. The Capture Agent listens on 0.0.0.0:8787 without authentication or TLS by default. Reachability is the trust boundary for capture, execution, and enabled input surfaces.