Understudy
An open-source local desktop agent you teach by demonstrating a task once
Understudy is an MIT-licensed, local-first desktop AI agent for macOS that operates your computer across GUI apps, browser, shell and messaging. You demonstrate a task once and it extracts the intent rather than recording coordinates, turning the demonstration into a reusable skill that generalises to new inputs.
Understudy is an open-source, local-first desktop AI agent that operates a computer the way a colleague would — across GUI applications, the browser, the shell and messaging channels — and learns from demonstration rather than from scripted automation. Given a natural-language task it researches, browses, clicks through desktop apps, manages files and replies through existing channels including Telegram, Slack, Discord, WhatsApp, Signal, LINE, iMessage and web. Its distinguishing claim is teach-by-demonstration: shown a task once, it extracts the intent rather than recording coordinates, producing a skill that generalises to new inputs. The architecture is a layered runtime — an HTTP and WebSocket gateway fronting multiple input interfaces onto a unified session runtime, with built-in tools for GUI automation, browser control, shell commands, web search, memory, scheduling and messaging, and a policy pipeline that gates every tool execution through safety and trust checks. The project describes five capability layers: operating software and teach-by-demonstration are active, implicit learning that crystallises successful patterns into reusable skills and route optimisation that prefers API over CLI over browser over GUI are both partial, and proactive autonomy remains a stated vision rather than shipped code. It is model-agnostic, expecting you to bring your own API key for Anthropic, OpenAI, Google Gemini, MiniMax and others including GitHub Copilot, OpenRouter, Groq and Bedrock, with OpenAI Codex GPT-5.4 as the default. Screenshots, recordings and task data stay on the local machine by default, though the project notes that teaching analysis may send selected screenshots to model providers. It is MIT licensed at 454 stars and 33 forks, installed with npm install -g @understudy-ai/understudy followed by a setup wizard, and GUI automation and teach-by-demonstration currently require macOS, with Linux and Windows planned.
A technical team automating work inside legacy or vendor desktop applications that expose no API, where sending screen recordings to a SaaS agent is not acceptable.
Demonstrate a task once on your own machine and get a reusable skill, with screenshots and task data staying local and the model provider entirely your choice.
At a Glance
- Category
- AI Agents & Orchestration
- Pricing
- Free
- Target Market
- CTOs, Enterprise Developers, Operations Leaders, Platform Engineers
- Deployment
- Open-source, Self-hosted, Edge-first
Key Features
- ✓Teach by demonstration
Show a task once and it extracts intent rather than recording coordinates, producing a skill that generalises to new inputs.
- ✓Cross-surface computer operation
Drives GUI applications, the browser, the shell and files in one session, so automation is not confined to the web.
- ✓Local-first data handling
Screenshots, recordings and task data stay on your machine by default, with only teaching analysis optionally reaching model providers.
- ✓Bring-your-own-model architecture
Works with Anthropic, OpenAI, Google, MiniMax, Groq, Bedrock, OpenRouter and Copilot using your own keys, avoiding vendor lock-in.
- ✓Policy pipeline over tool execution
Every tool call passes through safety and trust checks before running, which matters when an agent has shell and GUI access.
- ✓Messaging-channel interfaces
Accepts instructions and returns results through Telegram, Slack, Discord, WhatsApp, Signal, LINE, iMessage and web.
- ✓Route optimisation across execution paths
Partially implemented preference ordering that favours API over CLI over browser over GUI as faster paths are discovered.
Capabilities
Use Cases
- •Automating a legacy desktop application
Drive an internal app with no API by demonstrating the workflow once instead of maintaining fragile UI automation scripts.
- •Recurring research and reporting
Have the agent browse, gather sources and compile a recurring report, then reply through Slack or Telegram.
- •Chat-triggered desktop workflows
Send an instruction from Signal or iMessage and have the agent execute it on your machine and report back.
- •Building a personal skill library
Convert repeated manual processes into taught skills that generalise, reducing the same work to a single instruction.
- •Privacy-sensitive document handling
Process files locally where uploading screen recordings to a hosted agent service would breach internal policy.
Ideal For
Best For
- ✓Automating desktop applications that expose no API and cannot be driven any other way than through the GUI
- ✓Privacy-constrained automation where screenshots, recordings and task data must stay on the local machine by default
- ✓Teams that want to avoid model lock-in — it is provider-agnostic across Anthropic, OpenAI, Google, MiniMax, Groq, Bedrock and OpenRouter with your own keys
- ✓Turning repetitive knowledge work into reusable skills by demonstrating once rather than writing and maintaining brittle selector scripts
- ✓Triggering and monitoring automation from existing chat channels such as Slack, Telegram, Discord, Signal or iMessage
Not Ideal For
- ✗Linux or Windows shops — GUI automation and teach-by-demonstration are macOS-only, and one HN commenter noted macOS is the overserved platform while Linux is underserved
- ✗Complex processes where a single demonstration is not enough: a commenter with process-training experience argued real handover takes roughly three cycles of watching, doing under supervision and being checked
- ✗Latency-sensitive or high-volume automation — the demo's look-click-look-click loop was criticised on HN as visibly slow
- ✗Buyers who need layers 3 to 5 today: implicit learning and route optimisation are only partial, and proactive autonomy is explicitly a vision
- ✗Regulated environments needing vendor accountability — there is no company, no SLA, no support contract and no compliance attestation behind it
Integrations
Deployment
Market Analysis
Pros
- ✓MIT licensed and local-first, which is a materially different privacy posture from every hosted computer-use agent
- ✓Provider-agnostic across Anthropic, OpenAI, Google, MiniMax, Groq, Bedrock, OpenRouter and Copilot with bring-your-own keys
- ✓Teach-by-demonstration extracts intent rather than coordinates, so skills survive UI changes better than recorded macros
- ✓Honest, published maturity model — the project states plainly which of its five layers are active, partial and aspirational
- ✓Reached 120 points and 41 comments on Hacker News in March 2026 and sits at 454 stars with 33 forks
Cons
- ✗GUI automation and teach-by-demonstration are macOS-only; one HN commenter pointed out that macOS is already overserved with desktop agents while Linux is underserved
- ✗A commenter with real process-training experience argued one demonstration is not enough, citing roughly three cycles of watching, supervised doing and checked output
- ✗The demonstrated look-click-look-click loop was called visibly slow, raising questions about how capable a model it needs to be usable
- ✗The author acknowledged the demo was somewhat cherry-picked, so out-of-the-box reliability on arbitrary tasks is unproven
- ✗Layers 3 to 5 — implicit learning, route optimisation and proactive autonomy — are partial, minimal or unimplemented, so the headline learning story is not fully shipped
- ✗Teaching analysis may send selected screenshots to model providers, which qualifies the local-first claim
- ✗No company, SLA, support contract or compliance attestation stands behind it
Pricing
Open source (MIT)
$0
- ✓MIT licence, full source on GitHub
- ✓No subscription and no vendor lock-in
- ✓Bring your own model provider API key
- ✓All five architectural layers as implemented
- ✓Install via npm and a setup wizard
The software itself is free under an MIT licence with no subscription and no paid tier, but it is not free to run: you supply your own API key for Anthropic, OpenAI, Google, MiniMax or another provider, and the default model is OpenAI Codex GPT-5.4, so the real cost is per-token inference for a GUI agent that takes many screenshot-and-act steps per task. Budget by task volume against your chosen provider's rates.
Security & Compliance
Connect
Sources
This page was written from 4 sources, 3 on domains other than understudy-ai.github.io.
Stay Ahead of the Curve
Weekly enterprise AI insights for technology leaders. No spam, no vendor pitches—unsubscribe anytime.
SubscribeRelated Products
Sema4.ai
Enterprise AI agents for knowledge work, running natively in your Snowflake or cloud account
Sapiom
Agent infrastructure that routes, runs and meters AI agents in production
Itential FlowAI
Governed AI agents for network and infrastructure operations, with deterministic execution and full audit trails
OpenAI Presence
Deploy production-grade AI voice and chat agents with enterprise policies, guardrails and evals