Andrew Ng's OpenWorker: A Local-First Desktop AI Agent — The Paradigm Shift from 'Dialogue' to 'Deliverable'
1. Introduction: The Last-Mile Problem of AI Assistants
Over the past two years, large language models have advanced at a breathtaking pace. GPT-5 series, Claude Opus 4.8, Gemini 3.6 — each model delivers stunning responses and sets new benchmarks. Yet one fundamental problem remains unsolved: AI stays in the “talking” phase and never reaches the “doing” phase.
You ask an AI to “prepare a client renewal brief,” and it gives you a block of text. You ask it to “resolve this week’s calendar conflicts,” and it hands you a to-do list. You ask it to “triage a production incident,” and it writes an analysis report. Then what? You still have to do it yourself — open a document editor, switch to the calendar app, log into Slack to send the message. The AI replied to you, then left you to clean up the mess.
On July 23, 2026, Andrew Ng and former Alexa leader Rohit Prasad open-sourced OpenWorker — an MIT-licensed desktop AI agent. It has garnered 14.5k stars on GitHub, positioning itself as a “desktop AI coworker.” Its core promise is simple: AI should deliver finished work, not conversation.
┌──────────────────────────────────────────────────────────────────┐
│ Paradigm Shift: From "Dialogue" to "Deliverable" │
│ │
│ Traditional AI Assistant (Chatbot Paradigm) │
│ ┌──────────┐ ┌──────────┐ ┌──────────────────────┐ │
│ │ User asks│ -> │ AI replies│ -> │ User does the work │ │
│ └──────────┘ └──────────┘ └──────────────────────┘ │
│ ↑ │
│ "Write a report" → text block → open Word, format yourself │
│ │
│ OpenWorker (Deliverable Paradigm) │
│ ┌──────────┐ ┌──────────────────┐ ┌──────────────┐ │
│ │ User │ -> │ AI decomposes + │ -> │ Finished │ │
│ │ states │ │ executes with │ │ deliverable │ │
│ │ outcome │ │ approval gates │ │ ready to use │ │
│ └──────────┘ └──────────────────┘ └──────────────┘ │
│ ↑ │
│ "Prep for renewal call" → read CRM+Slack+analyze → HTML brief │
│ → request approval → email to CTO │
└──────────────────────────────────────────────────────────────────┘
This is not a simple chatbot upgrade. It is a profound shift in AI product design philosophy — from “optimizing response quality” to “optimizing delivery completeness.” This article will dissect OpenWorker’s technical implementation from the perspectives of architecture design, approval gating, connector ecosystem, and model routing.
2. Project Background: Why Andrew Ng
Andrew Ng needs little introduction. Stanford professor, co-founder of Coursera, early member of Google Brain, former chief scientist at Baidu, founder of DeepLearning.AI. His co-founder on this project, Rohit Prasad, was the head of Amazon Alexa, with over eight years of experience building large-scale voice AI products.
The signal is clear: AI agents must evolve from “toys” to “tools,” and productization is the critical step.
OpenWorker was created on July 20, 2026, officially open-sourced on July 23, and widely circulated by August. At the time of writing, the GitHub repository has 14.5k stars and contains:
coworker/: Python backend — ~119 files, 32,400 lines of code (agent engine, model providers, connectors, MCP client, memory system, automation scheduler)surfaces/gui/: React UI + Tauri desktop shell — 149 TypeScript/TSX filesstt/: Rust speech-to-text sidecar (based on whisper.cpp)packaging/: macOS DMG and Windows installer builds, auto-update manifesttests/: 78 backend test modules
This is not a demo project. This is a downloadable, installable desktop application.
3. Core Design Philosophy: Not “How Smart,” but “Does It Deliver”
The first line of OpenWorker’s README is deliberately understated: “AI that gets your everyday tasks done.”
The subtext: OpenWorker doesn’t compete on who is smarter — it competes on who is more reliable.
3.1 From “Prompt” to “Outcome”
Traditional AI usage revolves around writing prompts. You write “write me a summary,” and AI gives you a summary. If it’s not good, you rewrite the prompt and try again. OpenWorker changes this interaction model — you give an outcome, not a prompt.
You say “prepare the briefing for Monday’s client renewal call,” and it doesn’t just return text for you to organize yourself. Instead:
- Check HubSpot CRM for customer data
- Read relevant Slack conversations
- Analyze usage trends
- Generate an HTML/PDF briefing file
- Request your approval before emailing the brief to the client’s CTO
The final delivery is a finished product — a document you can open, a message already sent, a calendar already updated.
3.2 Four Agent Roles, Not One Universal Prompt
OpenWorker distinguishes four agent roles in the backend, each with its own toolset, system prompt, and behavioral model:
# coworker/agent.py - Four Agent Role Definitions (illustrative)
AGENT_ROLES = {
"chat": {
"description": "Pure conversation, no file or shell access",
"tools": ["web_search", "read_url"],
"system_prompt": "You are a friendly conversation assistant."
},
"code": {
"description": "Code-focused agent with single-dir workspace",
"tools": [
"read_file", "write_file", "shell",
"git_clone", "git_commit", "git_push",
"grep_search", "list_files"
],
"system_prompt": (
"You are a senior engineer. Read code before modifying, "
"verify after changes. Batch independent read/grep requests "
"concurrently rather than one at a time."
),
"workspace": "single_dir"
},
"coworker": {
"description": "Knowledge work agent, delivers finished artifacts",
"tools": [
"read_file", "write_file", "web_search",
"connector:slack", "connector:gmail",
"connector:calendar", "connector:github",
"connector:jira", "connector:notion"
],
"system_prompt": (
"Your task is to deliver usable finished files, not text replies. "
"Before sending messages, modifying calendars, or running commands, "
"you must request user approval."
),
"workspace": "multi_dir"
},
"myhelper": {
"description": "Persistent personal assistant, cross-time continuous thread",
"tools": ["read_file", "write_file", "web_search",
"connector:slack", "connector:gmail",
"connector:calendar", "scheduler"],
"system_prompt": "You are a long-term assistant, remember important items.",
"persistent": True
}
}
Four roles, four toolkits, four personalities. Different jobs get different personas — far cleaner than one universal prompt trying to handle everything. This design embodies the Separation of Concerns principle in software engineering.
4. Four-Layer Architecture Deep Dive
OpenWorker’s architecture follows a clean layered design, all running on the user’s local machine.
┌──────────────────────────────────────────────────────────────────┐
│ OpenWorker Four-Layer Architecture │
│ │
│ ┌────────────────────────────────────────────────────────┐ │
│ │ Layer 1: Desktop Shell (Tauri 2 + React 18) │ │
│ │ ┌────────────────────────────────────────────────┐ │ │
│ │ │ React UI (Composer, Transcript, Onboarding) │ │ │
│ │ ├────────────────────────────────────────────────┤ │ │
│ │ │ Tauri Rust Shell (Process Mgmt, Tray, STT) │ │ │
│ │ │ - Launch/monitor Python sidecar │ │ │
│ │ │ - Local speech-to-text (ocw-stt, whisper.cpp) │ │ │
│ │ │ - System tray & keep-awake (caffeinate) │ │ │
│ │ └────────────────────────────────────────────────┘ │ │
│ ├────────────────────────────────────────────────────────┤ │
│ │ Layer 2: Local Agent Server (Python FastAPI+uvicorn) │ │
│ │ ┌────────────────────────────────────────────────┐ │ │
│ │ │ TurnEngine (Core Execution Loop) │ │ │
│ │ │ ├─ SessionManager │ │ │
│ │ │ ├─ PermissionEngine (Approval Gate Engine) │ │ │
│ │ │ ├─ ToolRegistry (Tool Dispatch) │ │ │
│ │ │ └─ ProviderRouter (Model Routing) │ │ │
│ │ └────────────────────────────────────────────────┘ │ │
│ │ Bind: 127.0.0.1:8765 · Default 12 model-tool turns │ │
│ ├────────────────────────────────────────────────────────┤ │
│ │ Layer 3: Capability & Connector Layer │ │
│ │ ┌────────────────────┬─────────────────────────────┐ │ │
│ │ │ Local Tools │ External Connectors │ │ │
│ │ │ ├─ Filesystem │ ├─ Slack, GitHub, Jira │ │ │
│ │ │ ├─ Shell Execution │ ├─ Gmail, Outlook, Cal │ │ │
│ │ │ ├─ Git Operations │ ├─ Notion, Linear, │ │ │
│ │ │ ├─ ripgrep Search │ │ HubSpot, Attio │ │ │
│ │ │ ├─ Todo List │ ├─ Google Drive, Dropbox │ │ │
│ │ │ └─ MCP Client │ ├─ Asana, Monday.com │ │ │
│ │ │ │ └─ MCP Server (any) │ │ │
│ │ └────────────────────┴─────────────────────────────┘ │ │
│ ├────────────────────────────────────────────────────────┤ │
│ │ Layer 4: Model Router (aisuite) │ │
│ │ ┌────────────────────────────────────────────────┐ │ │
│ │ │ ProviderRouter → ProviderDescriptor → Client │ │ │
│ │ │ ├─ OpenAI / Anthropic / Google (native) │ │ │
│ │ │ ├─ DeepSeek / GLM / Kimi / Qwen / MiniMax │ │ │
│ │ │ │ Mistral / Grok (compatible) │ │ │
│ │ │ ├─ Together / Fireworks (open-weight) │ │ │
│ │ │ └─ Ollama (fully local) │ │ │
│ │ └────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────┘ │
│ │
│ Everything runs locally · Only cloud: OAuth handshake broker │
└──────────────────────────────────────────────────────────────────┘
4.1 Layer 1: Desktop Shell (Tauri 2 + React 18)
The desktop shell is the user’s entry point to OpenWorker. Tauri 2 provides native windows, system tray, and process management. React 18 provides the UI component layer.
Key design: The Tauri shell manages the complete lifecycle of the Python sidecar — launch, monitor, terminate. If the GUI crashes, the sidecar detects the parent process has exited and terminates itself, preventing orphaned processes.
# coworker/server/run.py - Sidecar Process Management (illustrative)
import os
import signal
import threading
def _exit_when_orphaned():
"""Detect if parent process is alive; exit if orphaned"""
parent_pid = os.getppid()
while True:
try:
# POSIX: os.kill(pid, 0) checks process liveness
os.kill(parent_pid, 0)
threading.Event().wait(2.0)
except ProcessLookupError:
# Parent terminated, exit self
os._exit(0)
except Exception:
threading.Event().wait(2.0)
def _watch_parent_windows(parent_handle):
"""Windows: use WaitForSingleObject to monitor parent"""
import ctypes
kernel32 = ctypes.windll.kernel32
while True:
ret = kernel32.WaitForSingleObject(parent_handle, 2000)
if ret == 0: # Parent process ended
os._exit(0)
4.2 Layer 2: Local Agent Server (Python FastAPI)
This is the “brain” of the entire system. The FastAPI server binds to 127.0.0.1:8765, providing roughly 120 REST endpoints and a WebSocket interface for streaming agent events.
The core component is the TurnEngine — implementing the “model-in-the-loop” execution pattern. A single user request may involve multiple rounds of model inference and tool calls:
# coworker/engine.py - TurnEngine Core Loop (illustrative)
class TurnEngine:
"""Execution engine for a single user turn"""
def __init__(self, provider_client, tool_registry, permission_engine):
self.provider = provider_client
self.tools = tool_registry
self.permissions = permission_engine
self.max_iterations = 12
async def run(self, user_input: str, history: list[dict]):
messages = history + [{"role": "user", "content": user_input}]
for iteration in range(self.max_iterations):
# 1. Model inference
response = await self.provider.complete(
messages, tools=self.tools.schemas
)
if response.tool_calls:
# 2. Execute low-risk tools in parallel,
# high-risk tools sequentially
results = await self._execute_tools(response.tool_calls)
messages.extend(results)
else:
# 3. Final response from model, end loop
return response.content
raise MaxIterationsError(
f"Reached max iterations {self.max_iterations}"
)
async def _execute_tools(self, tool_calls: list):
"""Execute tools based on risk level"""
read_calls = []
write_calls = []
for tc in tool_calls:
risk = self.permissions.classify(tc)
if risk == RiskLevel.READ:
read_calls.append(tc)
else:
write_calls.append(tc)
# Low risk: parallel execution
read_results = await asyncio.gather(
*[self.tools.execute(tc) for tc in read_calls]
)
# High risk: sequential with approval gates
write_results = []
for tc in write_calls:
outcome = await self.permissions.request_approval(tc)
if outcome == ApprovalOutcome.APPROVED:
write_results.append(
await self.tools.execute(tc)
)
else:
write_results.append(
{"error": "User rejected execution"}
)
return read_results + write_results
4.3 Layer 3: Capability & Connector Layer
This layer divides into local tools and external connectors. Local tools include filesystem operations (read_file/write_file/list_files), shell execution, Git operations, ripgrep search, and more. External connectors are OpenWorker’s core strength — 25+ built-in connectors plus MCP protocol extension.
4.4 Layer 4: Model Router (aisuite)
The bottom layer is the model router built on aisuite. aisuite is a lightweight Python library developed by Andrew Ng’s team that unifies the API calls of multiple LLM providers. OpenWorker adds agent capabilities on top — tool calling, state management, and task orchestration.
┌─────────────────────────────────────────────────────┐
│ ProviderRouter Model Flow │
│ │
│ User selects: "Claude Fable 5" │
│ │ │
│ ▼ │
│ model_labels() lookup │
│ │ │
│ ▼ │
│ Model ID: "anthropic:claude-fable-5" │
│ │ │
│ ▼ │
│ ProviderRouter.get_client("anthropic") │
│ │ │
│ ▼ │
│ registry.py: build_provider_client() │
│ │ │
│ ▼ │
│ ProviderDescriptor.build() → AnthropicProvider │
│ │ │
│ ▼ │
│ TurnEngine.process_turn() begins execution │
└─────────────────────────────────────────────────────┘
5. Approval Gate Mechanism: The Core Innovation
This is the most technically interesting part of OpenWorker. An agent that actually modifies your files, sends your messages, and runs your terminal commands — the worst thing it can do is act on its own. Most desktop agent projects treat approval as a UI afterthought. OpenWorker makes it a type system.
5.1 Four Risk Levels
Every tool call is annotated with a risk level at registration time:
# coworker/permissions.py - Risk Levels & Approval Gate (illustrative)
from enum import IntEnum
class RiskLevel(IntEnum):
"""Tool call risk classification"""
READ = 1 # Read-only: no side effects, always allowed
WRITE_LOCAL = 2 # Write local: mutates workspace, path-scoped
EXEC = 3 # Execute: runs shell commands
EXTERNAL = 4 # External: side effects off the machine
# Tool risk registry
TOOL_RISK_MAP = {
# Read-only - auto-approve
"read_file": RiskLevel.READ,
"list_files": RiskLevel.READ,
"grep_search": RiskLevel.READ,
"web_search": RiskLevel.READ,
"connector:slack_read": RiskLevel.READ,
# Write local - needs approval (path-scoped)
"write_file": RiskLevel.WRITE_LOCAL,
"create_file": RiskLevel.WRITE_LOCAL,
"git_commit": RiskLevel.WRITE_LOCAL,
# Shell execution - high risk, always needs approval
"shell": RiskLevel.EXEC,
"run_script": RiskLevel.EXEC,
# External - needs approval, but can set standing rules
"connector:slack_send": RiskLevel.EXTERNAL,
"connector:gmail_send": RiskLevel.EXTERNAL,
"connector:calendar_update": RiskLevel.EXTERNAL,
}
5.2 Five Operating Modes
OpenWorker defines five operating modes that determine how the same risk level is handled in different scenarios:
# coworker/permissions.py - Operating Modes (illustrative)
class Mode(IntEnum):
DISCUSS = 0 # Pure conversation, read-only
PLAN = 1 # Read-only + planning, no modifications
INTERACTIVE = 2 # Default: reads auto-approve, writes need approval
AUTO = 3 # Full auto-approve, but writes path-scoped
CUSTOM = 4 # User-defined whitelist-based auto-approve
class PermissionEngine:
"""Approval gate engine"""
def __init__(self, mode: Mode = Mode.INTERACTIVE):
self.mode = mode
self.standing_rules: dict[str, str] = {}
def check_tool(self, tool_call: ToolCall) -> PermissionResult:
"""Check if a tool call needs approval"""
risk = TOOL_RISK_MAP.get(tool_call.name, RiskLevel.WRITE_LOCAL)
# Read-only: always allowed in all modes
if risk == RiskLevel.READ:
return PermissionResult.ALLOW
# Check standing rules
rule_key = f"{tool_call.name}:{tool_call.args.get('target', '')}"
if rule_key in self.standing_rules:
return PermissionResult.ALLOW
# Mode-based decisions
if self.mode == Mode.DISCUSS or self.mode == Mode.PLAN:
if risk > RiskLevel.READ:
return PermissionResult.DENY
if self.mode == Mode.INTERACTIVE:
if risk >= RiskLevel.WRITE_LOCAL:
return PermissionResult.NEEDS_USER
if self.mode == Mode.AUTO:
if risk == RiskLevel.EXEC:
return PermissionResult.NEEDS_USER # Shell always needs approval
return PermissionResult.ALLOW
return PermissionResult.NEEDS_USER
5.3 Approval Gate Flow
┌─────────────────────────────────────────────────────────────────────┐
│ Approval Gate Execution Flow │
│ │
│ TurnEngine PermissionEngine │
│ ┌─────────────┐ ┌───────────────┐ │
│ │ Model │ │ │ │
│ │ requests │ │ │ │
│ │ tool call │ │ │ │
│ └──────┬──────┘ │ │ │
│ │ │ │ │
│ ▼ │ │ │
│ ┌──────────────┐ │ │ │
│ │ check_tool() │────────────►│ risk=EXEC │ │
│ │ risk check │ │ needs_user? │ │
│ └──────┬───────┘ │ │ │
│ │ │ │ │
│ ▼ │ │ │
│ ┌──────────────────┐ │ │ │
│ │ Result: NEEDS_USER│ │ │ │
│ └────────┬─────────┘ │ │ │
│ │ │ │ │
│ ▼ │ │ │
│ ┌──────────────────────┐ │ │ │
│ │ Emit PERMISSION_ │ │ │ │
│ │ REQUIRED event to UI │ │ │ │
│ └──────────┬───────────┘ │ │ │
│ │ │ │ │
│ ▼ │ │ │
│ ┌──────────────────────┐ │ │ │
│ │ User chooses: │ │ │ │
│ │ ├─ ONCE: this time │ │ │ │
│ │ ├─ ALWAYS_TOOL: │ │ │ │
│ │ │ remember tool+dest│ │ │ │
│ │ └─ DENY: reject │ │ │ │
│ └──────────┬───────────┘ │ │ │
│ │ │ │ │
│ ▼ │ │ │
│ ┌──────────────────────┐ │ │ │
│ │ ApprovalOutcome.ONCE │ │ │ │
│ │ → Execute tool call │ │ │ │
│ └──────────────────────┘ │ │ │
│ │
│ Key Design Decisions: │
│ • Standing rules only open for EXTERNAL risk class │
│ • Shell commands ALWAYS need approval, never bypassed │
│ • Unattended mode does not raise autonomy ceiling │
│ → approval requests go to Inbox instead │
└─────────────────────────────────────────────────────────────────────┘
Two design decisions stand out:
Unattended mode does not raise autonomy: When the agent runs unattended and encounters an action requiring approval, it does not act on its own. Instead, the request goes to the Inbox, and the session suspends until you return. The agent does not quietly gain permissions because you walked away.
Standing rules restricted to EXTERNAL only: When an external action (e.g., “send message to Slack #checkout-alerts”) is approved, the engine can remember “this tool to this target” and skip asking next time. But this shortcut is only available for EXTERNAL risk. Shell commands always require approval, every single time. The safety boundary is drawn at the most dangerous actions, not where it’s convenient.
6. Connector Architecture: 35+ Integrations
Another core capability of OpenWorker is its connector ecosystem. It ships with 25+ built-in connectors out of the box, plus MCP protocol extension, reaching over 35 in practice.
┌─────────────────────────────────────────────────────────────────────┐
│ OpenWorker Connector Architecture │
│ │
│ ConnectorManager │
│ ┌──────────────────────┐ │
│ │ OAuth Token Manager │ │
│ │ Credential Vault │ │
│ │ Connector Registry │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌──────────┬──────────┬─────┼──────────┬──────────┬────────┐ │
│ │ │ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ ▼ │ │
│ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ │ │
│ │Slack │ │GitHub│ │ Jira │ │Gmail │ │Calendar│ │Notion│ │ │
│ ├──────┤ ├──────┤ ├──────┤ ├──────┤ ├───────┤ ├──────┤ │ │
│ │Send │ │PR │ │Issue │ │Read │ │Read │ │Read │ │ │
│ │Msg │ │Create│ │Create│ │Send │ │Create │ │Write │ │ │
│ │Read │ │Read │ │Search│ │Search│ │Update │ │Search│ │ │
│ └──────┘ └──────┘ └──────┘ └──────┘ └───────┘ └──────┘ │ │
│ │
│ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ │
│ │Linear│ │HubSpot│ │Outlook│ │Drive │ │Dropbox│ │Asana │ │
│ ├──────┤ ├───────┤ ├───────┤ ├──────┤ ├───────┤ ├──────┤ │
│ │Issue │ │CRM │ │Email │ │File │ │File │ │Task │ │
│ │Mgmt │ │Data │ │Cal │ │Ops │ │Sync │ │Mgmt │ │
│ └──────┘ └───────┘ └───────┘ └──────┘ └───────┘ └──────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ MCP Client (Model Context Protocol) │ │
│ │ Any MCP Server can be connected, with per-tool control │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ │ Database │ │ Browser │ │ Custom │ │ File │ │ │
│ │ │ │ │ │ │ API │ │ System │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ Credential storage: All in local SecretStore │
│ Only cloud component: OAuth handshake broker │
│ Using Auth0 Authorization Code + PKCE flow │
│ Tokens go directly to the machine, never stored in cloud │
└─────────────────────────────────────────────────────────────────────┘
Connectors follow a unified interface protocol. Each connector exposes a set of tools (e.g., the Slack connector exposes slack_send_message, slack_read_channel, slack_search), which are registered with the ToolRegistry for the agent engine to call.
Key design decision: All connector credentials are stored in the local SecretStore. The only cloud component is an optional OAuth broker that uses the Auth0 Authorization Code + PKCE flow for one-click connection handshakes. Tokens are handed directly to the machine and are never stored in the cloud. Users can also skip sign-in entirely and use connectors with manually pasted API keys.
7. Model-Agnostic Architecture
OpenWorker is truly model-agnostic. It does not depend on any specific model provider, nor does it run its own inference service. You bring your own API key, or go fully local with Ollama.
7.1 30 Curated Models + Custom Extension
# coworker/providers/matrix.py - Model Matrix (illustrative)
from dataclasses import dataclass
@dataclass
class ModelCapabilities:
tools: bool = True
vision: bool = False
parallel_tool_calls: bool = True
streaming: bool = True
pdf: bool = False
# Curated model matrix
MATRIX = {
# OpenAI native
"gpt-5.6-sol": ModelEntry("GPT-5.6 Sol · OpenAI", _AGENTIC_VISION),
"gpt-5.6-terra": ModelEntry("GPT-5.6 Terra · OpenAI", _AGENTIC_VISION),
"gpt-5.5": ModelEntry("GPT-5.5 · OpenAI", _AGENTIC_VISION),
# Anthropic native
"anthropic:claude-fable-5": ModelEntry("Claude Fable 5 · Anthropic", _AGENTIC_VISION),
"anthropic:claude-opus-4.8": ModelEntry("Opus 4.8 · Anthropic", _AGENTIC_VISION),
"anthropic:claude-sonnet-4.6":ModelEntry("Sonnet 4.6 · Anthropic", _AGENTIC_VISION),
# Google native
"gemini:gemini-3.6-flash": ModelEntry("Gemini 3.6 Flash · Google", _AGENTIC_VISION),
"gemini:gemini-3.1-pro": ModelEntry("Gemini 3.1 Pro · Google", _AGENTIC_VISION),
# OpenAI-compatible vendors
"zai:glm-5.2": ModelEntry("GLM-5.2 · Z AI", _AGENTIC),
"deepseek:deepseek-v4": ModelEntry("DeepSeek V4", _AGENTIC),
"moonshot:kimi-k2.6": ModelEntry("Kimi K2.6 · Moonshot", _AGENTIC),
"minimax:m2.5": ModelEntry("MiniMax M2.5", _AGENTIC),
"qwen:qwen3-max": ModelEntry("Qwen3 Max · Alibaba", _AGENTIC_VISION),
"xai:grok-4.3": ModelEntry("Grok 4.3 · xAI", _AGENTIC),
"mistral:mistral-large": ModelEntry("Mistral Large", _AGENTIC),
# Open-weight (via Together / Fireworks)
"together:thinkingmachines/Inkling": ModelEntry("Inkling · via Together", _AGENTIC),
# Fully local (Ollama)
"ollama:llama-4": ModelEntry("Llama 4 · Ollama", _AGENTIC),
}
7.2 ProviderRegistry Factory Pattern
Model routing uses a factory pattern, mapping UI configuration to concrete ProviderClient instances via ProviderDescriptor:
# coworker/providers/registry.py - Provider Registry (illustrative)
from dataclasses import dataclass, field
@dataclass
class ProviderField:
"""Defines a single configuration input field"""
key: str
label: str
type: str = "string" # string, password, select
secret: bool = False
required: bool = True
placeholder: str = ""
@dataclass
class ProviderDescriptor:
"""Provider descriptor: config fields + build function"""
name: str
label: str
fields: list[ProviderField] = field(default_factory=list)
build: callable = None # (profile, secrets) -> ProviderClient
# Register all providers
DESCRIPTORS = {
"openai": ProviderDescriptor(
name="openai",
label="OpenAI",
fields=[
ProviderField(key="api_key", label="API Key", secret=True),
ProviderField(key="base_url", label="Base URL",
required=False,
placeholder="https://api.openai.com/v1"),
],
build=lambda p, s: OpenAIProvider(
api_key=s["api_key"],
base_url=p.get("base_url")
),
),
"anthropic": ProviderDescriptor(
name="anthropic",
label="Anthropic",
fields=[
ProviderField(key="api_key", label="API Key", secret=True),
],
build=lambda p, s: AnthropicProvider(api_key=s["api_key"]),
),
"ollama": ProviderDescriptor(
name="ollama",
label="Ollama (Local)",
fields=[
ProviderField(key="base_url", label="Base URL",
required=False,
placeholder="http://localhost:11434/v1"),
],
build=lambda p, s: OllamaProvider(
base_url=p.get("base_url", "http://localhost:11434/v1")
),
),
}
def build_provider_client(provider_name: str, profile: dict, secrets: dict):
"""Factory: build a client by provider name"""
desc = DESCRIPTORS.get(provider_name)
if not desc:
raise ValueError(f"Unknown provider: {provider_name}")
return desc.build(profile, secrets)
7.3 Local-First Configuration Storage
# config.toml - OpenWorker Configuration Example
# Global: ~/.config/coworker/config.toml
# Workspace: <project>/.coworker/config.toml (overrides global)
model = "anthropic:claude-sonnet-4.6" # Default model
mode = "interactive" # Default approval mode
max_iterations = 12 # Max turns per round
# Shell commands that don't need approval
allowed_commands = [
"ls",
"cat",
"pwd",
"git status",
"git diff",
]
# Auto-approved tools in custom mode
auto_allow = [
"write_file:reports/",
"connector:slack_read",
]
[server]
host = "127.0.0.1"
port = 8765
8. Slack Integration: From Message to Deliverable
Slack is one of OpenWorker’s most important entry points. Mention @OpenWorker in a Slack channel, a session opens on your desktop, work runs with your local tools, and the result comes back as a thread reply.
┌─────────────────────────────────────────────────────────────────────┐
│ Slack → Agent → Connectors → Deliverable Pipeline │
│ │
│ You: @OpenWorker checkout API is 500ing │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Slack (thread reply) │ │
│ │ "Got it, investigating..." │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ OpenWorker Desktop (Local Agent Engine) │ │
│ │ │ │
│ │ Step 1: Check recent deploys (GitHub Connector) │ │
│ │ └─→ Read: github.com/org/repo/deployments │ │
│ │ ← Found: 2:02am deploy payment-migration v3.8 │ │
│ │ │ │
│ │ Step 2: Check error logs (Local Shell) │ │
│ │ └─→ Run: grep "500" /var/log/app/error.log │ │
│ │ ← Found: errors from 2:04am, peak 18.4% │ │
│ │ │ │
│ │ Step 3: Cross-reference Runbook (Local File) │ │
│ │ └─→ Read: docs/runbook.md │ │
│ │ ← Confirmed: rollback conditions met │ │
│ │ │ │
│ │ Step 4: Generate incident timeline (Deliverable) │ │
│ │ └─→ Write: reports/checkout-incident-timeline.md │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Approval Gate: "Send this report to #checkout-alerts?" │ │
│ │ [Approve] [Not now] │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Slack (thread reply) │ │
│ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ Incident Timeline Report (sent) │ │ │
│ │ │ Service: checkout │ │ │
│ │ │ Peak error rate: 18.4% │ │ │
│ │ │ Impact duration: 10 minutes │ │ │
│ │ │ Root cause: payment-migration v3.8 │ │ │
│ │ │ connection pool regression │ │ │
│ │ │ Recommended action: rollback v3.8, │ │ │
│ │ │ monitor for 10 minutes │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ Key: Slack as entry point, but agent runs locally. │
│ Credentials and model calls stay on your machine. │
└─────────────────────────────────────────────────────────────────────┘
This pipeline illustrates OpenWorker’s core differentiator from traditional AI assistants: it doesn’t “reply” to you — it “completes” the task. You only said “checkout API is 500ing,” and it autonomously performed deployment inspection, log analysis, runbook cross-referencing, document generation, approval request, and message sending — the entire closed loop.
9. Scheduled Tasks & Unattended Mode
OpenWorker includes a built-in scheduler for recurring automations: morning briefings, weekly reports, and standing watches on channels.
┌─────────────────────────────────────────────────────────────────────┐
│ Scheduled Task Architecture │
│ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Scheduler (Persistent) │ │
│ │ ┌────────────────────┐ ┌──────────────────────────────┐ │ │
│ │ │ Cron Expression │ │ Task Queue (SQLite) │ │ │
│ │ │ Parser │ │ │ │ │
│ │ └────────────────────┘ └──────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ (Schedule trigger) │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Task Executor │ │
│ │ ├─ Create new session │ │
│ │ ├─ Inject user's preset prompt │ │
│ │ ├─ Run TurnEngine │ │
│ │ ├─ Approval requests → Inbox │ │
│ │ │ └─ User not online → session suspended │ │
│ │ └─ Save full transcript locally │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Inbox │ │
│ │ ┌──────────────────────────────────────────────────┐ │ │
│ │ │ ⏰ 08:00 Morning Brief → Pending: Send Slack │ │ │
│ │ │ ⏰ 09:30 Standup Report → Pending: Update Jira │ │ │
│ │ │ 👀 Channel Watch: #alerts → Pending: Send │ │ │
│ │ └──────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ Missed tasks during downtime: auto-catch-up on next startup │
│ Keep-awake: caffeinate (macOS) / SetThreadExecutionState (Win) │
│ Prevents sleep during task execution │
└─────────────────────────────────────────────────────────────────────┘
The power of scheduled tasks lies in their integration with the approval gate mechanism. Even when tasks run unattended, actions requiring approval are not auto-executed — they are placed in the Inbox for the user to handle. Missed tasks during downtime are automatically caught up on the next startup. The system also uses caffeinate (macOS) or SetThreadExecutionState (Windows) to prevent sleep during task execution.
10. Local-First Design
OpenWorker’s local-first design is its core security promise.
10.1 What Stays Local
┌─────────────────────────────────────────────────────────────────────┐
│ Local-First: Data Residency Boundary │
│ │
│ ✅ Stays Local (Never Leaves) ❌ May Leave Machine │
│ ┌────────────────────────────┐ ┌────────────────────────┐ │
│ │ Agent loop (TurnEngine) │ │ Model API calls │ │
│ │ Conversation store │ │ (to your chosen LLM) │ │
│ │ Connector credentials │ │ Connector data R/W │ │
│ │ Model API keys │ │ (to your SaaS tools) │ │
│ │ Generated files │ │ OAuth handshake (relay)│ │
│ │ Config files (config.toml) │ └────────────────────────┘ │
│ │ SQLite session database │ │
│ │ Scheduled task queue │ │
│ └────────────────────────────┘ │
│ │
│ SecretStore Security Design: │
│ • Keys never enter model context, prompts, or traces │
│ • All sensitive data in local encrypted storage │
│ • Fully offline capable (manual API key paste) │
│ │
│ OAuth Handshake Flow: │
│ User → OAuth Provider → Relay → Direct to Machine (no cloud store) │
│ Using Auth0 Authorization Code + PKCE flow │
└─────────────────────────────────────────────────────────────────────┘
10.2 Why Local-First Matters
In the enterprise space, data sovereignty is a hard requirement. Many organizations hesitate to adopt AI agents because data must be uploaded to the cloud. OpenWorker’s local-first design addresses this directly:
- Model calls go directly from your machine to the LLM provider, with no intermediary relay server
- Connector tokens are handed directly to your machine, never stored on OpenWorker’s servers
- Fully offline capable: manually paste API keys and credentials, no cloud services needed
- OAuth relay uses Auth0 Authorization Code + PKCE, tokens handed to machine after handshake
The only cloud component is an optional OAuth relay for one-click connection handshakes. Tokens go directly to the machine, never stored in the cloud.
11. Comparison with Similar Tools
OpenWorker was born in late July 2026, a time when desktop AI agents were already abundant. But each has a different positioning.
┌─────────────────────────────────────────────────────────────────────┐
│ Tool Comparison: Positioning Determines Design │
│ │
│ Tool │ Target User │ Core Scenario │ Runtime │ Model │
│ ─────────────────────────────────────────────────────────────────── │
│ OpenWorker │ Knowledge │ Cross-tool │ Desktop │ Any │
│ │ Workers │ Deliverables │ (Tauri) │ │
│ ─────────────────────────────────────────────────────────────────── │
│ Claude Code │ Developers │ Code editing │ Terminal │ Claude│
│ │ │ Repo-level │ + VS Code│ Only │
│ ─────────────────────────────────────────────────────────────────── │
│ Codex CLI │ Developers │ Code editing │ Cloud │ OpenAI│
│ │ │ Sandbox exec │ Sandbox │ Only │
│ ─────────────────────────────────────────────────────────────────── │
│ DeepSeek Harness │ Developers │ Plugin-based │ Local CLI│ Any │
│ │ │ Everything │ │ │
│ │ │ is a plugin │ │ │
│ ─────────────────────────────────────────────────────────────────── │
│ Hermes Agent │ Full-stack │ Content pipes │ CLI+Msg │ Any │
│ │ Automation │ Server ops │ Platform │ │
│ ─────────────────────────────────────────────────────────────────── │
│ │
│ Core Difference: │
│ • Claude Code: excels at "editing code" — read, modify, test │
│ • Codex: excels at "sandbox execution" — isolated environments │
│ • OpenWorker: excels at "filling gaps" — across Jira+GitHub+Slack │
│ + calendar, synthesizing data into documents, approving before │
│ execution │
│ │
│ Not competitors — complementary tools for different scenarios │
└─────────────────────────────────────────────────────────────────────┘
Detailed Comparison Table
| Dimension | OpenWorker | Claude Code | Codex CLI | DeepSeek Harness |
|---|---|---|---|---|
| Desktop GUI | ✅ Tauri native | ❌ Terminal only | ❌ Terminal+VS Code | ❌ Terminal only |
| Model-agnostic | ✅ 30+ models | ❌ Anthropic only | ❌ OpenAI only | ✅ Any |
| Tool integrations | 25+ connectors+MCP | MCP+FS+Browser | Sandbox+FS | Everything is a plugin |
| Local-first | ✅ All local | ❌ | ❌ | ✅ |
| Approval gates | ✅ Typed risk system | ✅ Approval | ✅ Approval | ⚠️ Basic |
| Scheduled tasks | ✅ Built-in scheduler | ❌ | ❌ | ❌ |
| Speech input | ✅ Rust STT | ❌ | ❌ | ❌ |
| Open source | MIT | Closed | Apache 2.0 | MIT |
| Primary focus | Knowledge worker tasks | Code editing | Code editing | Plugin agent framework |
12. Limitations
An objective evaluation must acknowledge the limitations. OpenWorker is currently in open beta, and the following issues deserve attention:
12.1 Production Environment Caution
Although OpenWorker is MIT-licensed, the project is less than one month old (at the time of writing). The community ecosystem is still in its early stages. While the repository has 179 Issues and 250 Pull Requests, code maturity, documentation completeness, and community support are all evolving rapidly.
12.2 Windows Build Lacks Code Signing
The Windows 10/11 x64 build is downloadable and usable, but since it has not yet been code-signed, Windows SmartScreen will display a security warning. For enterprise users, this is a risk factor that requires careful evaluation. The macOS build is signed, notarized, and supports auto-updates.
12.3 No Linux Desktop Build
Currently only macOS (Apple Silicon) and Windows (x64) builds are available. Linux users need to build from source or use browser mode (npm run dev) for the UI.
12.4 Integration Permissions Require Self-Configuration
While OpenWorker provides 25+ connectors, each connector’s OAuth authentication and permission scopes must be configured by the user. For non-technical users, understanding different tools’ permission models (e.g., Slack Bot Token scopes, Google API OAuth scopes) may present a learning curve.
12.5 Local Small Model Task Decomposition Quality
OpenWorker’s model-agnostic design allows it to use Ollama local models. However, practical testing shows that local small models (7B-13B parameter range) exhibit significant quality degradation in complex scenarios like task decomposition and tool call planning. This is a ceiling of current small model capabilities, not an OpenWorker-specific issue, but users need to be aware of this limitation.
13. Industry Significance & Outlook
13.1 Approval Gates as Default Design
OpenWorker’s most enduring contribution may be making approval gates a default design of AI agents.
Before OpenWorker, most agent frameworks treated security as a patch — let the agent work first, then figure out how to contain it. OpenWorker reverses this: security is a first-class citizen of the architecture. Every tool call is annotated with a risk level at registration, five operating modes define behavior in different scenarios, shell commands always need approval, and unattended mode does not raise the autonomy ceiling.
This design philosophy should become the reference standard for all AI agents that “can do real work.” An agent without gates is more dangerous the more capable it becomes.
13.2 From “Dialogue” to “Deliverable” Paradigm Shift
OpenWorker’s true innovation is not the technical architecture (though it is well-designed), but the shift in product philosophy.
It redefines what it means for AI to “complete a task” — not that AI replied with text, but that AI delivered a usable finished product. This standard sounds simple, but it has profound implications for product design, architecture design, and security design:
- For product design: interaction shifts from “writing a prompt” to “describing an outcome”
- For architecture design: requires connectors, approval gates, local storage, filesystem integration
- For security design: requires a typed risk system, operating modes, standing rules
13.3 Local-First = Trust-First
In an era of growing AI agent trust crises (data leaks, prompt injection, permission abuse), OpenWorker’s local-first design provides a clear answer: Trust is not built on promises — it is built on architecture. When the agent loop, conversation history, connector credentials, and model keys all live on the local machine, when model calls go directly from your machine to the LLM provider, when the only cloud component is an OAuth handshake relay — trust is no longer a “believe us” promise, but a verifiable fact.
14. Conclusion
OpenWorker is not a groundbreaking model, nor a disruptive technological breakthrough. It is a product that did the right things correctly — security as an architectural design, finished deliverables as the standard, local-first as the default deployment, and models as replaceable components.
In the “Warring States era” of AI agents, OpenWorker may not emerge as the ultimate winner. But it has defined a new baseline: AI agents should deliver finished work, not conversation; they should ask before acting, not act before asking; they should run locally, not be locked in the cloud.
For developers, OpenWorker’s codebase is a reference implementation worth studying in depth — especially its PermissionEngine, TurnEngine, ProviderRegistry, and connector framework. For knowledge workers, it is a desktop AI coworker worth trying — though still in beta, the direction is already clear.
Andrew Ng and Rohit Prasad made a subtle but important choice: upgrade AI from a “chat tool” to an “execution tool.” This choice may change how we collaborate with AI more than any model capability improvement ever could.
This article is based on the OpenWorker GitHub repository (https://github.com/andrewyng/openworker) and publicly available technical documentation. Code snippets are illustrative implementations. Please refer to the repository for the complete source code.