OpenProgram Docs

Computer and browser use#

OpenProgram gives the model one tool for everything that happens on a screen: computer_use_agent. It operates applications on macOS, Linux and Windows, pages in the App's built-in browser and the headless Chromium, and a remote virtual machine. This record explains the tool's shape, how it hands the model only the instructions it needs, how a click target becomes a screen position, how the model chooses whether the work runs in the current context, and how the application and browser backends underneath are built.

Three ways to give a model a screen#

Codex (ChatGPT.app) has one execution tool, js in the cua_repl MCP server: the model writes JavaScript against sky (Mac apps) and agent.browsers (the in-app browser, Chrome). The knowledge sits in three skills, computer-use, control-in-app-browser and control-chrome, each about 3,200–3,400 tokens. The context holds one line per skill; the model reads a SKILL.md itself when the task needs it, and the browser skill then has it run documentation() inside js to fetch the browser API reference. Loading costs two or three model calls, and a model that skips the reading does not know the API.

Claude Code has many small tools: the built-in browser's navigate, read_page, find, computer, form_input and about fifteen more, whose full schemas are always in context, and a second set for the user's Chrome, deferred as bare names until ToolSearch loads them. Every parameter is schema-checked, but two sets with the same verbs invite picking the wrong one, and the resident set costs context in every conversation.

OpenProgram takes Codex's single entry and Claude Code's structured arguments:

  • One tool. computer_use_agent is the only screen tool the model sees. Its schema is small and the same for every target; the actions and their arguments live in per-target guides.
  • Guides arrive with the first call. The tool description is a catalogue of the targets. The first call for a target returns that target's guide together with its current state (installed and running apps, open pages), so the guide costs no extra call and does not depend on the model remembering to read it. The guides are tool guides, delivered by the agent loop rather than by the tool.
  • Arguments or code. Simple steps are structured calls (action + args, or up to 20 actions). Steps with loops, conditions or data handling are a Python snippet (code) that runs in an isolated interpreter whose variables survive between calls.
  • The model chooses the context. context: "inherit" (default) runs in the current conversation; context: "fresh" starts an agent with a clean context and a written brief, which returns only its result.
  • Pointing is explicit. A click target is an element index, a page ref, or a pixel of the last screenshot.

computer_use_agent#

Shape#

computer_use_agent(target, action?, args?, actions?, code?, timeout?, context?, brief?)
Parameter Meaning
target app, browser or vm; all three are always in the schema, and a target this host cannot reach answers with how to set it up
action, args one step and its arguments; omitted, the call opens the target
actions up to 20 steps {action, args} on one target, stopping at the first error; the state comes back once
code Python run in the isolated interpreter (see Code)
timeout seconds a code call may run, at most 120 (default 60)
context inherit (default) or fresh
brief with fresh: goal, known facts, limits and what to hand back

action, actions and code are exclusive. Arguments are validated by the target, and an error names the right spelling, since the schema does not list every action's arguments.

Targets and guides#

Target Backend Opening returns
app openprogram/programs/tools/computer_use/computer_use_agent/desktop/ (macOS, Linux X11, Windows) args.app given: its state (tree and screenshot); otherwise the app list
browser the tab driver over the App's built-in tabs (below); with args.browser = chrome or edge, Playwright MCP in extension mode connected to the user's browser the tab list
vm an OSWorld-compatible server at computer_use.vm_url a screenshot

Each target has a guide (guides/<target>.md; the app guide has macOS, Linux and Windows variants): its actions, their arguments, how results look, and the confirmation policy. tool.py registers them as tool guides: computer_use_agent.<target> is included in the first result of a call that operates that target (context other than fresh); computer_use_agent.code is a lookup guide required by a call with code, so that call is refused once with the guide attached. Whether a guide is known is read from the request the agent loop last sent, so compaction, cleared tool results and a worker restart all lead to the guide being sent again, and a fresh agent, whose prompt holds the marked guides, is not sent them a second time. action: "guide" and help("computer_use_agent.<target>") return them on request.

The app actions are those of the application backend below, with the same arguments; the browser actions are the tab driver's (tabs, open, read, click, type …, see Browser target: the tab driver). As in Codex, the browser is chosen first and the actions do not change with it. args.browser omitted is the built-in browser through the tab driver; chrome or edge is the user's own browser, which the tab driver does not reach. For those, targets/browser.py hands each step to workflow/browser/extension_browser.py, which runs @playwright/mcp --extension --browser chrome|msedge (the version pinned for web_use) through _SyncMCPClient, one server per conversation and browser, kept between calls. The server opens chrome-extension://<id>/connect.html in that browser with the address of its relay, and the Playwright Extension connects back and relays CDP for the tab the user shares; with computer_use.chrome_extension_token set, the server passes PLAYWRIGHT_MCP_EXTENSION_TOKEN and the extension connects without asking. tabs, read (with tab an index, which selects that tab), open, close, screenshot, upload, evaluate and the logs map to the server's tab, snapshot, screenshot, file, evaluate and log tools; every element and pointer action goes through OfficialMCPPageBackend._action_call, the same translation the playwright_mcp backend applies to App Pages. Refs are the server's snapshot refs. Code mode registers no resource for the user's browser.

Pointing#

A step points at its target in one of two ways, both passed to the backend as they are:

Given Meaning
element_index (app) / ref (browser) the backend acts on the element
x, y pixels of the target's last screenshot

The VM has no element tree, so its pointer actions take x, y only.

Code#

code runs in the persistent sandboxed interpreter of computer_use_agent/sandbox/runner.py (seatbelt on macOS, no file, network or process access). The namespace has cu.app, cu.browser and cu.vm; await cu.app.click(app="Numbers", element_index=3) is the structured step click with those arguments, dispatched by the host through the same target code and checks, one effect receipt per step (computer_use_agent/sandbox/broker.py). The prelude gives each cu method the guide line of its action as its docstring, so dir(cu.app) and help(cu.app.click) work inside the interpreter. A code call names its target and, for app, args.app; the approval covers that application, and steps on another application are refused. evaluate is not available inside code, since every page script needs its own approval. One interpreter serves one execution attempt, so variables persist for the rest of the turn; it is replaced when it fails or times out. Each call may run 120 seconds and 100 steps. Code mode needs the macOS sandbox; elsewhere a code call reports that the sandbox is unavailable.

Context#

context: "fresh" hands the task to an agent with a clean context through the generic agent spawn (_agent_impl, start_from="clean"): same session, same model and thinking settings, the result returned as this call's result. Its prompt is the brief, a line naming the target (and args.app), and the target's guide. Its screenshots stay in its context. The spawn depth limit keeps it from starting another one.

A spawned agent cannot ask for approval, so the fresh call carries it: it names the target and, for app, args.app, and goes through the same approval as a direct call. While the agent runs, the call's exact rule is written as a session rule (unless one already exists) and removed when the agent returns, so the agent's own calls match it and nothing outlives the call.

Approval and access#

Call Approval
app (except list_apps) per application, as before: rule computer_use_agent(<app>), asked in the modes that ask; former computer_use(<app>) rules still match
browser the turn's Page grant when the conversation has Pages; otherwise the permission mode; evaluate runs in bypass; otherwise it asks, once or for this conversation
browser with args.browser the user's Chrome or Edge, once per conversation: rule computer_use_agent(browser:chrome) (or browser:edge), never the Page grant; evaluate is allowed apart from that grant (rule computer_use_agent(browser:chrome:evaluate))
vm per endpoint: rule computer_use_agent(vm:<url>)
fresh as the target it names

System access (Accessibility, Screen Recording) gates app calls other than list_apps as before.

Module layout#

Module Role
openprogram/programs/tools/computer_use/computer_use_agent/tool.py the registered tool: schema, catalogue, routing, guide registration, approval reason
…/computer_use_agent/guides/ the target guides
…/computer_use_agent/targets/app.py application actions over openprogram/programs/tools/computer_use/computer_use_agent/desktop/
…/computer_use_agent/targets/browser.py the browser actions: the tab driver (the browser service's tabs), or the user's browser (its extension)
…/computer_use_agent/targets/vm.py screenshot and pyautogui input over the VM server's HTTP API
…/computer_use_agent/code.py code mode over computer_use_agent/sandbox/runner.py and sandbox/broker.py
…/computer_use_agent/fresh.py the clean-context agent

There is no separate plan → decide → act → verify loop: a long task goes to a fresh agent working with this same tool.

Application backend#

Shape#

computer_use_agent(target="app", action, args={app, element_index | x,y, …})

Actions come in three groups:

  • Reading: list_apps, list_windows, get_app_state (include = all / tree / screenshot, window or window_id), zoom (a region of the last screenshot at full pixel density), find (text and role search over the last tree).
  • Acting, Codex's target API: click (button, count, held modifiers, hold duration), type_text, press_key, set_value, scroll (pages or pixels), drag, perform_secondary_action, select_text, paste; and from the Claude app's app_* set: copy, focus, menu, show_window; and open_file (macOS).
  • Several steps (actions): up to 20 acting steps on one app, in order, stopping at the first error; one settle and one state at the end.

Only list_apps is approval-free. A step in actions cannot name a different app, because the approval the call carries names only the call's app.

Every acting call returns Done: <summary> followed by the Accessibility tree diff against the state the model last saw: - line for elements that vanished, + index line for new or changed ones. When more than half the lines (and at least twelve) changed, the whole tree is returned instead, because a diff of a rebuilt view is longer than the view. get_app_state(full=true) returns the full tree on demand. The tree is capped at 1500 nodes and rendered one element per line: index, role (AXTextField → text field, with the subrole in parentheses), label, value, placeholder, flags (focused, selected, checked, disabled, secure, settable …) and the element's secondary actions other than the primary press. Bidirectional marks some apps wrap numbers in are dropped from the rendering and from find.

The tree covers the window, the menus, popovers and dialogs beside it, and the menu bar with only open menus expanded. Closed menus are read by the menu action, so every read covers the same ground and a diff never shows a whole menu bar appearing or vanishing.

Results that changed nothing are marked: menu listings, copy and an already-visible show_window return a Report, and the tool returns it without settling or reading the state again.

Stable indices#

tree.number gives indices against the window's previous Snapshot (its index-free lines, their indices, and the next unused index). A line that is equal, or replaced by a line of the same depth and role (a new value, a renamed button), keeps its index; any other new line takes the next unused index. Indices freed by removed elements are never reused. The model's earlier indices therefore never point at a different element after an insertion above them, which Codex's positional numbering does not guarantee.

Settling#

After an action the tool polls a cheap fingerprint of the first 400 elements every 0.2 seconds, starting after 0.3 seconds, and stops when two fingerprints agree, after 1.5 seconds, or after 5 seconds while a busy indicator or an unfinished progress indicator is present — Codex's "about a second, up to five while loading".

menu walks AXMenuBar by title, case-insensitively and ignoring a trailing … or .... A path that ends at a submenu lists its items with shortcut glyphs (⌃⌥⇧⌘ from AXMenuItemCmdModifiers), [checked] and [disabled]; a leaf is pressed with AXPress. Reading the menu tree through Accessibility opens nothing on screen. copy presses the app's own menu item whose shortcut is ⌘C, found by shortcut rather than by its localized title, and falls back to posting ⌘C; when that item exists but is disabled (nothing selected, or focus on a control) it reports so at once. Otherwise it waits for the pasteboard change count, reads the string, and restores the previous pasteboard.

copy, press_key, type_text and paste without an element_index act on the app's current focus and need no earlier get_app_state; menu, show_window and wait never need one. From Claude's computer-use tool come hover (three mouseMoved events posted to the process, approaching the point, then a 0.8-second rest; the real cursor stays put) and wait (up to 10 seconds, cancellable, followed by the usual settle and tree diff). focus sets AXFocused and waits up to half a second for the app to report it, since apps apply focus on a later run-loop turn.

The screenshot is the window's own image captured through ScreenCaptureKit, scaled so the long side is at most 1568 pixels; coordinates the model sends are pixels of that image. A step maps them to screen points with the scale that produced the image (2 on a Retina display, less when a large window was captured smaller) and the window's origin as the window server reports it at that moment (screen_point, after _follow_window), so a window the user moved since the read is still hit where the model pointed.

Text on the screenshot#

A window whose own walk yields fewer than 12 elements (frame, title bar buttons and little else) is drawn without Accessibility. For such a window every full or screenshot-only read also runs Vision's accurate text recognizer on the screenshot it just took and lists each line as "text" at (cx, cy), WxH in screenshot pixels, top to bottom and left to right, at most 150 lines. The first recognition language picks Vision's model, so the languages are a hint (the user's preferred languages, then Chinese, Japanese, Korean and English) and language detection chooses per image. find with text falls back to these lines when no element matches; for a window that was not sparse it captures and reads the window once at the same scale, so the positions stay valid for actions. OCR text has no indices and is not part of the tree diff: it is evidence for coordinate actions, and a failure of the recognizer leaves the read without it. The same reads list icons and controls without text (ocr.detect_shapes): Vision's contour detector runs twice, once for dark marks on a light window and once for light on dark, at most 1024 pixels on the long side. Top-level outlines within 3 points of each other merge into one shape, and a shape is dropped when it covers a recognized line of two or more characters, is under 8 points on its short side, over 80 points on its long side, more than 4:1, or touches the screenshot's edge (the window's rounded corners and border). A one-character line does not rule a shape out, because the recognizer often reads an icon as a glyph. Shapes the two passes both find are kept once, and the list is sorted in 16-point rows, left to right, at most 60. A shape carries no name: the model reads the screenshot to tell a gear from a trash can.

Audio#

Codex records computer audio with start_audio_recording / stop_audio_recording (60 seconds by default, 300 at most) and hands the clip to the model as audio input. OpenProgram keeps the two actions and their limits. The clip becomes a WAV on disk plus a transcript from the Settings → Voice engine, which every model can read. ToolReturn.audio also carries the WAV as an AudioContent block. When the model lists audio among its inputs, the Gemini converter passes it as inline audio and the Chat Completions converter as input_audio, each in a user turn after the run of tool results, which must stay together. That Chat Completions turn also carries the run's screenshots for vision models, since a tool message holds only text. The Responses and Anthropic converters skip audio blocks, and tool-result media is not written to the conversation history, so the clip costs nothing for models that cannot hear it. audio.py opens an SCStream whose filter is the display with only the target application included, asks for 16 kHz mono float samples, excludes the OpenProgram process, and attaches only an audio output: the 2×2, one-frame-a-second video side is never consumed. Samples past the limit are dropped and a timer stops the stream half a second after it, so a forgotten recording releases the capture. One recording runs per conversation; it is converted to 16-bit PCM on stop and saved under the state directory, keeping the newest 20. The action keeps the per-application approval: only the named app's sound is in the clip. A clip whose peak stays under 1e-4 is reported silent and not transcribed. An application without a window has no SCRunningApplication and cannot be recorded.

Module layout#

Module Role
openprogram/programs/tools/computer_use/computer_use_agent/desktop/base.py what every backend shares: ComputerUseError, Report, Observation, AudioClip, argument checks, recording storage
openprogram/programs/tools/computer_use/computer_use_agent/desktop/backends.py support() and backend() for the host: macos on Darwin, linux on Linux, windows on Windows
openprogram/programs/tools/computer_use/computer_use_agent/desktop/keys.py xdotool chord parsing to macOS key codes and modifier flags (super+shift+z, ctrl++, KP_0, lone modifiers)
openprogram/programs/tools/computer_use/computer_use_agent/desktop/tree.py Node, rendering, signatures, stable numbering (Snapshot, number) and the diff
openprogram/programs/tools/computer_use/computer_use_agent/desktop/audio.py ScreenCaptureKit audio capture of one application, sample limit, WAV encoding
openprogram/programs/tools/computer_use/computer_use_agent/desktop/ocr.py Vision text recognition (tesseract TSV on Linux), box conversion to screenshot pixels, reading order, language hints, contour shapes
openprogram/programs/tools/computer_use/computer_use_agent/desktop/skylight.py ctypes bindings to SkyLight for background pointer and keyboard routing
openprogram/programs/tools/computer_use/computer_use_agent/desktop/macos.py MacComputer: app resolution, window selection, capture, tree walk, every action
openprogram/programs/tools/computer_use/computer_use_agent/desktop/linux.py LinuxComputer: the same interface over AT-SPI, xdotool, Gdk capture and PipeWire/PulseAudio
openprogram/programs/tools/computer_use/computer_use_agent/desktop/windows.py WindowsComputer: the same interface over UI Automation (comtypes), window messages, SendInput and PrintWindow
openprogram/programs/tools/computer_use/computer_use_agent/desktop/windows_audio.py WASAPI process-loopback capture of one process tree
openprogram/programs/tools/computer_use/computer_use_agent/targets/app.py the app target: step validation, several steps, result assembly

Background delivery#

Element actions go through Accessibility: AXPress / AXConfirm / AXPick, then AXSelected, then focus, then a pointer click at the element's centre as the last resort; AXShowMenu for right clicks; AXValue for set_value; AXSelectedTextRange (UTF-16 offsets) for select_text; page scrolling actions, then the scrollbar's value, then wheel events for scroll.

Pointer events that have to be real (coordinate clicks, drags, wheel) are posted with the recipe the open-source cua-driver established: a focus-without-raise record posted to the target process serial number, a mouseMoved primer, the event stamped with click state, button, subtype 3, target pid, window id and a click group, CGEventSetWindowLocation with the point relative to the window's top left (window_location: the screen point minus the window's origin, in points, as KWWK computer-use-core and cua-driver set it; AppKit places the click in the window from the event's screen location, and a probe window took a click at the same point whether this field held the window-relative or the screen point), and SLEventPostToPid for delivery. Keyboard events carry an SLSEventAuthenticationMessage so the receiving process accepts them; without SkyLight the code falls back to CGEventPostToPid. The result is a click the application handles as if the user clicked, while the real cursor stays where it is and the front application does not change.

AppKit still orders the clicked window to the front of its level when the application handles the mouse-down, so a coordinate click or drag first lists the other applications' windows stacked over the target and overlapping it (the front application's own windows excluded), and afterwards raises them again with AXRaise, deepest first, which puts the stack back as it was.

A window is chosen by window (title substring or index) or window_id; without either, the window the model read last stays selected while it is open, then the focused or main window. Each state records the app's windows (window id, or title without one); when the focused or main window is one that was not there at the last read, the read follows it and says which window it opened over, so a dialog or document a step brought up is what the next steps act on. A new window with a dialog subrole (AXDialog, AXSystemDialog) or a new file dialog is followed even when the focus stays on the window under it, as it does when a background app opens its out-of-process file dialog.

The system Open and Save dialogs carry AXIdentifier open-panel or save-panel, as a window, an AXSheet or a child window of the window under them. A sandboxed app's dialog is drawn by com.apple.appkit.xpc.openAndSavePanelService, and the window server lists it twice with the same bounds: as the service's off-screen layer-0 window, which holds the Accessibility tree, and as the app's on-screen window that shows it (panel_hosts). A window of the app with those bounds is a file dialog whatever its identifier, and the service's window reports the app window's id, so captures, list_windows and window_id all name the window on screen. The app lists the dialog among its windows while the dialog's link to it holds; when it stops listing a dialog that is still on screen, the dialog's window is taken from the service's AXWindows by window id, so list_windows, window=<title> and the states keep reaching it. Its Accessibility elements are reachable and AXPress and AXValue work, SkyLight-routed clicks land, but key events posted to the app pid, to the service pid, or after a focus-without-raise never arrive, and setting the search field's AXValue does not run the search. While a file dialog is the focused window or in it, stands as a window of its own, or is on screen without Accessibility reaching it, press_key, type_text and paste fail with that explanation and point at open_file instead of sending keys to the window under it; a sheet on another document window leaves keys to the focused one alone. A state of the dialog says it is the file dialog, points at open_file and always adds the OCR text, since the file browser exposes no rows; a state of another window of the app says a file dialog is open. open_file replaces the dialog: a modal dialog holds back the request to open a document, so it first presses an open dialog's Cancel button (AXCancelButton, else a button titled Cancel) and waits for the dialog to close, and fails with what to click when Accessibility does not reach it; then NSWorkspace openURLs:withApplicationAtURL:configuration: with activates off opens the document in the app (launching it in the background if needed), the front app and its windows stay where they are, and the call waits for a window whose AXDocument is the file's URL or whose title names it. It refuses relative or missing paths, bundles and launchers (.app, .command, shell scripts, .workflow, installer packages, location files), files with an execute bit, and terminal apps, which run what they open. A minimized window or a hidden app still gives its tree; the screenshot is unavailable until show_window un-minimizes it (AXMinimized) or unhides the app, neither of which activates it. A window on another Space is read and operated where it is. AXWindows lists only the current Space's windows, so the app's off-screen ordinary windows (layer 0, at least 80×60) that no listed window covers are looked up by SkyLight's remote token: _AXUIElementCreateWithRemoteToken with the pid, 0, 'coco' and an element id, trying ids from zero (at most 4000) until each window id has its element, as AltTab does; the result is kept per app until that set of ids changes. ScreenCaptureKit captures such a window and SkyLight's pointer route reaches it, but the window server drops key events for it whatever their route, so type_text and paste there set the focused element's AXSelectedText instead, press_key reports that keys do not arrive, and show_window says the window stays on its Space. Moving it is not possible: since macOS 14.5 only the Dock may (SLSMoveWindowsToManagedSpace from any other process needs System Integrity Protection relaxed, as yabai documents).

paste writes the pasteboard (plain text, or HTML rendered from Markdown with markdown-it), sends Cmd+V to the target process, and restores the previous pasteboard contents when the change count shows nothing else wrote in between.

Agent cursor#

Before a step that names an element or a point, MacOSComputer.act computes where it lands (the element's centre from its AXFrame, or the screenshot point mapped through the window origin and scale) and calls native_window_resources.note_action_point, then waits CUE_LEAD_SECONDS (0.15 s) so the cursor arrives first. The point is in screen points with the origin at the top left of the main display, the space Electron also uses. The worker broadcasts it as a native.action frame (session_id, x, y, sequence, at, action, app_name); an agentic subprocess forwards it through the same parent channel as operated windows (kind: "action"). In the web client native-window-store.ts hands the frame to openprogramDesktop.nativeActionCue unless Show actions is off. The desktop main process (main/worker/agent-control.js) draws it in the conversation's control overlay (below): the cursor glides there and stays. Every App window receives the frame, so the main process drops a step it has already shown by session_id, sequence and at. Linux and Windows do not emit points yet.

Window marker#

The operated window carries the macOS screen-sharing marker, as under Codex Computer Use: a badge over its title-bar buttons and the sharing icon in the menu bar. macOS draws both for any window under a live ScreenCaptureKit stream, so window_watch.WindowWatch keeps one open on it. MacOSComputer touches it with the window id, the app's name and the frame of its title-bar buttons (read through Accessibility) on every act and every get_state that names a window. The stream filters to that one window (SCContentFilter.initWithDesktopIndependentWindow), asks for a 2x2 frame at one frame a second and attaches no output, so nothing is recorded. Each session watches at most one window: moving to another window releases the previous one, and a window with no step for IDLE_SECONDS (20 s) is released by a sweep timer. Opening the stream runs on its own thread and its failures are ignored, so a step never waits on or fails because of the marker.

While a window is watched, a second thread reports its place every REPORT_SECONDS (0.12 s) when it changed: window_place reads the on-screen window list for the window's frame and the windows in front of it within GLOW_MARGIN (32 pt), leaving out windows titled OVERLAY_TITLE (the App's own overlays). The report goes through native_window_resources.note_control with the owner captured when the window was touched, as a native.control frame (session_id, window_id, active, app_name, frame, buttons, occluders); a subprocess forwards it as kind: "control". Releasing the window sends active: false. The web client passes reports to openprogramDesktop.nativeControl while Show actions is on, and always passes the end. The desktop main process keeps one transparent, click-through, unfocusable window per conversation over the window plus the margin and loads /menu-overlay/agent-control, which draws the inward glow, the accent lights over the buttons and the cursor. The overlay sits in the window stack directly above the controlled window (moveAbove with its window id, ordered again on every report because the stack changes whenever the user switches apps), so a window in front of the controlled one also covers the overlay; only when the window server refuses that order does it float above all windows. Either way covering windows are cut out of the glow and lights, and the cursor hides while its point lies under one. Its look is set in Agent control overlay.

Refusals#

The tool never types into AXSecureTextField elements or while macOS reports secure keyboard input (kCGSSessionSecureInputPID), and refuses to target Keychain Access, Passwords, loginwindow, SecurityAgent and OpenProgram's own bundles. The refusal is returned as an error that asks the user to do the step; no approval scope overrides it.

Linux backend#

LinuxComputer keeps MacComputer's interface, so the tool, the server's app lookup and the tests drive either one through backends.backend(). It runs only in an X11 session: under Wayland no client may read or inject input into another client's surface, and support() says so.

  • Apps. The AT-SPI desktop's children that own a window, plus top-level X windows whose owner does not join the accessibility bus. The owner of an X window is _NET_WM_PID, or, for Xt and Motif clients that do not set it, the newest process whose name or executable starts with the window's WM_CLASS instance. Each app is matched to a desktop entry (Gio.AppInfo) by executable or name, which gives the identifier (org.gnome.Calculator), the display name, and the icon looked up in the hicolor theme. Launching goes through Gio.DesktopAppInfo.launch and then waits for the app's AT-SPI root as well as its window, since GTK 4 apps join the bus after mapping.
  • Windows. Top-level windows are the window manager's _NET_CLIENT_LIST (read with xprop), which leaves out GTK's 10×10 client-leader window; without a client list, windows of 10×10 or smaller are skipped. An AT-SPI frame is paired with its X window by title, then by size. A window missing from the mapped set is reported minimized; show_window maps it again.
  • Tree. Roles are AT-SPI role names (push button, text, check box), so role_name prints them unchanged. A subtree is read when it is on screen: GTK 3 sets SHOWING, GTK 4 sets only VISIBLE below the window, so the walk uses SHOWING when any child of the window reports it and VISIBLE otherwise. A closed combo box's popup items and GTK's tooltip window are skipped. Unnamed fillers, panels and sections are transparent. GTK 4 names unnamed widgets after their class (AdwGizmo, GtkMenuButton); such names are not labels. Flags come from states (CHECKED or PRESSED, FOCUSED, SELECTED, EXPANDED, not ENABLED/SENSITIVE), settable from EditableText or Value, secure from the password text role. Secondary actions are the element's Action names other than click, press, activate, toggle, jump, open and select.
  • Element actions in the background. Click runs the element's primary action; set_value uses EditableText.set_text_contents, Value.set_current_value, or a toggle when a check box's state differs; type_text and paste insert at the caret with EditableText.insert_text (UTF-8 byte length), replacing a selection, and a newline runs the field's activate action; press_key Return on a field runs activate; select_text uses Text.set_selection or the caret offset (character offsets, not UTF-16); copy reads Text.get_selection without the clipboard; scroll by element moves the nearest scroll bar's Value by 90 % of its length per page; menu walks the window's menu bar and runs a leaf's action. Component.grab_focus makes GTK present the window, so wherever focus has to move, the window that was active gets focus back.
  • Foreground input. GTK and Qt read pointer and key input through XInput 2 and ignore core events sent with XSendEvent to an unfocused window, so a coordinate click, drag, wheel scroll, hover or key press is sent with XTEST: the window is activated (windowactivate, or windowraise plus windowfocus without an EWMH window manager), the input runs, and the pointer position and previously active window are restored; hover leaves the pointer. Chords use xdotool names with cmd, command, meta and win mapped to super and option to alt.
  • Screenshots. Gdk.pixbuf_get_from_window on a foreign X window gives the window's own pixels at X resolution, scaled down to 1568 pixels on the long side; a minimized window has none. Screenshot pixels map to screen pixels through the window's position from xdotool getwindowgeometry, the same frame AT-SPI's screen extents use.
  • OCR. For sparse windows (and the find fallback), tesseract --psm 11 tsv with the installed languages among eng, chi_sim, chi_tra, jpn and kor; words with confidence 30 or more are joined into lines. Contour shapes are not detected on Linux.
  • Audio. pactl list sink-inputs gives each stream's application.process.id; the recorder polls until a stream of the app or any of its descendants (re-read on every poll, since players are usually children started after recording began) appears, then reads it with parec --monitor-stream=<index> as 16 kHz mono float. Reading continues until parec ends, so the samples it buffers when stop terminates it are kept. Under PipeWire the first samples arrive within a quarter second; under PulseAudio's timer-based scheduling the first monitor of a stream starts 1.2–1.7 s late and that span is lost.
  • Refusals. password text elements, Seahorse, KWallet, gcr prompts, polkit agents and OpenProgram's own entries.

The app guide is rewritten for Linux when it is sent (_LINUX_GUIDE in targets/app.py): which steps bring the window forward, plain-text paste, clipboard-free copy, the Linux refusals. The computer_use_apps WebSocket reply carries the worker's platform, and on Linux the approval card states for coordinate input, drags, hovers and keys other than Return (needsForeground) that the window comes forward for that step.

Windows backend#

WindowsComputer keeps the same interface. The process is per-monitor DPI aware (SetProcessDpiAwarenessContext(-4)), so window frames, UI Automation bounds and input coordinates are all physical screen pixels. Every Win32 function it calls has its restype and argtypes declared, so 64-bit handles pass intact.

  • Apps. Top-level windows from EnumWindows that are visible, unowned, not tool windows and titled, grouped by process. A UWP frame (ApplicationFrameWindow) belongs to ApplicationFrameHost; its app is the process of the Windows.UI.Core.CoreWindow child. Installed apps are Get-StartApps (desktop and Store alike, cached for a minute); a running process is matched to a row by executable or name, and launching runs explorer.exe shell:AppsFolder\<AppID>, then waits for a new window.
  • Tree. IUIAutomation.ElementFromHandle on the window, walked with the control view walker; offscreen subtrees are skipped except menu items. Roles are the control types (50000–50040) named in lower case (button, edit, check box). Unnamed panes, groups and custom elements without actions are transparent. Flags come from patterns: Toggle state, IsEnabled, HasKeyboardFocus, SelectionItem, ExpandCollapse, a writable Value or a RangeValue (settable), IsPassword (secure). Values come from RangeValue, Value or the Text pattern's document range. Secondary actions are expand, collapse, scroll, select and a LegacyIAccessible default action other than press, click, invoke or jump.
  • Element actions in the background. Click tries Invoke, Toggle, SelectionItem.Select, ExpandCollapse and LegacyIAccessible.DoDefaultAction in order, and focuses an edit; set_value uses Toggle, RangeValue, Value or LegacyIAccessible; select_text moves a Text pattern range by characters; copy reads the Text pattern's selection; scroll by element uses the nearest Scroll pattern by large increments; menu walks a classic HMENU (GetMenu) and posts WM_COMMAND, or expands and invokes UI Automation menu items.
  • Posted input. A coordinate click, drag, wheel scroll or hover is posted as mouse messages to the deepest child window under the point (ChildWindowFromPointEx), in its client coordinates; a second click is WM_LBUTTONDBLCLK. A key without named modifiers is WM_KEYDOWN/WM_CHAR/WM_KEYUP to the thread's focus window (GetGUIThreadInfo); type_text posts WM_CHAR per UTF-16 unit. paste into a classic edit or rich edit control sends WM_PASTE.
  • Foreground input. Shortcuts read GetKeyState, which posted messages do not change, and UWP frames, Chromium and Firefox windows read raw input, so a chord with modifiers, pointer input to those window classes, and paste elsewhere use SendInput inside _Forward: it joins the foreground thread's input queue (AttachThreadInput) to call SetForegroundWindow, runs the input, and then puts the cursor and the previous foreground window back.
  • Screenshots. PrintWindow with PW_RENDERFULLCONTENT renders the window, covered or not, into a bitmap the size of its outer rectangle; the visible frame (DWMWA_EXTENDED_FRAME_BOUNDS) is cut from it and scaled with StretchBlt in HALFTONE mode to 1568 pixels on the long side, then GetDIBits gives BGRA rows that png_encode writes as a PNG without any imaging library. A minimized window has no screenshot; show_window restores it with SW_SHOWNOACTIVATE.
  • Clipboard. paste writes CF_UNICODETEXT and, for md and html, the "HTML Format" payload whose header offsets cf_html computes, and restores the previous text afterwards.
  • OCR. As on Linux, tesseract when it is installed.
  • Audio. ActivateAudioInterfaceAsync on VAD\Process_Loopback with PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE gives an IAudioClient initialised in shared loopback mode with auto-conversion to 16 kHz mono float, read event-driven through IAudioCaptureClient on its own COM thread. The completion handler is an agile COM object, since Windows calls it on a worker thread.
  • Refusals. IsPassword elements, CredentialUIBroker, consent (UAC), LogonUI, LockApp, credwiz, keymgr and OpenProgram.

The app guide is rewritten for Windows when it is sent (_WINDOWS_GUIDE), and the approval card says for a chord or a paste (needsForeground(args, "windows")) that the window comes forward for that step.

Permissions and approval#

For the app target requires_approval returns computer_use_agent controls <app> for every action except list_apps; a call with code, fresh or no action is approved as reading the app. parse_command reduces a call to its normalised application name (vm:<url> for the VM, none for the browser), so exact_rule_for_call yields computer_use_agent(calculator) and rule_to_string prints it in the usual form. The approval card offers four scopes:

Scope Where the rule is saved
once nowhere
session the session's run config (permission_rules.allow) — computer_use_agent is in _SESSION_SCOPE_TOOLS
always the project rules
deny nowhere

The grant applies in the modes that ask; Bypass allows the call and Auto sends it to the Auto review. permission_decision treats computer_use_agent as an app-grant tool (_APP_GRANT_TOOLS) for its app and vm targets: when the tool's own requires_approval says the call needs approval and no allow rule names the application, the decision is ask with reason APP_GRANT_REQUIRED, Unattended sources therefore deny such calls outside Bypass, and in the modes that ask an allow rule is the only way past the card. _RULE_ALIASES lets rules saved under the former name, computer_use(<app>), match computer_use_agent calls. The browser target has no rule of its own: a conversation with Pages gets the turn's SURFACE_GRANT, and an upload or a website outside this computer asks.

Permission rules are reloaded at every operation boundary (global < project < session), so a session rule applies to the next call without a restart.

System access: computer_use:desktop is an operation of both the accessibility and screen_recording capabilities; _desktop_operation maps every app call except list_apps and guide to it, so a missing grant pauses the call with the standard "waiting" state and the Settings → System access flow; the waiting request names computer_use_agent.

The tool is in DEFAULT_TOOLS and resident: its schema is short, so it is not deferred. When the conversation has Pages, _configure_web_use_tools keeps it and removes the independent browser tools; web_use is never offered to the model. unsafe_in hides it in WeChat and Telegram channels and in plan mode.

Web presentation#

lib/computer-use/computer-use-agent-call.ts reads a computer_use_agent call: appStepArgs turns an app call into the flat step the app renderers take ({action, app, …}; several steps become a batch, code and a fresh agent their own actions), and screenStep names a browser or VM call's action. tool-presentation.ts renders app steps with the monitor icon and the system tone; the title names the action and the application ("Clicked in Calculator"), the target line shows the key, text, action name or window where one is given. Browser calls show the action and URL, VM calls the action, code "Ran code in ", and a fresh agent "Handed to a new agent" with the brief's first line. The approval card's scope buttons add "Allow this conversation" between once and always, and the answer summary prints "Allowed for this conversation". Calls recorded by the earlier tools (computer_use, web_use, gui_agent) keep their renderers so old conversations read as before.

The browser application#

The built-in browser is the application openprogram/programs/applications/browser/. The core never imports it: the application's setup(ctx) provides the service browser, and every core caller looks it up with _services.lookup("browser") and runs without a browser when the lookup returns None.

Module Role
__init__.py PROGRAM_ENTRIES (web_use) and setup: provides the service, registers the L2 context section pages and the resource provider web
service.py BrowserService, the service's members (below)
pages.py BrowserResourceStore, per-conversation Page projections, resource control, human page input
tabs.py the tab driver for computer_use_agent(target="browser")
extension.py the user's Chrome or Edge through the Playwright Extension
chrome.py, privacy.py the Chrome bootstrap that openprogram browser maintains; privacy filters
contract.py the web_use argument contract shared by agents and MCP
prompt.py the open-Pages section of a call's prompt
resource.py the web resource provider and the Page lease for native presentation commands
web_use/ the web_use runtime: page controller, sessions and their registry, recovery, the exact-page MCP backend
engine/ the Playwright verb engine the page controller drives

The service's members are pages, tabs, chrome, extension, user_browser_names, web_use_parameters(), normalize_web_use_arguments(), execute_direct_web_use(arguments, owner_id=), registry(), release_owner(), park_owner(), authorized_page() and browser_of().

Without the service the core degrades in one way per caller: a call's prompt has no Pages section and the turn gets no web_use; web_use arguments fail validation with "the browser is not enabled"; MCP lists no web_use and refuses its calls; /api/web-use answers browser_not_enabled; the process routes list no browser resources and answer 404 for browser control; the web tab bridge skips Page bookkeeping; computer_use_agent(target="browser") reports the target unavailable and the desktop targets keep working; the permission policy sees no user browser.

The web_use runtime#

web_use is the application's one registered entry. Programs and MCP clients call it directly; the browser target of computer_use_agent uses the tab driver below instead. Its runtime lives in web_use/ of the application, and the argument contract in contract.py is shared by agents and MCP.

Names. Agents and MCP see one tool, web_use; sessions are web_session_id in a WebUseSessionRegistry. The HTTP route is /api/web-use; /api/computer-use remains a hidden compatibility route on the same handler, kept out of OpenAPI, and old computer_session_id request fields are converted to the new ones.

Five commands.

Command What it does
list_pages lists the Pages of every registered OpenProgram window, each with a single-use page_context_token
observe selects a Page by token, opens a WebSession and returns bounded text, ARIA and element refs
act performs one navigate, click, type, press, scroll, hover or select against the latest expected_frame_id
verify runs a Runtime assertion (text_contains, title_contains, url_contains and so on); a model's claim of completion never makes a result succeeded
close closes the observed Page's tab and releases its WebSession, Page binding and held capabilities
web_use({ command: "list_pages" })
web_use({ command: "observe", page_context_token: "pct_…", arguments: { detail: "interactive" } })
web_use({ command: "act", web_session_id: "cs_…", arguments: { action: "click", expected_frame_id: "f_…", ref: "e1" } })
web_use({ command: "verify", web_session_id: "cs_…", arguments: { action: "verify", expected_frame_id: "f_…", assertion: "text_contains", value: "Saved" } })
web_use({ command: "close", web_session_id: "cs_…" })

Page inventory. list_pages returns windows[], one snapshot per registered window with its window_id, inventory_revision, active_tab_entry_id, focused_page and tab_entries, and a flattened pages list. A tab entry is one item in the top tab bar: single holds one Page, split two ordered panes (tab:<tab_id> and group:<CenterTabGroup.id>). Each Page carries its window_id, tab_id, tab_entry_id, placement, title, origin, visible, focused, an optional opener_tab_id and its token. Pane order follows the rendered order; left and right derive from actual geometry and are never Page identity. Grouping, reordering or resizing increments inventory_revision without changing a Page. A popup becomes its own tab and Page with opener_tab_id; it never rebinds the opener's session, and the caller picks it up on the next list_pages. When no Desktop window is registered, as after a worker or App restart, discovery waits up to DESKTOP_RECONNECT_WAIT_SECONDS (20 s) and then returns desktop_unavailable with retry_command=list_pages.

Binding and revisions. A token binds the caller's owner, the window, the exact CDP target and the server's page and access revisions; missing, replayed, cross-owner or replaced-target tokens are refused before any backend call. Each visible web tab also carries a renderer-owned geometry_revision that changes with its in-App bounds or visibility (not with focus, and not with macOS stacking, occlusion or minimization). The WebSession stores all three revisions, the worker compares page and access, and the exact renderer compares geometry before native activation; any mismatch returns page_context_stale without touching the backend. Writes to one Page hold an exclusive lease; different Pages run concurrently. Background Pages resolve their target directly, with no tab switch, window focus, bringToFront() or OS input. The inventory, WebSession registry and renderer WebSocket registry run in the worker's main process; a workflow subprocess reaches the same WebSession through the parent bridge, which forwards revisions unchanged.

Owners and cleanup. A chat turn's owner is its session and assistant turn. Normal return, retry, cancellation and exceptions release every WebSession, Page lease and binding the owner holds, whether or not the model called close. A disconnected owner WebSocket revokes all its bindings at once. Explicit close checks the owner and live binding, asks the App to close the tab and then releases the session; a foreign owner closes nothing.

Tokens and recovery. act or verify with a token and no live web_session_id reuses the WebSession that token opened; with an unused token the runtime first observes read-only, and navigate, screenshot, wait and ref-free verify continue on that frame, while ref or coordinate actions return the observation with action_performed=false. The registry remembers, bounded and in memory, the tabs of WebSessions that ended and of tokens that expired. A later page_context_not_found, page_context_consumed or web_session_not_found from the same owner scope carries recovery_tab_id, recovery_window_id and recovery_command=observe; the agent tool then observes that tab with a fresh token and returns recovered_page=true, previous_action_replayed=false. Nothing is replayed. Every recoverable failure names exactly one next call.

Backends. observe picks one of three backends and freezes it for the WebSession: playwright_mcp (default, pinned @playwright/mcp), chrome_devtools_mcp (pinned chrome-devtools-mcp) or open_claude_chrome (OpenProgram's own Electron/CDP controller that follows the Open Claude in Chrome tool contract, with no extension or native host). A later request for another backend returns backend_mismatch; failures, timeouts and stale frames never fall back to another backend, so one action writes at most once. Each official-MCP WebSession owns a private MCP process started with npx --prefer-offline, bound to the exact target with tab selection disabled. If that process stops, the next observe restarts it read-only; act never writes through a restarted process and returns computer_use_backend_unavailable with observe_required=true. All three run one parity matrix through WebUseSessionRegistry.execute(): the same results, frame invalidation, reason codes and verification semantics.

Frames and failures. Every successful write invalidates the frame and sets observe_required=true. Missing arguments return invalid_arguments with the complete missing_arguments list and leave a valid observation in place; an old frame returns stale_observation; an unknown action returns unsupported_action; a backend error returns backend_action_failed. Reason codes (invalid_capability, ambiguous_page, page_context_stale, page_access_disabled, target_lost, needs_user, timeout, cancelled, verification_failed among them) come from the runtime; backend text never overrides them.

Cost. Observation is DOM and ARIA first: URL, title, text, snapshot and refs, without screenshots. screenshot returns one current-viewport PNG, taken from the exact Page's CDP session, for canvas pages or failed semantic location. The image goes only to the next provider request; tool text, the DAG and final results keep only frame_id, the viewport and image_attached, and normal return, provider errors, timeouts and cancellation all clear the image and revoke coordinate permission. No OCR, object detection, crop and zoom, visual memory or workflow replay runs. Login, CAPTCHA, sensitive submissions and irreversible actions return needs_user, and page content is untrusted data that cannot change targets or permissions.

Agent cursor. During an agent click, all three backends arm a capture listener in the exact Page only; a real pointer event shows a text-free, aria-hidden, pointer-events: none cursor and ripple at the viewport point, which then fades. User clicks, other Pages and the OS mouse are untouched, and a feedback failure never changes the click's result.

Browser target: the tab driver#

tabs.py of the browser application drives the App's built-in tabs for the browser target. It replaces, for that target, the web_use runtime's page context tokens, sessions, frames, leases and revision checks: none of them reach the model any more. The web_use runtime stays for the programs and MCP clients that still call it directly; it is the only page entry the application registers.

Tabs. A tab is (window_id, tab_id) as the renderer lists it (surface_context.capture_pages_for_listing, which now also carries each tab's CDP target_id). The driver names it t plus the first four base32 letters of the SHA-1 of window_id and tab_id (one letter more on a collision), so every process, the worker and a forked workflow subprocess alike, gives the same tab the same id in every turn, conversation and sub-agent, and listing again in any order does not rename. A tab that moved to another window keeps its id: a listed tab whose target_id belongs to a known tab takes that tab over. A tab whose target_id changed (the App restarted) forgets its refs, document baselines and screenshots. Each listing binds every page again, so the binding a tab had is released when a new one replaces it. An action without tab goes to the tab its agent used last (see below), else to the focused tab, else to the one visible tab. open asks the renderer for a background tab (surface_context.open_page) and makes it the current one; close and screenshot go through the tab's binding (request_close_tab, request_bound_screenshot), and a binding that expired is renewed by listing again.

Connection. Playwright's sync API belongs to one thread, so one owner thread holds connect_over_cdp to the App (desktop_app_ws_url) and runs every page call; callers wait on it with their context variables. A page is found by matching Target.getTargetInfo against the tab's target_id. A connection does not always learn of tabs the App opens after it, so a target that is not found makes the driver connect again and look once more; a dropped connection reconnects on the next call. When a page listener is first attached, the driver records its console, page errors and requests (the last 300 of each), and counts its document navigations and in-flight fetch/XHR requests.

Several agents. Agents of one conversation share its session id, and sub-agents run in parallel, so no tab is leased to one agent for longer than one action or one list of actions: leases held across turns and their hand-over were what made the earlier runtime refuse its own conversation's turns. Each Tab has a lock. An action takes it before its first page call and keeps it through settling and its result, so another agent's input never lands between an action and the state it reports. A list of actions (targets/browser.run with several steps, inside tabs.batch()) keeps every lock it took until the list ends, so a form filled in one list is not interleaved. Waiting for a lock polls with cancellation checks, and after TAB_WAIT_SECONDS (60 s) the action fails with tab_busy; two lists that each wait for the other's tab end that way rather than deadlock. The agent a call acts for is <session>/<execution> (the _VIEWER context variable, carried into the owner thread). Each agent remembers its own current tab; an agent without one starts from the tab its conversation used last. Snapshots are tracked per agent (track = openprogram:<agent>, and each agent's document baseline), so an agent's changes cover everything since it last looked, including what other agents did. Each tab records its write actions with the agent that made them; a result adds a note when another agent acted on the tab since this agent's last result. Only single page calls run on the owner thread. wait and the settling after an action poll on the caller's thread with short owner calls (_shows, _activity, _loaded), so one agent's wait never holds up another agent's click on another tab.

Writes are sent once. Only tab_gone, which _page raises before any input, makes the driver list the tabs again and try the action once more. An action that failed after Playwright may have sent its input counts as a write, and its error says it may have taken effect when it timed out, with the page as it is; a tab that closed during the action is tab_closed. Once an action succeeded, any failure while settling or reading the page comes back as a successful result with a note, never as an error the caller would retry. close treats the renderer's response timeout as close_uncertain, an expired binding as tab_gone (nothing was sent), and other failures as close_failed. evaluate counts as a write.

Tabs an action opens. The connection discovers targets (Target.setDiscoverTargets) and counts each new page target. When the count changed during an action, the driver lists the tabs and the result's opened names the new ones whose opener_tab_id is the acted-on tab or unknown. The agent's current tab stays where it was.

Coordinates. A screenshot records, per agent, the CSS-to-image ratio, the tab's last write, its URL and its scroll offset. A click by x,y needs that agent's screenshot, and fails with screenshot_stale when any agent wrote to the tab, it navigated or it scrolled since.

Preview follow. The conversation's page preview follows the operations the driver reports, as it does for the web_use runtime: the frontend follows only a live browser.resource event whose last_operation is new and dispatched or acknowledged, never a snapshot. Every write action (anything but read, console, network, wait, screenshot, tabs and close) first retains the tab's resource row for the current execution (publish_bound_page), then reports dispatched with follow (pages.report_browser_operation), which stores the operation and sets the resource's control state to active before the input is sent. After the action it reports acknowledged, or failed with the reason code, under the same operation id; a failed one does not set the control state again. The page shows as operated only while the execution that set it is running: the resource projection turns active back into idle once that execution has ended, as for web_use. A click's outcome carries the marker point: before clicking, _aim scrolls the element into view and records the centre of its bounding box, or the CSS point a click by x,y maps to, with the viewport's innerWidth and innerHeight; sanitize_operation turns that into point with the viewport size as width and height, which browser-marker-geometry maps onto the screenshot. Reads (read, console, network, wait, screenshot) report an acknowledged operation without follow, so the preview follows them and the page is not marked operated, and open reports the same for the new tab after publishing it, which makes it the tab followed. Each step of a list reports on its own. Reporting never fails an action: errors are logged at debug level.

Workflow subprocesses. A forked child drops the parent's driver (os.register_at_fork), because its owner thread and connection belong to the parent, and starts its own. Its renderer calls (capture_pages_for_listing, open_page, request_close_tab, request_bound_screenshot) go through the parent's webtab bridge, which confines them to the originating window, authorizes only bindings it handed out and closes the tabs the child opened when the child ends. The child holds no App connection, so its operation reports go to the worker as well: pages.forward_operations_to_parent hands each one to the child's event queue as __op_browser_operation__, and the worker reports it with report_forwarded_operation, which keeps only the operation's own fields and falls back to the execution it launched. Tab locks and the write log are per process, so a child and the worker on one tab interleave at single page calls.

Access and cancellation. When the turn's surface has agent access disabled (preview_status disabled), every built-in browser action is refused with page_access_disabled. Waiting for a tab, wait, settling and each step of a list check for cancellation.

Reading. read returns Playwright's AI snapshot: the ARIA tree with a ref on each element (e12, and f1e3 inside a frame). Playwright 1.60 serves it as Frame.ariaSnapshot(mode="ai", track), one text that is incremental when the same document has an earlier tracked snapshot; earlier versions serve Page.snapshotForAI, which returns both. The driver asks for the whole page when the document is new or a full read is wanted, and otherwise only for the changes. Changed lines are marked <changed> and [unchanged] stands for what was already sent. Password field values are redacted from both, and a page is cut at 40,000 characters.

Refs. Playwright resolves aria-ref=<ref> against its latest snapshot, and a ref stays the same for as long as its element exists, so refs survive any number of actions and turns. The driver keeps each ref's role and name from the snapshots. A ref that no longer resolves is looked up by that role and name, and so is an element named directly by role and name. If that still finds nothing, the action fails with element_gone and the page as it is now.

Acting. Clicks, hovers, key presses, typing (fill, or insert_text with clear: false), select_option, drag_to and set_input_files are Playwright's own trusted input. They reach a tab in the background without bringing it forward, which was checked against the App: a background tab receives isTrusted pointer, mouse and click events. Scrolling uses page_actions.SCROLL_SCRIPT, and paste uses PASTE_SCRIPT. A click by x,y maps screenshot pixels to CSS pixels with the ratio measured when the screenshot was taken, if the screenshot is still current (above). Before a click, the App's agent cursor is armed and placed on the element. After an action, the driver waits up to 0.3 s for a navigation to start. If one starts, it waits for domcontentloaded. It then waits up to 2 s for fetch/XHR requests the action started. Then it returns the changes, or the whole page when the URL changed or the document is new. An exception becomes a result with a reason code and the page as it is now, never a raw error.

Approval. The App's tabs need none. open and navigate to an origin outside this computer ask; upload asks; evaluate asks outside bypass, as below. Code mode's validator refuses all three, and its read-only actions (tabs, read, wait, screenshot, console, network) are marked as reads for the broker.

File upload. Pages rarely show their file input: it sits behind a styled upload button with display: none, so the outline has only the button, or the page creates the input when the button is clicked. The driver therefore uploads the way a person does, as Playwright MCP does. Every write action runs with a filechooser listener on the page (_catch_choosers), which makes Chromium hand the chooser to the driver instead of opening a window on the user's screen; the listener is removed when the action has settled, so the user's own clicks open the usual window. A chooser caught that way is kept on the tab and the result carries file_chooser (multiple, accept); upload(path) without a ref answers it, a bad path leaves it waiting, and any other write drops it. upload(ref, path) is the one-step form. When the ref is a file input or the label of one (FILE_INPUT_SCRIPT), the file is set on that input, hidden or not; anything else is clicked inside expect_file_chooser and the chooser it opens takes the file, or the call fails with no_file_chooser after 3 s. The driver never guesses an input near a button: a page with a second hidden input would take the file into the wrong field and report success, which is also why Codex exposes uploads only through the chooser. A button that opens nothing is still not a dead end. The page may be stuck in an earlier upload, or open its input only after an await that spent the click's user activation. So the error says what the button reads now and whether it is disabled (ELEMENT_STATE_SCRIPT), and lists the page's enabled file inputs, numbered (FILE_FIELDS_SCRIPT). Each entry gives its id or name, accept, multiple, the text it sits in, and whether it is the only one in the button's section. The model picks one with upload(field=n, path), which sets the file on that input (Tab.file_fields). Any other write action, or a new page, drops the list, and the user's own browser does not support field. The text result names a caught chooser and its accept, so the model knows to answer it. The user's Chrome or Edge follows the same model through Playwright MCP's own file-chooser state: upload with a ref clicks it first, without a ref it answers the chooser the last click opened. Drop zones with no file input at all are not covered.

evaluate reads everything a signed-in page holds, including what snapshots redact, so it is gated in the permission policy rather than by the tool. page_script marks a computer_use_agent browser call with an evaluate step (single or in actions), or a web_use evaluate. Such a call runs in bypass, as every other call does; in the other modes it asks (PAGE_SCRIPT_APPROVAL, before the Page grant is consulted) unless an allow rule covers it. The card offers "once" and "this conversation", which saves computer_use_agent(browser:evaluate) (or browser:chrome:evaluate) in the session layer; it never offers a project rule, since one approval should not reach every site in every later conversation, and it shows the arguments untruncated. Operating a browser (browser:chrome) does not allow its page scripts. Non-interactive sources are refused, in bypass too. The result passes the same password redaction. An expression is wrapped as () => (expr); scripts are capped at 8000 characters and results at 20,000.

Interface work#

The approval card for an app call (computer-use-approval.tsx) leads with the application itself: its icon, display name and bundle id come from the computer_use_apps WebSocket action, which calls the backend's app_info and launches nothing, and the web caches them per name (lib/net/computer-use-apps.ts). Below it a chip names the step in words (computer-use-step.ts), two lines state that the step runs in the background (or, for audio, that only that application's sound is recorded) and what the offered scopes cover, and the arguments without empty values fold into "Execution details".

The project's Settings tab has a Computer use section (computer-use-apps-section.tsx) between the default settings and the permission rules. It lists the project's computer_use_agent(<app>) allow rules (and those saved as computer_use(<app>)) as one row per application with icon, name, bundle id and a remove control, and the generic rule list leaves those rules out. Removing a row removes the rule. Conversation-scoped grants are session rules that end with their conversation, so the section does not list them.

The person points the agent at things the way Codex lets them:

  • In a web tab, Comment on the page… picks one element and adds its selector, text and the comment to the draft; Screenshot a region… lets them drag a rectangle and adds those pixels as a PNG attachment. Both run in one isolated world (PICKER_WORLD in apps/desktop/main/web-views/web-views.js) that shares a cancel hook, so one mode ends the other. The region overlay is removed and two frames are painted before capturePage takes the rectangle, scaled by the page zoom. The file reaches the target conversation through lib/chat/composer-inbox.ts: the composer showing that conversation takes it as a dropped file, and it waits there until one does.
  • The composer's @ menu lists open web tabs and running apps above the files (attach/mention/mention-sources.ts). A tab is written as a markdown link to its address and an app as its name and bundle id, so "this tab" resolves without the model guessing. Running apps come from computer_use_apps with running: true, which lists regular apps other than the refused ones and is kept for ten seconds in the page.

Implementation status#

  • Implemented: computer_use_agent with the app, browser and vm targets, per-target guides on first use, several steps, code mode, fresh agents with a carried session rule, the VM target over the OSWorld HTTP API (through the outbound policy as the computer_use.vm consumer), and the web presentation; unit tests cover routing, guides, the VM scripts and approval, and a sandbox test runs code against stub targets. Not yet run against a live VM server.
  • Implemented: the app target on macOS with the reading and acting actions above, stable indices, adaptive settling, background pointer and keyboard delivery, tree diffs, per-app approval with the four scopes, system-access gating, web presentation, unit tests for keys, tree numbering, menu matching, several steps, rules and approval wiring.
  • Implemented: the built-in browser as the application programs/applications/browser/, reached only through the browser service, with tests for each caller running without it.
  • Implemented: the web_use runtime with the cross-window Page inventory, page/access/geometry revisions, owner cleanup, recovery hints, the three backends and their parity matrix, the single-viewport screenshot release and the agent cursor.
  • Implemented: the tab driver for the browser target, with unit tests for choosing tabs, ref recovery, both snapshot protocols and the result shape, and two end-to-end runs in the App in which an agent filled and submitted a local two-page form (text, email, radio, checkboxes, select, textarea, a confirmation box) through background tabs.
  • Implemented: OCR text for windows with a sparse tree and as the find fallback, with a test that reads a rendered image through Vision.
  • Implemented: per-application audio recording with WAV output and Voice-engine transcription.
  • Implemented: visible tabs and changes_only observations on the native backends.
  • Implemented: recordings as audio input for Gemini and Chat Completions audio models; tool-result screenshots on Chat Completions vision models.
  • Implemented: element comments and region screenshots in web tabs; @-mentions of open tabs and running apps.
  • Implemented: outlines of icons and controls without text in sparse windows.
  • Implemented: the Linux (X11) backend described above. It is tested in a Debian 12 container with Xvfb, Openbox, the AT-SPI bus and PipeWire against a GTK 3 window (menu bar, entry, password entry, check box, slider, scrolled text view, drawing area, a button that plays sound from a child process), GNOME Calculator (GTK 4 with libadwaita, launched from its desktop entry, 7×6 computed by element clicks) and ImageMagick display (an Xt client without AT-SPI, read by OCR). Host-independent parts have unit tests.
  • Implemented: windows on other Spaces, tested from Desktop 1 against Calculator on Desktop 2 (listing, screenshot, element presses computing 7×6, a coordinate click, a scroll, the key errors), with the pointer and the front app unchanged and the window left on its Space; inserting text there is not covered, since Calculator has no text field.
  • Implemented, not yet run on Windows: the Windows backend described above. The macOS machines it is built on have no Windows guest, so only its host-independent parts (PNG encoding, key mapping, the HTML clipboard payload, Start menu parsing, the control-type table, refusals, the recording format) have unit tests.
  • Implemented on macOS: the control overlay over desktop apps (inward glow, accent title-bar lights and agent cursor), with unit tests for the window place and step point frames and for the desktop overlay window. Not implemented: the Linux and Windows backends report neither window places nor step points, so no overlay shows there.
  • Implemented: the user's Chrome and Edge in the browser target over Playwright MCP's extension mode, with unit tests for the action mapping, Edge, stopping on failure, the tab list and the approval subject. Not yet run against the Playwright Extension in a live browser.
  • Implemented on macOS: the window marker, with unit tests for watching, switching and idle release. Not implemented on Linux and Windows.
  • Implemented on macOS: following a newly opened window, file-dialog detection with the key refusal, and open_file, tested against Preview and PDF Expert (Open dialog followed, keys refused, Cancel pressed, a PDF opened and its text read) with the front app unchanged throughout, plus unit tests. Following a dialog that leaves the focus where it was, telling the out-of-process dialog by the window list, reading it from the service process, refusing keys while it is open beside the focused window, and cancelling it from open_file have unit tests; of these only the window-list detection has run live, against PDF Expert's Open dialog. Not implemented on Linux and Windows.
  • Not implemented: audio input over the Responses and Anthropic APIs, whose models take no audio; moving a window from another Space, and typing into it with keys (see above); naming an icon from its outline (the model reads the screenshot).
Last updated · 2026-11-06