MusePi

browser

Open, reuse, close, and script browser tabs against headless Chromium, CDP-attached apps, or cmux surfaces.

Source

Inputs

Shared fields

Field Type Required Description
action "open" \| "close" \| "run" Yes Dispatches to the open/close/run path.
name string No Tab id. Defaults to "main". Tabs live in a process-global map, so the same name is reused across later calls and in-process subagents until closed.
timeout number No Tool wall-clock timeout in seconds. Defaults to 30; clamped to the browser tool range before execution.

action: "open"

Field Type Required Description
url string No Navigate after the tab is ready. Existing reusable tabs also navigate when url is supplied.
viewport { width: number; height: number; scale?: number } No Requested viewport. For headless launch this becomes the initial viewport; for a page it is applied with page.setViewport(). scale maps to Puppeteer deviceScaleFactor.
wait_until "load" \| "domcontentloaded" \| "networkidle0" \| "networkidle2" No Navigation wait condition. Defaults to "load" where omitted, including open navigation and later tab.goto(...).
dialogs "accept" \| "dismiss" No Installs a page dialog handler that auto-accepts or auto-dismisses dialogs. Omitted means no handler.
app { path?: string; cdp_url?: string; args?: string[]; target?: string } No Selects browser kind. With no app, a configured browser.cdpUrl setting attaches to that endpoint; otherwise the cmux backend is used when a cmux socket is available (CMUX_SOCKET_PATH, gated by the browser.cmux setting / PI_BROWSER_CMUX override); otherwise the session browser.headless setting applies. app.path is resolved against the session cwd and used as the executable path for spawn/attach reuse. app.cdp_url connects to an existing CDP endpoint. args are appended only when spawning app.path. target is only used for attached/spawned-app page selection.

action: "close"

Field Type Required Description
all boolean No Close every known tab. Omitted closes only name.
kill boolean No When a tab release drops a spawned-app browser handle to refcount 0, also terminate its process tree. Has no effect on headless shutdown and only disconnects connected CDP browsers.

action: "run"

Field Type Required Description
code string Yes Async-function body executed by the shared JsRuntime (src/eval/js/shared/runtime.ts, the same engine as the eval JS tool). In scope: browser-specific page, browser, tab, assert(cond, msg?), and wait(ms), plus the runtime prelude helpers (display, print, read, write, append, tree, env, tool, completion, agent, parallel, pipeline, log, phase, budget, …) and ambient Bun globals (console, timers, URL, TextEncoder/TextDecoder, Buffer).

Outputs

The tool returns one result per call; no streaming partial output is emitted from the browser implementation itself.

Flow

  1. BrowserTool.execute() (packages/coding-agent/src/tools/browser.ts) abort-checks, clamps timeout via clampTimeout("browser", ...), defaults name to "main", and dispatches on action.
  2. open resolves browser kind with resolveBrowserKind():
    • app.cdp_url → { kind: "connected" } after trimming trailing slashes.
    • app.path → { kind: "spawned" } after resolving against session cwd.
    • otherwise, a non-empty browser.cdpUrl setting → { kind: "connected" } after trimming whitespace and trailing slashes.
    • otherwise, resolveCmuxKind() → { kind: "cmux", socketPath, password?, surface? } when CMUX_SOCKET_PATH is set and cmux is enabled (browser.cmux setting, overridable by PI_BROWSER_CMUX).
    • otherwise → { kind: "headless", headless: session.settings.get("browser.headless") }.
  3. open rejects reusing the same tab name across different browser kinds (sameBrowserKind()); callers must close first.
  4. open acquires a browser handle through acquireBrowser() (packages/coding-agent/src/tools/browser/registry.ts):
    • existing connected handle is reused by browser-kind key;
    • stale disconnected handles are disposed and recreated;
    • headless attaches to the project-shared broker-owned Chromium (ensureSharedBrowser()); in a CLI-host process a broker failure is a hard error, while non-CLI hosts (bun test, SDK embedding) launch a process-local Chromium via launchHeadlessBrowser();
    • connected waits for ${cdpUrl}/json/version, then puppeteer.connect();
    • spawned first tries findReusableCdp(), else kills same-path processes, allocates a free loopback port, spawns the executable with --remote-debugging-port=<port>, waits for CDP, then connects.
    • cmux connects a CmuxSocketClient to the cmux unix socket; existing cmux handles are reused unconditionally (no connection-liveness recheck).
  5. open acquires a tab through acquireTab() (packages/coding-agent/src/tools/browser/tab-supervisor.ts):
    • same-name + same-browser + alive tab is reused unless dialogs changed;
    • same-name but different browser handle, dead state, or changed dialog policy forces release and recreation;
    • reusing with a new url navigates by issuing await tab.goto(...) through the worker, defaulting to waitUntil: "load" when wait_until is omitted.
  6. New tabs build a WorkerInitPayload in buildInitPayload():
    • headless mode sends url, waitUntil, viewport, dialogs, and timeout; the worker defaults missing waitUntil to "load".
    • attach mode resolves a page with pickElectronTarget(), gets its target id, and sends targetId plus dialogs.
  7. acquireTab() spawns a dedicated Bun Worker from tab-worker-entry.ts; if that fails it falls back to inline execution in the main thread (spawnInlineWorker()), preserving behavior but losing protection against synchronous infinite loops.
  8. WorkerCore.#init() (packages/coding-agent/src/tools/browser/tab-worker.ts) connects back to the browser websocket endpoint. Headless mode opens a new page, applies stealth patches, applies viewport, installs dialog handling if requested, and optionally navigates. Attach mode resolves the requested target page and optionally installs dialog handling.
  9. On success the worker sends ready with { url, title, viewport, targetId }; the supervisor stores a TabSession, increments browser-handle refcount with holdBrowser(), and keeps the tab in a process-global Map<string, TabSession>.
  10. run requires non-empty code, looks up the tab with getTab(), then delegates to runInTab().
  11. runInTabWithSnapshot() rejects dead tabs and concurrent runs (Tab ... is busy), captures session cwd plus optional browser.screenshotDir, registers an abort hook, sends a run message to the worker, and races the result against timeoutMs + 750 ms. Timeouts force-kill the tab worker and, for headless tabs, close the orphaned page target.
  12. WorkerCore.#run() builds the tab API, lazily creates a shared JsRuntime via #ensureRuntime(), injects page/browser/tab/assert/wait with runtime.setRunScope(), and executes the user code through runtime.run(code, ...) raced against a cancel/timeout rejection. Cmux tabs take a parallel path through runCmuxCode(), which drives the same JsRuntime.
  13. The tab helper API implemented in #createTabApi() is:
    • tab.name: string
    • tab.page: Page
    • tab.signal?: AbortSignal
    • tab.url(): string
    • tab.title(): Promise<string>
    • tab.goto(url, { waitUntil? })
    • tab.observe({ includeAll?, viewportOnly? })
    • tab.ariaSnapshot(selector?, { depth?, boxes? })
    • tab.ref(id)
    • tab.screenshot({ selector?, fullPage?, silent? })
    • tab.extract(format = "markdown")
    • tab.click(selector)
    • tab.type(selector, text)
    • tab.fill(selector, value)
    • tab.press(key, { selector? })
    • tab.scroll(deltaX, deltaY)
    • tab.drag(from, to)
    • tab.waitFor(selector, { timeout? })
    • tab.evaluate(fn, ...args)
    • tab.scrollIntoView(selector)
    • tab.select(selector, ...values)
    • tab.uploadFile(selector, ...filePaths)
    • tab.waitForUrl(pattern, { timeout? })
    • tab.waitForResponse(pattern, { timeout? })
    • tab.waitForSelector(selector, { timeout?, visible?, hidden? })
    • tab.waitForNavigation({ waitUntil?, timeout? })
    • tab.id(n)
    • tab.ref(id)
  14. Selector handling in normalizeSelector() accepts plain CSS and Puppeteer query handlers, and rewrites legacy Playwright-style prefixes p-text/, p-xpath/, p-pierce/, p-aria/; other p-* prefixes throw a ToolError. Playwright-only engines/pseudos (:has-text(), :text(), :visible, :nth-match(), :near()/:above()/…) on a CSS selector throw a ToolError pointing at the text//aria/ equivalents instead of stalling the action timeout.
  15. tab.observe() clears the element cache, takes a Puppeteer accessibility snapshot, filters to interactive nodes unless includeAll, optionally filters to viewport-visible nodes, assigns numeric ids, caches ElementHandles, and returns URL/title/viewport/scroll metadata plus elements. 15a. tab.ariaSnapshot() resolves the optional selector (via normalizeSelector() → page.$, defaulting to the whole document) and runs the generated Playwright ARIA-snapshot bundle (src/tools/browser/aria/aria-snapshot.bundle.txt) via captureAriaSnapshot(). The bundle is wrapped in a new Function built worker-side (so page CSP never applies) and serialized to a CDP page.evaluate in the page’s main world, returning Playwright-format YAML. It always runs in ai mode: every node gets a [ref=eN] id, clickables get [cursor=pointer], and matched DOM nodes are tagged with an _ariaRef expando. Existing _ariaRef expandos are cleared before each snapshot so ids renumber deterministically from e1 (the fresh module’s counter resets each call); refs stay valid until the next snapshot. The cmux backend uses buildAriaSnapshotScript() over browser.eval instead (no ElementHandle; CSS selectors only for the root).
  16. tab.id(n) resolves the cached ElementHandle, verifies el.isConnected, and throws a stale-id error after cache invalidation if the DOM changed or the cache was cleared. 16a. tab.ref(id) resolves a [ref=eN] id from the latest ariaSnapshot() to a live ElementHandle via resolveAriaRefHandle() (page.evaluateHandle in the main world, walking the document + shadow roots for the matching _ariaRef), throwing if no element matches; it accepts a bare eN or a prefixed form. For inline selector use, parseAriaRefSelector() recognizes only the explicit aria-ref=eN / aria-ref/eN / ariaref/eN forms inside tab.click/type/fill/waitFor/scrollIntoView — a bare eN is intentionally rejected there so it does not collide with cmux’s native observe ids. The cmux backend resolves the same explicit forms through its aria-ref SelectorSpec kind in findElement.
  17. tab.goto() clears the cached element ids before navigating. Any new tab.observe() also clears and rebuilds the cache.
  18. tab.click() uses a custom retry loop for text/... selectors to find an actionable visible match; other selectors use page.locator(...).click(). Interactive actions (click/fill/type/press/scroll/drag/scrollIntoView/select/uploadFile) and the waitFor* helpers run under a per-op deadline (min(cellBudget − slack, ceiling)) threaded into both the puppeteer signal and .setTimeout(), so a stalled helper aborts the CDP action and rejects with a named tab.<op> timed out after <ms>ms that leaves cell budget — never the opaque whole-cell timeout. goto/evaluate stay uncapped.
  19. tab.screenshot() captures the page or selected element as PNG, resizes a model copy, saves under browser.screenshotDir or the OS temp directory, returns that path, records metadata, and optionally emits text plus image content.
  20. display() calls accumulate in an array. After code finishes, the worker posts { displays, returnValue, screenshots }; BrowserTool.#run() appends the return value as trailing text content when not undefined.
  21. close releases one tab or all tabs via releaseTab() / releaseAllTabs(). Each tab aborts pending runs, asks the worker to close, waits up to 750 ms for a closed ack, terminates the worker, decrements browser refcount, and disposes the browser handle when refcount reaches zero.

Modes / Variants

Side Effects

Limits & Caps

Errors

Notes