all repos — sz-chrome-remote-cdp @ 8fff91522918a8309dea2e9fa725e15f6ed1c3ea

pi-browser-harness, patched to drive a Chrome on another host over CDP, with castle_cdp_search web search

Initial release: pi-browser-harness

Browser control extension for pi β€” 20 browser_* tools for navigating,
clicking, typing, screenshotting, and extracting data from real Chrome
via CDP. Includes self-extending harness (register_tool/remove_tool),
guided /browser-setup, session persistence, and custom TUI renderers.
amankumarsingh77 amankumarsingh7702@gmal.com
Wed, 29 Apr 2026 00:40:36 +0530
commit

8fff91522918a8309dea2e9fa725e15f6ed1c3ea

A .gitignore

@@ -0,0 +1,6 @@

+node_modules/ +dist/ +*.tsbuildinfo +.DS_Store +*.log +package-lock.json
A .npmignore

@@ -0,0 +1,9 @@

+.gitignore +.npmignore +node_modules/ +dist/ +*.tsbuildinfo +.DS_Store +*.log +.git/ +.github/
A CHANGELOG.md

@@ -0,0 +1,19 @@

+# Changelog + +All notable changes to pi-browser-harness will be documented in this file. + +## [0.1.0] - Unreleased + +### Added +- Initial release of pi-browser-harness. +- 20 browser control tools (`browser_navigate`, `browser_screenshot`, `browser_click`, `browser_type`, `browser_press_key`, `browser_scroll`, `browser_execute_js`, `browser_http_get`, `browser_new_tab`, `browser_open_urls`, `browser_switch_tab`, `browser_list_tabs`, `browser_current_tab`, `browser_page_info`, `browser_go_back`, `browser_go_forward`, `browser_reload`, `browser_wait`, `browser_wait_for_load`, `browser_handle_dialog`). +- Self-extending harness: `list_dynamic_tools`, `register_tool`, `remove_tool` β€” the agent can write new browser tools at runtime. +- Guided setup command (`/browser-setup`) with Chrome detection, automatic browser-harness installation via `uv` or `git clone`. +- `/browser-status` and `/browser-reload-daemon` commands for daemon health monitoring. +- `--browser-namespace` and `--browser-debug-clicks` CLI flags. +- Session persistence for tab history and daemon namespace across reloads and branch navigation. +- System prompt injection with browser usage guidance and common workflow patterns. +- Custom TUI renderers for screenshots and tab listings. +- Dialog detection and handling for JS `alert`/`confirm`/`prompt`/`beforeunload`. +- Parallel URL opening via `browser_open_urls` with live progress streaming. +- Output truncation with temp-file fallback for large JS evaluation and HTTP responses.
A LICENSE

@@ -0,0 +1,21 @@

+MIT License + +Copyright (c) 2025 Pi Browser Harness Contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE.
A README.md

@@ -0,0 +1,513 @@

+# pi-browser-harness + +> **Full browser control for pi agents.** Navigate, click, type, screenshot, and extract data from a real Chrome browser β€” all through CDP, all from your coding agent. + +<p align="center"> + <img alt="Version" src="https://img.shields.io/npm/v/@pi-browser-harness?style=flat-square&color=2563eb"> + <img alt="License" src="https://img.shields.io/badge/license-MIT-blue?style=flat-square"> + <img alt="Pi Package" src="https://img.shields.io/badge/pi-package-2563eb?style=flat-square"> + <img alt="Node" src="https://img.shields.io/badge/node-%3E%3D18-339933?style=flat-square&logo=node.js&logoColor=white"> +</p> + +**pi-browser-harness** gives pi agents full control of a real Chrome browser. No headless mode, no emulation β€” your agent sees what you see, clicks where you'd click, and reads pages exactly as they render. Browser tools coexist seamlessly with all standard pi tools (`read`, `bash`, `edit`, `write`). + +--- + +## Why pi-browser-harness? + +| | | +|---|---| +| πŸ–±οΈ **Real browser, real pixels** | Connects to your running Chrome via CDP. Compositor-level clicks work through iframes, shadow DOM, and cross-origin content. | +| 🧩 **Seamless pi integration** | `browser_*` tools appear alongside `read`, `bash`, `edit` β€” no context switching, no separate daemon management. | +| πŸ”§ **Self-extending harness** | The agent writes its own tools mid-task. Need a paginated scraper? Bulk CSV exporter? Register it at runtime, callable next turn. | +| ⚑ **Fast data extraction** | `browser_http_get` bypasses the browser entirely for APIs β€” 10-50x faster than DOM scraping. `browser_execute_js` for rendered data. | +| πŸ›‘οΈ **Dialog-aware** | Auto-detects JS dialogs (alert, confirm, prompt, beforeunload) before they block your session. | +| πŸ“Έ **Visual feedback** | Screenshots before and after every action. The agent can see exactly what it's doing. | + +--- + +## Quick Start + +```bash +# 1. Install the package +pi install npm:pi-browser-harness + +# 2. Start Chrome with remote debugging +# macOS: +/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 & + +# Linux: +google-chrome --remote-debugging-port=9222 & + +# 3. Run the guided setup inside pi +/browser-setup +``` + +**That's it.** Your pi agent now has 20 browser control tools. Try: + +``` +> navigate to https://github.com and take a screenshot +``` + +--- + +## Features + +### 🧭 Navigation & Tabs + +| Tool | What it does | +|------|-------------| +| `browser_navigate` | Navigate to a URL (safe: creates new tab on first call) | +| `browser_new_tab` | Open a new tab, optionally navigate | +| `browser_open_urls` | Open multiple URLs in parallel tabs (great after web search) | +| `browser_go_back` / `browser_go_forward` | History navigation | +| `browser_reload` | Reload the current page | +| `browser_page_info` | Get URL, title, viewport, scroll position, or detect JS dialogs | +| `browser_list_tabs` / `browser_current_tab` | Tab discovery and identification | +| `browser_switch_tab` | Switch to a specific tab by targetId | + +### πŸ–±οΈ Interaction + +| Tool | What it does | +|------|-------------| +| `browser_click` | Click at viewport CSS-pixel coordinates (compositor-level) | +| `browser_type` | Type text into the focused element | +| `browser_press_key` | Press any key with modifier support (Ctrl, Alt, Meta, Shift) | +| `browser_scroll` | Scroll the page by delta pixels | +| `browser_handle_dialog` | Accept or dismiss JS dialogs | + +### πŸ“Έ Visual & Data + +| Tool | What it does | +|------|-------------| +| `browser_screenshot` | Capture a PNG screenshot of the current page | +| `browser_execute_js` | Execute JavaScript and return the result | +| `browser_http_get` | Direct HTTP GET (bypasses browser β€” 10-50x faster for APIs) | +| `browser_wait` | Wait N seconds | +| `browser_wait_for_load` | Wait for `document.readyState === 'complete'` | + +### πŸ”§ Self-Extending Harness + +| Tool | What it does | +|------|-------------| +| `list_dynamic_tools` | See what's already been registered | +| `register_tool` | Inject a new tool (available immediately, next turn) | +| `remove_tool` | Retire a tool that's no longer needed | + +> **Philosophy:** "The agent writes what's missing, mid-task." When the built-in tools aren't enough, the agent extends the harness itself β€” no reload, no restart. + +--- + +## Installation + +### Prerequisites + +| Requirement | Version | Notes | +|------------|---------|-------| +| **pi** | Latest | The pi coding agent CLI | +| **Node.js** | β‰₯ 18 | Runtime for the extension | +| **Chrome / Chromium / Edge** | Any recent | Must support CDP (Chrome DevTools Protocol) | +| **browser-harness daemon** | Latest | Installed automatically by `/browser-setup` | +| **uv** (optional) | Latest | Python package manager; used for daemon installation | +| **git** (optional) | Any | Fallback installation method | + +### Installing the package + +```bash +# From npm (recommended) +pi install npm:pi-browser-harness + +# From git +pi install git:github.com/browser-use/browser-harness + +# Locally (for development) +pi install ./pi-browser-harness +``` + +### Installing the browser-harness daemon + +The daemon (`daemon.py`) is what connects pi-browser-harness to Chrome. Run the guided setup: + +``` +/browser-setup +``` + +This will: +1. Check for an existing browser-harness installation +2. Auto-install via `uv tool install browser-harness` or `git clone` +3. Verify Chrome is running with remote debugging +4. Start the daemon and test the connection + +**Manual installation:** + +```bash +# Option A: uv (recommended) +uv tool install browser-harness + +# Option B: git clone +git clone https://github.com/browser-use/browser-harness ~/Developer/browser-harness +cd ~/Developer/browser-harness && uv sync +``` + +### Enabling Chrome Remote Debugging + +Start Chrome with the `--remote-debugging-port` flag: + +```bash +# macOS +/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \ + --remote-debugging-port=9222 + +# Linux +google-chrome --remote-debugging-port=9222 + +# Windows +"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222 +``` + +> **Edge / Chromium**: Works identically β€” just replace the binary path. + +--- + +## Usage + +### Core Workflow + +The fundamental pattern: **screenshot β†’ interact β†’ screenshot β†’ verify.** + +``` +browser_screenshot() β†’ find the target β†’ browser_click(x, y) β†’ browser_screenshot() +``` + +Every interaction is verified visually. Never assume an action worked. + +### Navigation + +```bash +# First navigation: creates a new tab (preserves user's active tab) +browser_navigate("https://github.com/browser-use/browser-harness") +browser_wait_for_load() +browser_screenshot() +``` + +### Form Filling + +```bash +browser_screenshot() # See the form +browser_click(400, 200) # Click the search input +browser_type("pi browser harness") # Type the query +browser_press_key("Enter") # Submit +browser_wait_for_load() +browser_screenshot() # Verify results +``` + +### Data Extraction + +```bash +# Fast: Direct HTTP for APIs +browser_http_get("https://api.github.com/repos/browser-use/browser-harness") + +# DOM scraping: Execute JS in the rendered page +browser_execute_js("document.querySelector('.price').innerText") + +# Structured: Extract arrays of data +browser_execute_js(` + JSON.stringify( + Array.from(document.querySelectorAll('.result')).map(el => ({ + title: el.querySelector('h3')?.textContent, + link: el.querySelector('a')?.href, + })) + ) +`) +``` + +### Research Workflow (Search + Tabs) + +```bash +# Step 1: Search +browser_navigate("https://google.com/search?q=chrome+devtools+protocol+reference") + +# Step 2: Open result links in parallel tabs +browser_open_urls({ urls: [ + "https://chromedevtools.github.io/devtools-protocol/", + "https://developer.chrome.com/docs/devtools", +] }) + +# Step 3: List all tabs to get target IDs +browser_list_tabs() + +# Step 4: Switch to the tab you want +browser_switch_tab({ targetId: "ABC123..." }) + +# Step 5: Inspect +browser_wait_for_load() +browser_screenshot() +browser_execute_js("document.querySelector('.main-content').innerText") +``` + +### SPAs (React, Vue, etc.) + +`browser_wait_for_load()` only checks `document.readyState === 'complete'`. SPAs often render content after this. Check for specific elements: + +```bash +browser_navigate("https://my-spa.example.com") +browser_wait_for_load() + +# Poll for the element to appear +browser_execute_js("!!document.querySelector('.loaded-content')") +``` + +### Keyboard Modifiers + +`browser_press_key` supports modifier bitfield combinations: + +| Combo | Bitfield | Example | +|-------|----------|---------| +| Ctrl+C | 2 | `browser_press_key("c", { modifiers: 2 })` | +| Cmd+V | 4 | `browser_press_key("v", { modifiers: 4 })` | +| Ctrl+Shift+T | 10 (2\|8) | `browser_press_key("T", { modifiers: 10 })` | +| Cmd+Shift+N | 12 (4\|8) | `browser_press_key("N", { modifiers: 12 })` | + +--- + +## Self-Extending Harness + +When the built-in tools aren't enough, the agent writes its own: + +``` + ● agent: needs to scrape 47 paginated search result pages + β”‚ + ● calls list_dynamic_tools() β†’ empty + β”‚ + ● calls register_tool({ + β”‚ name: "scrape_paginated", + β”‚ implementation: "..." + β”‚ }) + β”‚ + βœ“ next turn: scrape_paginated() is available and callable +``` + +### Tool Lifecycle + +1. **`list_dynamic_tools`** β€” Check what's already registered +2. **`register_tool`** β€” Inject a new tool with custom JavaScript +3. **`remove_tool`** β€” Retire a tool you no longer need + +### Dynamic Tool Bindings + +Tools receive these bindings at runtime: + +| Binding | Description | +|---------|-------------| +| `params` | Tool arguments (as passed by the caller) | +| `daemon` | Browser daemon β€” `daemon.cdp()`, `daemon.evaluateJS()`, `daemon.getPageInfo()` | +| `require` | Node.js `require()` for builtins and installed packages | +| `signal` | `AbortSignal` for cancellation | +| `onUpdate` | Progress callback: `onUpdate({ content: [{ type: 'text', text: '...' }] })` | +| `ctx` | `ExtensionContext` with `cwd`, `sessionManager`, `ui`, `signal` | +| `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout` | Standard globals | + +### Example: Structured Data Extractor + +```javascript +// register_tool implementation: +const info = await daemon.getPageInfo(); +if ("dialog" in info) throw new Error("Dialog is blocking: " + info.dialog.message); + +const data = await daemon.evaluateJS(` + JSON.stringify( + Array.from(document.querySelectorAll('${params.rowSelector}')).map(el => ({ + title: el.querySelector('${params.titleSelector}')?.textContent?.trim(), + price: el.querySelector('${params.priceSelector}')?.textContent?.trim(), + link: el.querySelector('a')?.href, + })) + ) +`); + +const parsed = JSON.parse(data); +return { + content: [{ type: 'text', text: `Extracted ${parsed.length} items:\n\n${JSON.stringify(parsed, null, 2)}` }], + details: { items: parsed, count: parsed.length } +}; +``` + +--- + +## Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ pi agent β”‚ +β”‚ β”‚ β”‚ +β”‚ β”œβ”€ browser_navigate() β”‚ +β”‚ β”œβ”€ browser_screenshot() β”‚ +β”‚ β”œβ”€ browser_click(x, y) β”‚ +β”‚ β”œβ”€ browser_execute_js(...) β”‚ +β”‚ β”œβ”€ list_dynamic_tools() β”‚ +β”‚ β”œβ”€ register_tool(...) β”‚ +β”‚ └─ remove_tool(...) β”‚ +β”‚ β”‚ +β”‚ pi-browser-harness (TypeScript extension) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ Unix socket + β”‚ /tmp/bu-<namespace>.sock + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ daemon.py β”‚ + β”‚ (Python) β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ CDP WebSocket + β”‚ ws://localhost:9222 + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Chrome / Edge β”‚ + β”‚ (user's real β”‚ + β”‚ browser) β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Data Flow + +1. **pi agent** calls a `browser_*` tool (e.g., `browser_screenshot()`) +2. **pi-browser-harness extension** translates the call into a CDP command +3. **JSON-line request** sent over Unix socket to `daemon.py` +4. **daemon.py** relays the command over CDP WebSocket to Chrome +5. **Chrome** executes the command and returns the result +6. Result flows back through the chain to the agent + +### Session Persistence + +- Tab history, daemon namespace, and preferences persist across session reloads and branch navigation +- State is stored via `pi.appendEntry()` in the session's JSONL file +- On `session_start`, state is restored by scanning the branch for the last `browser-harness-state` entry + +### Key Design Decisions + +- **One socket connection per request** β€” stateless, matches the daemon's internal `_send` pattern +- **Session auto-recovery** β€” stale CDP sessions are automatically re-attached +- **Compositor-level clicks** β€” `Input.dispatchMouseEvent` works through iframes, shadow DOM, and cross-origin content without selectors +- **Daemon auto-restart** β€” if the daemon socket dies mid-request, it's restarted and the request is retried once + +--- + +## Configuration + +### CLI Flags + +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--browser-namespace` | `string` | auto-generated | Daemon namespace (`BU_NAME`). Use to run multiple instances. | +| `--browser-debug-clicks` | `boolean` | `false` | Enable debug click overlay (saves annotated screenshots to `/tmp`). | + +### Commands + +| Command | Description | +|---------|-------------| +| `/browser-setup` | Guided setup wizard β€” installs browser-harness, connects to Chrome | +| `/browser-status` | Show daemon health, current page URL/title/viewport | +| `/browser-reload-daemon` | Restart the browser daemon | + +--- + +## Troubleshooting + +| Symptom | Likely Cause | Fix | +|---------|-------------|-----| +| Daemon not starting | Chrome remote debugging not enabled | Start Chrome with `--remote-debugging-port=9222` | +| "DevToolsActivePort not found" | Chrome didn't accept the debugging connection | Open `chrome://inspect/#remote-debugging`, tick the checkbox, click Allow | +| Daemon starts but no tools work | Stale or dead CDP session | Run `/browser-reload-daemon` to restart | +| Dialog blocking page | JS `alert`/`confirm`/`prompt` is open | Run `browser_page_info()` to detect it, then `browser_handle_dialog()` | +| Page not fully loaded | SPA with async rendering | After `browser_wait_for_load()`, use `browser_execute_js()` to check for specific elements | +| `uv` not found | Python package manager not installed | Install via `curl -LsSf https://astral.sh/uv/install.sh \| sh` or use the git fallback | +| Socket permission denied | Stale socket file from previous run | `rm /tmp/bu-*.sock /tmp/bu-*.pid` and restart | +| Windows: `ps` not found | `setup.ts` uses Unix commands | Windows detection is best-effort; see manual installation steps above | + +--- + +## Security + +> **Warning:** This extension gives your pi agent full control over your real Chrome browser. It can see your open tabs, read page content, submit forms, and interact with authenticated sessions. + +- **No sandboxing** β€” the agent has access to whatever Chrome has access to +- **Credentials** β€” if the agent hits an auth wall, it should stop and ask you. Never store credentials in agent prompts. +- **Dynamic tools** β€” `register_tool` evaluates user-supplied JavaScript in the pi process with full `require` access. Review any dynamic tool implementations you didn't write yourself. +- **Data handling** β€” Screenshots are saved to `/tmp`. HTTP responses and JS evaluation results may contain sensitive page data. +- **Session persistence** β€” Tab history and daemon state are stored in pi's session file (`.jsonl`). These contain page URLs and titles. + +--- + +## Contributing + +We welcome contributions! Here's how to get started: + +```bash +# Clone the repository +git clone https://github.com/browser-use/browser-harness +cd browser-harness/pi-browser-harness + +# Install dependencies +npm install + +# Type-check +npm run typecheck + +# Test locally +pi install ./ # Install from local path +``` + +### Development Guidelines + +- **TypeScript strict mode** β€” all code must pass `tsc --noEmit` +- **Error handling** β€” tools must catch errors and return `isError: true` with a descriptive message +- **Tool names** β€” use `snake_case` for consistency with pi conventions +- **Parameters** β€” use `typebox` schemas for all tool parameters +- **Prompt guidelines** β€” every tool must include `promptSnippet` and `promptGuidelines` for LLM context + +### Project Structure + +``` +pi-browser-harness/ +β”œβ”€β”€ src/ +β”‚ β”œβ”€β”€ index.ts # Extension entry point, lifecycle, flags, commands +β”‚ β”œβ”€β”€ daemon.ts # BrowserDaemon: spawns daemon.py, CDP transport +β”‚ β”œβ”€β”€ tools.ts # All browser_* tool registrations +β”‚ β”œβ”€β”€ dynamic-tools.ts # Self-extending harness (register_tool, etc.) +β”‚ β”œβ”€β”€ protocol.ts # Unix socket protocol types and low-level transport +β”‚ β”œβ”€β”€ prompt.ts # System prompt injection +β”‚ β”œβ”€β”€ renderers.ts # Custom TUI renderers +β”‚ β”œβ”€β”€ setup.ts # /browser-setup wizard +β”‚ └── state.ts # Session persistence +β”œβ”€β”€ skills/ +β”‚ └── pi-browser-harness/ +β”‚ └── SKILL.md # User-facing skill documentation +β”œβ”€β”€ package.json +β”œβ”€β”€ tsconfig.json +β”œβ”€β”€ LICENSE +β”œβ”€β”€ CHANGELOG.md +└── README.md +``` + +--- + +## Roadmap + +- [ ] Windows-native setup wizard (remove `ps`/`grep` dependency) +- [ ] iframe-target evaluation support in `browser_execute_js` +- [ ] Remote browser support (connect to Chrome running on another machine) +- [ ] Screenshot annotations (click markers, element highlights) +- [ ] Performance benchmarks and CI test suite +- [ ] Configurable screenshot output directory +- [ ] Recording/playback of interaction sequences +- [ ] Better audio/video support for media-heavy pages + +--- + +## License + +MIT β€” see [LICENSE](LICENSE) for details. + +--- + +<p align="center"> + <sub>Built with ❀️ for the pi ecosystem. Makes coding agents see the web.</sub> +</p>
A package.json

@@ -0,0 +1,63 @@

+{ + "name": "pi-browser-harness", + "version": "0.1.0", + "description": "Browser control extension for pi β€” navigate, click, type, screenshot, and extract data from real Chrome via CDP", + "keywords": [ + "pi-package", + "pi-extension", + "browser", + "cdp", + "chrome", + "web-automation", + "screenshot", + "scraping" + ], + "author": "Pi Browser Harness Contributors", + "license": "MIT", + "repository": { + "type": "git", + "url": "https://github.com/browser-use/browser-harness" + }, + "bugs": { + "url": "https://github.com/browser-use/browser-harness/issues" + }, + "homepage": "https://github.com/browser-use/browser-harness#readme", + "engines": { + "node": ">=18.0.0" + }, + "files": [ + "src/", + "skills/", + "tsconfig.json", + "LICENSE", + "README.md", + "CHANGELOG.md" + ], + "pi": { + "extensions": [ + "./src/index.ts" + ], + "skills": [ + "./skills" + ], + "image": "https://raw.githubusercontent.com/browser-use/browser-harness/main/assets/pi-browser-harness-hero.png" + }, + "peerDependencies": { + "@mariozechner/pi-coding-agent": "*", + "@mariozechner/pi-tui": "*", + "typebox": "*" + }, + "devDependencies": { + "@mariozechner/pi-coding-agent": "*", + "@mariozechner/pi-tui": "*", + "typebox": "*", + "typescript": "^5.0.0" + }, + "scripts": { + "typecheck": "tsc --noEmit", + "prepublishOnly": "tsc --noEmit" + }, + "publishConfig": { + "access": "public" + } +}
A skills/pi-browser-harness/SKILL.md

@@ -0,0 +1,216 @@

+--- +name: pi-browser-harness +description: Direct browser control via CDP. Use when the user wants to automate, scrape, test, or interact with web pages. Connects to the user's already-running Chrome. +--- + +# pi-browser-harness + +Direct browser control via CDP. Gives pi agents full control of a real Chrome browser. +Browser tools coexist with all standard pi tools (read, bash, edit, write, etc.). + +## Setup + +Run `/browser-setup` to install browser-harness and connect to Chrome. Chrome must be running +with remote debugging enabled. Start Chrome with: + +``` +--remote-debugging-port=9222 +``` + +Check status with `/browser-status`. Restart the daemon with `/browser-reload-daemon`. + +## Core Workflow + +``` +browser_screenshot() β†’ browser_click(x, y) β†’ browser_screenshot() +``` + +1. **Screenshot first** β€” see what's on screen before any interaction +2. **Coordinate click** β€” read the target pixel from the screenshot, click at (x, y) +3. **Screenshot again** β€” verify the action worked + +Compositor-level clicks (`Input.dispatchMouseEvent`) work through iframes, shadow DOM, +and cross-origin content. No selectors needed. + +## Available Tools + +### Navigation +- `browser_navigate` β€” navigate to a URL (creates new tab on first call, reuses current tab after) +- `browser_new_tab` β€” open a new tab, optionally navigate (use for first navigation) +- `browser_open_urls` β€” open multiple URLs in parallel tabs (use after search to open result links) +- `browser_go_back` / `browser_go_forward` / `browser_reload` β€” history navigation +- `browser_page_info` β€” get URL, title, viewport, scroll position, or dialog info +- `browser_list_tabs` / `browser_current_tab` / `browser_switch_tab` β€” tab management + +### Interaction +- `browser_click` β€” click at viewport CSS-pixel coordinates +- `browser_type` β€” type text into focused element +- `browser_press_key` β€” press a key (Enter, Tab, Escape, arrows, etc.) +- `browser_scroll` β€” scroll the page at coordinates + +### Visual +- `browser_screenshot` β€” capture PNG screenshot + +### Data Extraction +- `browser_execute_js` β€” execute JavaScript and return result +- `browser_http_get` β€” direct HTTP GET outside browser (10-50x faster for APIs) + +### Utility +- `browser_wait` β€” wait N seconds +- `browser_wait_for_load` β€” wait for document.readyState === 'complete' +- `browser_handle_dialog` β€” accept or dismiss JS dialogs (alert/confirm/prompt) + +## Pattern Reference + +### Navigation +``` +browser_new_tab("https://example.com") β†’ browser_wait_for_load() β†’ browser_screenshot() +``` + +### Form Filling +``` +browser_screenshot() β†’ find input coordinates +β†’ browser_click(x, y) β†’ browser_type("text") +β†’ browser_press_key("Tab") β†’ browser_screenshot() +``` + +### Data Extraction +``` +browser_execute_js("document.querySelector('.price').innerText") +// or for APIs directly: +browser_http_get("https://api.example.com/data") +``` + +### Scrolling +``` +browser_screenshot() β†’ browser_scroll({ deltaY: -500 }) β†’ browser_screenshot() +``` + +### Research (search engine + browser_open_urls) +``` +browser_new_tab("https://google.com/search?q=query") β†’ search engine results +browser_open_urls({urls: ["url1", "url2"]}) β†’ open result pages in parallel tabs +browser_list_tabs() β†’ see all open tabs with targetIds +browser_switch_tab({targetId: "..."}) β†’ switch to a specific tab +browser_wait_for_load() β†’ wait for the page to render +browser_screenshot() β†’ visually inspect the page +browser_execute_js("document.querySelector('.content').innerText") β†’ extract text +``` + +## Self-Extending Harness + +The harness can write its own tools when you need a capability that doesn't exist yet. +This is the same philosophy as browser-harness: "The agent writes what's missing, mid-task." + +``` + ● agent: needs to scrape 47 paginated search result pages + β”‚ + ● calls list_dynamic_tools() β†’ empty + β”‚ + ● calls register_tool({ name: "scrape_paginated", implementation: "..." }) + β”‚ + βœ“ next turn: scrape_paginated() is available and callable +``` + +**Tool lifecycle:** +- `list_dynamic_tools` β€” see what's already been registered +- `register_tool` β€” inject a new tool (available next turn, no reload) +- `remove_tool` β€” retire a tool you no longer need + +**When to extend:** +- You're about to do the same 3+ step sequence more than once +- You need a capability the built-in tools don't provide (e.g., bulk CSV export, structured extraction, pagination) +- You discover a domain-specific pattern that should be reusable +- The alternative is a fragile multi-turn loop of screenshots + clicks + +**What NOT to do with dynamic tools:** +- Don't register tools for one-off actions. Use the built-in browser_* tools directly. +- Don't duplicate existing tools. Call list_dynamic_tools first. +- Don't try to call browser_* tools from within a dynamic tool implementation. Sequence them as separate tool calls. + +**Implementation bindings available inside dynamic tools:** +- `params` β€” the tool's arguments object +- `daemon` β€” browser daemon (daemon.cdp(), daemon.evaluateJS(), daemon.getPageInfo(), etc.) +- `require` β€” Node.js require() (use 'node:fs/promises', 'node:path', 'node:crypto', etc.) +- `signal` β€” AbortSignal for cancellation +- `onUpdate` β€” progress callback: onUpdate({ content: [{ type: 'text', text: '...' }] }) +- `ctx` β€” ExtensionContext (ctx.cwd, ctx.signal, etc.) +- `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout` + +**Example β€” a dynamic tool that extracts structured data:** +```javascript +// register_tool implementation: +const info = await daemon.getPageInfo(); +if ("dialog" in info) throw new Error("Dialog is blocking: " + info.dialog.message); +const data = await daemon.evaluateJS(` + JSON.stringify(Array.from(document.querySelectorAll('${params.rowSelector}')).map(el => ({ + title: el.querySelector('${params.titleSelector}')?.textContent?.trim(), + price: el.querySelector('${params.priceSelector}')?.textContent?.trim(), + }))) +`); +return { content: [{ type: 'text', text: data }], details: { raw: JSON.parse(data) } }; +``` + +## What NOT To Do + +- **Don't launch your own browser** β€” you're connected to the user's real Chrome +- **Don't type credentials** β€” if you hit an auth wall, stop and ask the user +- **Don't assume page state** β€” screenshot to confirm +- **Don't write pixel coordinates into documentation** β€” describe how to locate targets (selectors, text, aria-label) +- **Don't use browser_navigate for first navigation** β€” use browser_new_tab to explicitly preserve the user's active tab (browser_navigate auto-creates a new tab on first call, but explicit is clearer) +- **Don't ignore dialogs** β€” if browser_page_info returns a dialog, handle it first + +## Dialogs + +JS dialogs (alert, confirm, prompt, beforeunload) freeze the page's JS thread. +If browser_page_info returns a dialog, handle it with browser_handle_dialog +before any other browser action. + +``` +// Check page_info first β€” if it shows a dialog: +browser_handle_dialog({ action: "accept" }) // accept/confirm +browser_handle_dialog({ action: "dismiss" }) // dismiss/cancel +browser_handle_dialog({ action: "accept", promptText: "hello" }) // prompt with text +``` + +## SPAs (React, Vue, etc.) + +`browser_wait_for_load()` only checks `document.readyState === "complete"`. +Single-page apps often paint content after this. After wait_for_load, use +`browser_execute_js` to check for specific elements: + +``` +browser_execute_js("!!document.querySelector('.loaded-content')") +``` + +## Keyboard Modifiers + +`browser_press_key` supports modifier bitfield: +- 1 = Alt, 2 = Ctrl, 4 = Meta/Cmd, 8 = Shift +- Combine: Ctrl+Shift = 2|8 = 10, Cmd+Shift = 4|8 = 12 + +## Troubleshooting + +- **Daemon not starting**: Run `/browser-setup` to re-run the guided setup +- **"DevToolsActivePort not found"**: Open chrome://inspect/#remote-debugging, tick checkbox, click Allow +- **Stale session**: Run `/browser-reload-daemon` to restart the daemon +- **Dialog blocking**: Run browser_page_info to detect it, then browser_handle_dialog +- **Status check**: Run `/browser-status` to see daemon health and current page + +## Architecture + +``` +pi agent β†’ pi-browser-harness (TypeScript) + β”‚ Unix socket + β–Ό + /tmp/bu-<namespace>.sock + β”‚ + β–Ό + daemon.py β†’ CDP WebSocket β†’ Chrome + +Dynamic tools (registered at runtime): + list_dynamic_tools / register_tool / remove_tool + β”‚ + β–Ό + pi.registerTool() β†’ available next turn, no reload +```
A src/daemon.ts

@@ -0,0 +1,551 @@

+/** + * BrowserDaemon β€” manages the browser-harness daemon process lifecycle. + * + * Architecture: + * Node.js β†’ Unix socket (/tmp/bu-<namespace>.sock) β†’ daemon.py β†’ CDP WS β†’ Chrome + * + * The daemon is spawned as a child process on start() and communicates via + * the protocol defined in protocol.ts. All CDP calls go through this class. + */ + +import { spawn, type ChildProcess } from "node:child_process"; +import { existsSync } from "node:fs"; +import { access, readFile, unlink, writeFile } from "node:fs/promises"; +import { homedir, tmpdir } from "node:os"; +import { basename, join } from "node:path"; +import { + sendRequest, + daemonAlive, + healthProbe, + isInternalUrl, + type CDPEvent, + type DaemonRequest, + type DaemonResponse, + type DaemonStatus, + type DialogInfo, + type PageInfo, + type PageInfoResult, + type RemoteConfig, + type TabInfo, +} from "./protocol"; + +// ── Constants ──────────────────────────────────────────────────────────────── + +/** Common locations where browser-harness might be cloned */ +const BH_SEARCH_PATHS = [ + join(homedir(), "Developer", "browser-harness"), + join(homedir(), "src", "browser-harness"), + join(homedir(), "browser-harness"), + join(homedir(), "dev", "browser-harness"), + join(homedir(), "Projects", "browser-harness"), +]; + +const DAEMON_START_TIMEOUT_MS = 30_000; +const DAEMON_POLL_INTERVAL_MS = 200; +const DAEMON_SHUTDOWN_TIMEOUT_MS = 5_000; +const SESSION_ATTACH_TIMEOUT_MS = 5_000; + +const VIRTUAL_KEY_CODES: Record<string, number> = { + Enter: 13, + Tab: 9, + Backspace: 8, + Escape: 27, + Delete: 46, + " ": 32, + ArrowLeft: 37, + ArrowUp: 38, + ArrowRight: 39, + ArrowDown: 40, + Home: 36, + End: 35, + PageUp: 33, + PageDown: 34, +}; + +// ── BrowserDaemon ──────────────────────────────────────────────────────────── + +export class BrowserDaemon { + readonly namespace: string; + readonly socketPath: string; + readonly pidPath: string; + readonly logPath: string; + + private proc: ChildProcess | null = null; + private _sessionId: string | null = null; + private _pid: number | null = null; + private remote: RemoteConfig | null = null; + + constructor(namespace = "default", remote?: RemoteConfig) { + this.namespace = namespace; + this.socketPath = `/tmp/bu-${namespace}.sock`; + this.pidPath = `/tmp/bu-${namespace}.pid`; + this.logPath = `/tmp/bu-${namespace}.log`; + this.remote = remote ?? null; + } + + // ── Public getters ─────────────────────────────────────────────────────── + + get sessionId(): string | null { + return this._sessionId; + } + + get pid(): number | null { + return this._pid; + } + + getStatus(): DaemonStatus { + return { + alive: this._sessionId !== null, + sessionId: this._sessionId, + pid: this._pid, + namespace: this.namespace, + socketPath: this.socketPath, + remoteBrowserId: this.remote?.browserId, + }; + } + + // ── Lifecycle ──────────────────────────────────────────────────────────── + + /** + * Start the daemon process and wait for it to become ready. + * Idempotent β€” returns immediately if already alive and healthy. + */ + async start(): Promise<void> { + // Already healthy? + if (await healthProbe(this.socketPath)) { + await this.refreshSessionId(); + return; + } + + // Stale socket but daemon alive? Stop it first. + if (await daemonAlive(this.socketPath)) { + await this.stop(); + } + + const bhDir = await findBrowserHarnessDir(); + const env = this.buildEnv(); + + this.proc = spawn("uv", ["run", "daemon.py"], { + cwd: bhDir, + env: { ...process.env, ...env }, + stdio: ["ignore", "ignore", "ignore"], + detached: true, + }); + this.proc.unref(); + + if (this.proc.pid) { + this._pid = this.proc.pid; + } + + // Poll until socket accepts connections + const deadline = Date.now() + DAEMON_START_TIMEOUT_MS; + let lastError = ""; + + while (Date.now() < deadline) { + if (this.proc.exitCode !== null) { + lastError = await this.readLogTail(); + throw new Error( + `Browser daemon exited (code ${this.proc.exitCode}): ${lastError || "unknown error"}`, + ); + } + + if (await daemonAlive(this.socketPath)) { + // Verify CDP is healthy (not just socket) + if (await healthProbe(this.socketPath)) { + await this.refreshSessionId(); + return; + } + // Socket alive but CDP not ready β€” keep polling + } + + await sleep(DAEMON_POLL_INTERVAL_MS); + } + + lastError = await this.readLogTail(); + // Check if Chrome needs remote debugging setup + if (this.needsChromeSetup(lastError)) { + throw new Error( + `Chrome remote debugging not enabled. Start Chrome with --remote-debugging-port=9222, then retry.\n\nDaemon log: ${lastError}`, + ); + } + + throw new Error( + `Daemon "${this.namespace}" failed to start after ${DAEMON_START_TIMEOUT_MS / 1000}s. ` + + `Log: ${lastError || "no log output"}`, + ); + } + + /** + * Gracefully stop the daemon: send shutdown, wait for exit, force-kill if needed. + */ + async stop(): Promise<void> { + // Send shutdown via socket + try { + await sendRequest(this.socketPath, { meta: "shutdown" }, 3000); + } catch { + // Socket may already be dead β€” that's fine + } + + // Wait for process to exit + if (this.proc && this.proc.exitCode === null) { + const deadline = Date.now() + DAEMON_SHUTDOWN_TIMEOUT_MS; + while (Date.now() < deadline) { + if (this.proc.exitCode !== null) break; + await sleep(100); + } + + // Force kill if still alive + if (this.proc.exitCode === null) { + try { + this.proc.kill("SIGTERM"); + } catch { + // already dead + } + } + } + + // Clean up files + for (const f of [this.socketPath, this.pidPath]) { + try { + await unlink(f); + } catch { + // file doesn't exist + } + } + + this.proc = null; + this._sessionId = null; + this._pid = null; + } + + /** + * Ensure daemon is alive and healthy. Restarts if stale. + * Call before every tool execution. + */ + async ensureAlive(): Promise<void> { + if (!(await healthProbe(this.socketPath))) { + await this.stop(); + await this.start(); + } + } + + // ── CDP Communication ──────────────────────────────────────────────────── + + /** + * Send a raw request to the daemon and get the response. + */ + async send(request: DaemonRequest): Promise<DaemonResponse> { + await this.ensureAlive(); + + try { + const response = await sendRequest(this.socketPath, request); + return response; + } catch (err) { + // Socket died mid-request β€” restart and retry once + const message = err instanceof Error ? err.message : String(err); + if (message.includes("ENOENT") || message.includes("ECONNREFUSED") || message.includes("EPIPE")) { + await this.stop(); + await this.start(); + return sendRequest(this.socketPath, request); + } + throw err; + } + } + + /** + * Convenience: send a CDP method and return the result. + * Handles session management β€” Target.* methods skip session, + * others use explicit sessionId or the daemon's default session. + */ + async cdp( + method: string, + params?: Record<string, unknown>, + sessionId?: string | null, + ): Promise<unknown> { + // Browser-level Target.* calls must NOT use a session + const sid = method.startsWith("Target.") ? null : (sessionId ?? this._sessionId); + + const response = await this.send({ + method, + params: params ?? {}, + session_id: sid, + }); + + if (response.error) { + // Auto-recover from stale session + if ( + response.error.includes("Session with given id not found") && + sid === this._sessionId + ) { + await this.attachFirstPage(); + const retry = await this.send({ + method, + params: params ?? {}, + session_id: this._sessionId, + }); + if (retry.error) { + throw new Error(`CDP error after session recovery (${method}): ${retry.error}`); + } + return retry.result; + } + throw new Error(`CDP error (${method}): ${response.error}`); + } + + return response.result; + } + + // ── High-level Browser Helpers ──────────────────────────────────────────── + + /** Get page state or dialog info */ + async getPageInfo(): Promise<PageInfoResult> { + // Check for JS dialog first + try { + const dialogResp = await sendRequest(this.socketPath, { meta: "pending_dialog" }); + if (dialogResp.dialog) { + return { dialog: dialogResp.dialog }; + } + } catch { + // ignore β€” dialog check is best-effort + } + + const js = `JSON.stringify({url:location.href,title:document.title,w:innerWidth,h:innerHeight,sx:scrollX,sy:scrollY,pw:document.documentElement.scrollWidth,ph:document.documentElement.scrollHeight})`; + const raw = await this.evaluateJS(js); + return JSON.parse(raw as string) as PageInfo; + } + + /** Capture a PNG screenshot, write to path, return path */ + async captureScreenshot(outputPath: string, fullPage = false): Promise<string> { + const result = (await this.cdp("Page.captureScreenshot", { + format: "png", + captureBeyondViewport: fullPage, + })) as { data: string }; + + const buf = Buffer.from(result.data, "base64"); + await writeFile(outputPath, buf); + return outputPath; + } + + /** List browser tabs */ + async listTabs(includeChrome = true): Promise<TabInfo[]> { + const result = (await this.cdp("Target.getTargets")) as { + targetInfos: Array<{ targetId: string; type: string; title: string; url: string }>; + }; + + const tabs: TabInfo[] = []; + for (const t of result.targetInfos) { + if (t.type !== "page") continue; + if (!includeChrome && isInternalUrl(t.url)) continue; + tabs.push({ targetId: t.targetId, title: t.title || "", url: t.url || "" }); + } + return tabs; + } + + /** Get current tab info */ + async currentTab(): Promise<TabInfo> { + const result = (await this.cdp("Target.getTargetInfo")) as { + targetInfo: { targetId: string; url: string; title: string }; + }; + const t = result.targetInfo; + return { targetId: t.targetId, url: t.url || "", title: t.title || "" }; + } + + /** Create a new tab and switch to it. Optionally navigate. */ + async newTab(url?: string): Promise<string> { + const result = (await this.cdp("Target.createTarget", { + url: "about:blank", + })) as { targetId: string }; + + await this.switchTab(result.targetId); + if (url && url !== "about:blank") { + await this.cdp("Page.navigate", { url }); + } + return result.targetId; + } + + /** Activate and attach to a tab */ + async switchTab(targetId: string): Promise<void> { + await this.cdp("Target.activateTarget", { targetId }); + const result = (await this.cdp("Target.attachToTarget", { + targetId, + flatten: true, + })) as { sessionId: string }; + + // Tell the daemon to use this as default session + this._sessionId = result.sessionId; + await sendRequest(this.socketPath, { + meta: "set_session", + session_id: result.sessionId, + }); + + // Mark the tab with a green circle for user visibility + try { + await this.cdp("Runtime.evaluate", { + expression: `if(!document.title.startsWith('🟒'))document.title='🟒 '+document.title`, + }); + } catch { + // best-effort + } + } + + /** Evaluate JavaScript in the page (or a specific iframe target) */ + async evaluateJS(expression: string, targetId?: string): Promise<unknown> { + // Auto-wrap return statements in IIFE + if (expression.includes("return ") && !expression.trim().startsWith("(")) { + expression = `(function(){${expression}})()`; + } + + const sid = targetId || null; // explicit target or daemon default + const result = (await this.cdp( + "Runtime.evaluate", + { + expression, + returnByValue: true, + awaitPromise: true, + }, + targetId ? sid : undefined, + )) as { result?: { value?: unknown }; exceptionDetails?: unknown }; + + if (result.exceptionDetails) { + throw new Error(`JS evaluation failed: ${JSON.stringify(result.exceptionDetails)}`); + } + return result.result?.value; + } + + /** Drain buffered CDP events from the daemon */ + async drainEvents(): Promise<CDPEvent[]> { + const resp = await sendRequest(this.socketPath, { meta: "drain_events" }); + return resp.events ?? []; + } + + /** Get virtual key code for a key name or character */ + getVirtualKeyCode(key: string): number { + return VIRTUAL_KEY_CODES[key] ?? (key.length === 1 ? key.charCodeAt(0) : 0); + } + + /** Get the key code string for CDP (uses the key name or the key itself) */ + getKeyCode(key: string): string { + return key.length === 1 && !VIRTUAL_KEY_CODES[key] ? key : key; + } + + // ── Internal Helpers ────────────────────────────────────────────────────── + + /** Attach to the first real page target. Creates about:blank if none exist. */ + private async attachFirstPage(): Promise<string> { + const result = (await this.cdp("Target.getTargets")) as { + targetInfos: Array<{ targetId: string; type: string; url: string }>; + }; + + const pages = result.targetInfos.filter( + (t) => t.type === "page" && !isInternalUrl(t.url), + ); + + if (pages.length === 0) { + const created = (await this.cdp("Target.createTarget", { + url: "about:blank", + })) as { targetId: string }; + pages.push({ targetId: created.targetId, type: "page", url: "about:blank" }); + } + + const attachResult = (await this.cdp("Target.attachToTarget", { + targetId: pages[0].targetId, + flatten: true, + })) as { sessionId: string }; + + this._sessionId = attachResult.sessionId; + + // Enable core domains + for (const domain of ["Page", "DOM", "Runtime", "Network"]) { + try { + await this.cdp(`${domain}.enable`); + } catch { + // best-effort + } + } + + return attachResult.sessionId; + } + + /** Refresh session ID from the daemon's current session */ + private async refreshSessionId(): Promise<void> { + try { + const resp = await sendRequest(this.socketPath, { meta: "session" }); + if (resp.session_id) { + this._sessionId = resp.session_id; + } + } catch { + // will be set on first attach + } + } + + /** Build environment variables for the daemon process */ + private buildEnv(): Record<string, string> { + const env: Record<string, string> = { + BU_NAME: this.namespace, + }; + + if (this.remote) { + env.BU_CDP_WS = this.remote.cdpUrl; + env.BU_BROWSER_ID = this.remote.browserId; + } + + return env; + } + + /** Read the last line of the daemon log */ + private async readLogTail(): Promise<string> { + try { + const content = await readFile(this.logPath, "utf-8"); + const lines = content.trim().split("\n"); + return lines[lines.length - 1] || ""; + } catch { + return ""; + } + } + + /** Check if the error message suggests Chrome needs remote debugging setup */ + private needsChromeSetup(errorMsg: string): boolean { + const lower = errorMsg.toLowerCase(); + return ( + lower.includes("devtoolsactiveport not found") || + lower.includes("enable chrome://inspect") || + lower.includes("not live yet") || + (lower.includes("ws handshake failed") && + (lower.includes("403") || lower.includes("timed out"))) + ); + } +} + +// ── Utility ────────────────────────────────────────────────────────────────── + +function sleep(ms: number): Promise<void> { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** + * Find the browser-harness repository directory. + * Searches common locations and falls back to cwd. + */ +async function findBrowserHarnessDir(): Promise<string> { + // Check common locations + for (const dir of BH_SEARCH_PATHS) { + if ( + existsSync(join(dir, "daemon.py")) && + existsSync(join(dir, "helpers.py")) + ) { + return dir; + } + } + + // Check cwd + const cwd = process.cwd(); + if (existsSync(join(cwd, "daemon.py")) && existsSync(join(cwd, "helpers.py"))) { + return cwd; + } + + throw new Error( + "browser-harness not found. Clone it first:\n" + + " git clone https://github.com/browser-use/browser-harness ~/Developer/browser-harness\n" + + " cd ~/Developer/browser-harness && uv sync\n\n" + + `Searched: ${BH_SEARCH_PATHS.join(", ")}`, + ); +}
A src/dynamic-tools.ts

@@ -0,0 +1,335 @@

+/** + * dynamic-tools.ts β€” self-writing tools for pi-browser-harness. + * + * Mirrors the browser-harness philosophy: the agent writes what's missing, + * mid-task. No framework, no recipes, no rails. + * + * Provides three management tools: + * list_dynamic_tools β€” see what's already registered + * register_tool β€” inject a new tool into the harness + * remove_tool β€” retire a tool that's no longer needed + * + * Tools registered via register_tool (which internally calls pi.registerTool()) + * are available immediately in the same session β€” the LLM can call them on + * the very next sub-turn within the same agent processing cycle, without + * needing /reload. The agent decides to extend the harness the same way + * it decides to write any code: it sees a gap and fills it. + */ + +import type { ExtensionAPI } from "@mariozechner/pi-coding-agent"; +import { Type } from "typebox"; +import type { BrowserDaemon } from "./daemon"; + +// ── Types ──────────────────────────────────────────────────────────────────── + +interface DynamicToolDef { + name: string; + label: string; + description: string; + execute: (params: Record<string, unknown>, signal: AbortSignal | undefined, onUpdate: ((update: unknown) => void) | undefined, ctx: unknown) => Promise<{ content: Array<{ type: "text"; text: string }>; details?: Record<string, unknown> }>; +} + +// ── AsyncFunction constructor ──────────────────────────────────────────────── + +const AsyncFunction = Object.getPrototypeOf(async () => {}).constructor as new (...args: string[]) => (...args: unknown[]) => Promise<unknown>; + +// ── Registration ───────────────────────────────────────────────────────────── + +export function registerDynamicTools(pi: ExtensionAPI, daemon: BrowserDaemon): void { + const registry = new Map<string, DynamicToolDef>(); + + // ═══════════════════════════════════════════════════════════════════════════ + // list_dynamic_tools + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "list_dynamic_tools", + label: "List Dynamic Tools", + description: + "List all dynamically registered tools with their names, labels, and descriptions. " + + "Call this FIRST when you're unsure whether a needed capability already exists β€” " + + "before writing a new tool.", + promptSnippet: "List all dynamically registered tools (names + descriptions)", + promptGuidelines: [ + "Call list_dynamic_tools before register_tool to avoid duplicating existing capabilities.", + "If a needed tool exists but has the wrong shape, use remove_tool to replace it with a corrected version.", + ], + parameters: Type.Object({}), + async execute() { + if (registry.size === 0) { + return { + content: [{ + type: "text" as const, + text: "No dynamic tools registered yet. The harness is clean β€” use register_tool to teach it something new.", + }], + details: { tools: [], count: 0 }, + }; + } + + const tools = Array.from(registry.values()).map((t) => ({ + name: t.name, + label: t.label, + description: t.description, + })); + + return { + content: [{ + type: "text" as const, + text: `${tools.length} dynamic tool(s) registered:\n\n` + + tools.map((t) => `β€’ ${t.name} β€” ${t.description}`).join("\n"), + }], + details: { tools, count: tools.length }, + }; + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // register_tool + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "register_tool", + label: "Register Dynamic Tool", + description: + "Register a new tool by providing its JavaScript implementation. " + + "The tool becomes available immediately in the same session β€” " + + "the LLM can call it on the very next sub-turn within the same " + + "agent processing cycle, without reloading. " + + "Use this whenever you discover you need a capability the harness doesn't " + + "already have. Don't work around limitations, extend the harness.\n\n" + + "The implementation runs as an async function body. It receives these bindings:\n" + + " params β€” the tool's arguments (as passed by the caller)\n" + + " daemon β€” the browser daemon (for composing browser actions: daemon.cdp(), daemon.evaluateJS(), daemon.getPageInfo(), etc.)\n" + + " require β€” Node.js require() for builtins and installed packages\n" + + " signal β€” AbortSignal for cancellation\n" + + " onUpdate β€” progress callback: onUpdate({ content: [{ type: 'text', text: '...' }] })\n" + + " ctx β€” ExtensionContext with cwd, sessionManager, ui, signal, etc.\n" + + " console, fetch, JSON, Buffer, setTimeout, clearTimeout\n\n" + + "The implementation MUST return { content: [{ type: 'text', text: '...' }], details?: {...} }.\n" + + "For errors, throw: throw new Error('something went wrong').\n\n" + + "Example implementation:\n" + + ' const info = await daemon.getPageInfo();\n' + + ' if ("dialog" in info) throw new Error("Dialog is blocking the page");\n' + + ' return { content: [{ type: "text", text: `Current URL: ${info.url}` }], details: { url: info.url } };', + promptSnippet: "Register a new tool with custom JavaScript logic (available immediately, next sub-turn)", + promptGuidelines: [ + "Use register_tool when you need a capability not provided by the built-in browser_* tools. The new tool is available immediately β€” the LLM can call it on the very next sub-turn within the same agent processing cycle, without reloading.", + "Before registering, always call list_dynamic_tools to check if a suitable tool already exists.", + "The implementation has access to the browser daemon via `daemon` β€” use daemon.cdp(), daemon.evaluateJS(), daemon.getPageInfo(), etc. to compose browser actions inside the tool.", + "For pure data processing or file operations, use require() to access Node.js modules like fs, path, crypto.", + "If the implementation needs existing browser tools (browser_navigate, browser_screenshot, etc.), break those into separate tool calls before or after the dynamic tool β€” do not try to invoke them from within the implementation.", + ], + parameters: Type.Object({ + name: Type.String({ + description: "Unique tool name in snake_case (e.g. 'scrape_paginated_results', 'extract_structured_data')", + }), + label: Type.String({ + description: "Human-readable label for the tool (e.g. 'Scrape Paginated Results')", + }), + description: Type.String({ + description: "What the tool does, including expected parameter names and their types. This is shown to the LLM when it considers which tool to call.", + }), + implementation: Type.String({ + description: + "JavaScript code (async function body). Bindings: params, daemon, require, signal, onUpdate, ctx, console, fetch, JSON, Buffer, setTimeout, clearTimeout. Must return { content: [{ type: 'text', text: '...' }], details?: {...} }.", + }), + }), + async execute(_id, p, _signal, _onUpdate) { + const { name, label, description, implementation } = p as { + name: string; + label: string; + description: string; + implementation: string; + }; + + // ── Validation ────────────────────────────────────────────────── + if (!/^[a-z][a-z0-9_]*$/.test(name)) { + return { + content: [{ + type: "text" as const, + text: `Invalid tool name '${name}'. Must be snake_case: lowercase letters, digits, and underscores, starting with a letter.`, + }], + details: { error: "invalid_name" }, + }; + } + + if (registry.has(name)) { + const existing = registry.get(name)!; + return { + content: [{ + type: "text" as const, + text: `Tool '${name}' already exists: ${existing.description}\n\nUse remove_tool("${name}") first if you want to replace it, then call register_tool again.`, + }], + details: { exists: true, existingLabel: existing.label }, + }; + } + + // ── Compile ───────────────────────────────────────────────────── + let executeFn: (...args: unknown[]) => Promise<unknown>; + try { + // The LLM writes a function body. We wrap it in an async function. + // Prepend "use strict" so accidental global leaks become errors. + const source = `"use strict";\n${implementation}`; + executeFn = new AsyncFunction( + "params", "daemon", "require", "signal", "onUpdate", "ctx", + "console", "fetch", "JSON", "Buffer", "setTimeout", "clearTimeout", + source, + ); + } catch (err: unknown) { + const msg = err instanceof Error ? err.message : String(err); + return { + content: [{ + type: "text" as const, + text: `Syntax error in implementation: ${msg}\n\nFix the JavaScript syntax and try again.`, + }], + details: { error: "syntax_error", message: msg }, + }; + } + + // ── Smoke test ────────────────────────────────────────────────── + // Run the function once with empty params to catch obvious errors early. + try { + const testResult = await executeFn( + {}, + daemon, + require, + undefined, + (_update: unknown) => {}, + { cwd: process.cwd() }, + console, + fetch, + JSON, + Buffer, + setTimeout, + clearTimeout, + ); + if (!testResult || typeof testResult !== "object" || !Array.isArray((testResult as any).content)) { + return { + content: [{ + type: "text" as const, + text: `Implementation must return an object with a 'content' array. Got: ${JSON.stringify(testResult)}`, + }], + details: { error: "invalid_return" }, + }; + } + } catch (err: unknown) { + // Smoke test failure is ok β€” the params were empty, so runtime errors + // from missing params are expected. Only reject if the error looks + // like a code bug (e.g. ReferenceError for undefined variables). + const msg = err instanceof Error ? err.message : String(err); + if (msg.includes("is not defined") || msg.includes("Cannot find module")) { + return { + content: [{ + type: "text" as const, + text: `Implementation failed smoke test: ${msg}\n\nCheck for undefined variables, missing imports, or broken require() calls.`, + }], + details: { error: "smoke_test_failed", message: msg }, + }; + } + // Otherwise it's likely just missing params β€” proceed. + } + + // ── Register ──────────────────────────────────────────────────── + const def: DynamicToolDef = { + name, + label, + description, + execute: async (params, signal, onUpdate, ctx) => { + const result = await executeFn( + params, + daemon, + require, + signal, + onUpdate, + ctx, + console, + fetch, + JSON, + Buffer, + setTimeout, + clearTimeout, + ); + return result as { content: Array<{ type: "text"; text: string }>; details?: Record<string, unknown> }; + }, + }; + + registry.set(name, def); + + // Register with pi β€” available immediately on the next sub-turn + pi.registerTool({ + name, + label, + description, + parameters: Type.Object({}, { additionalProperties: true }), + async execute(_toolCallId, execParams, execSignal, execOnUpdate, execCtx) { + const tool = registry.get(name); + if (!tool) { + throw new Error(`Tool '${name}' was removed. Use list_dynamic_tools to see available tools.`); + } + return tool.execute( + (execParams ?? {}) as Record<string, unknown>, + execSignal, + execOnUpdate, + execCtx, + ); + }, + }); + + return { + content: [{ + type: "text" as const, + text: `βœ… Tool '${name}' registered and available immediately.\n\nLabel: ${label}\nDescription: ${description}\n\nYou can now call it as: ${name}(...)`, + }], + details: { name, label }, + }; + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // remove_tool + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "remove_tool", + label: "Remove Dynamic Tool", + description: + "Remove a previously registered dynamic tool. The tool will not be available on subsequent turns. " + + "Use this to replace a broken tool or clean up tools you no longer need.", + promptSnippet: "Remove a dynamically registered tool", + promptGuidelines: [ + "Use remove_tool to retire a dynamic tool you no longer need or to replace it with a corrected version.", + "After removing a tool, you can immediately register a replacement with the same name via register_tool.", + ], + parameters: Type.Object({ + name: Type.String({ description: "Name of the tool to remove (as shown by list_dynamic_tools)" }), + }), + async execute(_id, p) { + const { name } = p as { name: string }; + const existed = registry.has(name); + + if (!existed) { + return { + content: [{ + type: "text" as const, + text: `Tool '${name}' is not in the dynamic registry. Use list_dynamic_tools to see registered tools.`, + }], + details: { removed: false }, + }; + } + + // Remove from registry + registry.delete(name); + + // Deactivate from pi's active tool set so it doesn't appear in the system prompt + const activeNames = pi.getActiveTools() + .map((t: unknown) => (t as { name: string }).name) + .filter((n: string) => n !== name); + pi.setActiveTools(activeNames); + + return { + content: [{ + type: "text" as const, + text: `πŸ—‘οΈ Tool '${name}' removed. It will no longer appear in the system prompt or be callable.`, + }], + details: { removed: true }, + }; + }, + }); +}
A src/index.ts

@@ -0,0 +1,228 @@

+/** + * pi-browser-harness β€” browser control extension for pi. + * + * Gives pi agents full control of a real Chrome browser via CDP. + * Spawns the browser-harness daemon, registers browser_* tools, + * and injects browser control guidance into the system prompt. + * + * Install: + * pi install npm:pi-browser-harness + * # or copy to .pi/extensions/pi-browser-harness/ + * + * Commands: + * /browser-setup β€” guided setup wizard + * /browser-status β€” show daemon status and current page + * /browser-reload-daemon β€” restart the daemon + * + * Flags: + * --browser-namespace <name> β€” override BU_NAME (default: auto) + * --browser-debug-clicks β€” enable debug click overlay + */ + +import type { ExtensionAPI, ExtensionContext } from "@mariozechner/pi-coding-agent"; +import { BrowserDaemon } from "./daemon"; +import { getBrowserSystemPrompt } from "./prompt"; +import { registerRenderers } from "./renderers"; +import { registerSetupCommand } from "./setup"; +import { type BrowserState, defaultState, persistState, restoreState } from "./state"; +import { cleanupTempDirs, registerTools } from "./tools"; +import { registerDynamicTools } from "./dynamic-tools"; + +export default function browserHarnessExtension(pi: ExtensionAPI) { + // ── Resolve namespace ────────────────────────────────────────────────────── + const flagNs = pi.getFlag("browser-namespace") as string | undefined; + const namespace = flagNs || `pi-${Math.random().toString(36).slice(2, 10)}`; + + // ── State ────────────────────────────────────────────────────────────────── + let state: BrowserState = defaultState(namespace); + let daemon: BrowserDaemon | null = null; + + // ── Tool initialization flag ─────────────────────────────────────────────── + let toolsRegistered = false; + + // ═══════════════════════════════════════════════════════════════════════════ + // Flags + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerFlag("browser-namespace", { + description: "Browser daemon namespace (BU_NAME). Default: auto-generated", + type: "string", + }); + pi.registerFlag("browser-debug-clicks", { + description: "Enable debug click overlay (saves annotated screenshots to /tmp)", + type: "boolean", + default: false, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // Commands + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerCommand("browser-status", { + description: "Show browser daemon status and current page", + handler: async (_args, ctx) => { + if (!daemon) { + ctx.ui.notify("Browser daemon not started. Run /browser-setup first.", "warning"); + return; + } + + const status = daemon.getStatus(); + const lines = [ + `Browser Daemon: ${status.alive ? "🟒 Connected" : "πŸ”΄ Disconnected"}`, + `Namespace: ${status.namespace}`, + `Socket: ${status.socketPath}`, + `Session: ${status.sessionId || "none"}`, + `PID: ${status.pid || "none"}`, + ]; + + if (status.remoteBrowserId) { + lines.push(`Remote Browser: ${status.remoteBrowserId}`); + } + + // Try to get current page info + if (status.alive) { + try { + const info = await daemon.getPageInfo(); + if ("dialog" in info) { + lines.push(`\n⚠️ Dialog open: ${info.dialog.type} β€” "${info.dialog.message}"`); + } else { + lines.push(`\nCurrent Page:`); + lines.push(` URL: ${info.url}`); + lines.push(` Title: ${info.title}`); + lines.push(` Viewport: ${info.width}x${info.height}`); + } + } catch { + lines.push("\n(Could not read page info)"); + } + } + + ctx.ui.notify(lines.join("\n"), "info"); + }, + }); + + pi.registerCommand("browser-reload-daemon", { + description: "Restart the browser daemon", + handler: async (_args, ctx) => { + if (!daemon) { + ctx.ui.notify("Browser daemon not started.", "warning"); + return; + } + + ctx.ui.notify("Restarting browser daemon...", "info"); + try { + await daemon.stop(); + await daemon.start(); + ctx.ui.notify("Browser daemon restarted βœ“", "info"); + } catch (err) { + ctx.ui.notify( + `Restart failed: ${err instanceof Error ? err.message : String(err)}`, + "error", + ); + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // Session lifecycle + // ═══════════════════════════════════════════════════════════════════════════ + pi.on("session_start", async (_event, ctx) => { + // Restore persisted state (preserving the current namespace from flag) + state = restoreState(ctx, state.namespace); + + // Create daemon instance + daemon = new BrowserDaemon(state.namespace); + + // Attempt to start daemon (will fail gracefully if Chrome not connected) + try { + await daemon.start(); + } catch { + // Don't block β€” user can run /browser-setup later + } + + // Register tools (once) + if (!toolsRegistered && daemon) { + registerTools(pi, daemon); + registerDynamicTools(pi, daemon); + toolsRegistered = true; + } + + // Register renderers + registerRenderers(pi); + + // Setup command (always available) + if (daemon) { + registerSetupCommand(pi, daemon); + } + + // Update status + if (daemon.getStatus().alive) { + ctx.ui.setStatus("browser", "🟒 Browser connected"); + } else { + ctx.ui.setStatus("browser", "πŸ”΄ Browser β€” run /browser-setup"); + } + }); + + pi.on("session_shutdown", async (_event) => { + // Persist current state + if (state) { + persistState(pi, state); + } + + // Stop daemon + if (daemon) { + try { + await daemon.stop(); + } catch { + // best-effort + } + daemon = null; + } + + toolsRegistered = false; + + // Clean up temp files from truncated tool outputs + await cleanupTempDirs(); + }); + + // Restore state on tree navigation + pi.on("session_tree", async (_event, ctx) => { + state = restoreState(ctx, daemon?.namespace); + if (daemon) { + persistState(pi, state); + } + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // System prompt injection + // ═══════════════════════════════════════════════════════════════════════════ + pi.on("before_agent_start", async (event) => { + if (!daemon || !daemon.getStatus().alive) { + // Browser not connected β€” inject minimal note + return { + systemPrompt: + event.systemPrompt + + `\n\n## Browser Control\n\nBrowser tools (browser_*) are available but the browser daemon is not connected. ` + + `Run /browser-setup to connect to Chrome, or /browser-status to check.`, + }; + } + + return { + systemPrompt: event.systemPrompt + getBrowserSystemPrompt(), + }; + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // Tool result hook: track tab history + // ═══════════════════════════════════════════════════════════════════════════ + pi.on("tool_result", async (event) => { + if (event.toolName === "browser_new_tab" || event.toolName === "browser_navigate") { + // Track new tabs in history + const details = event.details as { targetId?: string } | undefined; + if (details?.targetId) { + state = { + ...state, + tabHistory: [details.targetId, ...state.tabHistory.filter((id) => id !== details.targetId)].slice(0, 20), + }; + persistState(pi, state); + } + } + }); +}
A src/prompt.ts

@@ -0,0 +1,78 @@

+/** + * System prompt guidance for browser usage. + * + * Injects a compact browser control reference into the pi system prompt. + * Per-tool guidelines are injected via promptGuidelines on each tool registration. + * This prompt provides only the cross-cutting patterns and tool reference table. + */ + +export function getBrowserSystemPrompt(): string { + return ` +## Browser Control + +You have full control of a real Chrome browser via \`browser_*\` tools. +The browser connects to the user's running Chrome (not a headless instance). + +**These tools coexist with all other standard pi tools** β€” read, bash, edit, +write, and any other built-in tools are still available. Use browser_* tools +to interact with pages visually, and other tools alongside them. + +### Browser-Specific Tools + +| Tool | Purpose | +|------|---------| +| browser_screenshot | Capture a PNG screenshot of the current page | +| browser_click | Click at viewport coordinates | +| browser_type | Type text into the focused element | +| browser_press_key | Press a keyboard key (Enter, Tab, Escape, arrows, etc.) | +| browser_scroll | Scroll the page at coordinates | +| browser_navigate | Navigate to a URL | +| browser_new_tab | Open a new tab, optionally navigate | +| browser_open_urls | Open multiple URLs in new tabs (parallel) | +| browser_go_back / browser_go_forward / browser_reload | History navigation | +| browser_page_info | Get page URL, title, viewport, scroll position, or dialog info | +| browser_list_tabs | List all open browser tabs | +| browser_current_tab | Get current tab info | +| browser_switch_tab | Switch to a different tab by targetId | +| browser_execute_js | Execute JavaScript and return the result | +| browser_http_get | Direct HTTP GET (outside browser, for APIs) | +| browser_wait | Wait N seconds | +| browser_wait_for_load | Wait for document.readyState === 'complete' | +| browser_handle_dialog | Accept or dismiss a JS dialog | + +### Common Patterns + +**Navigation:** +\`\`\` +browser_new_tab("https://example.com") β†’ browser_wait_for_load() β†’ browser_screenshot() +\`\`\` + +**Form filling:** +\`\`\` +browser_screenshot() β†’ find input coordinates β†’ browser_click(x, y) +β†’ browser_type("text") β†’ browser_press_key("Tab") β†’ browser_screenshot() +\`\`\` + +**Data extraction:** +\`\`\` +browser_execute_js("document.querySelector('.price').innerText") +// or for APIs: +browser_http_get("https://api.example.com/data") +\`\`\` + +**Scrolling:** +\`\`\` +browser_screenshot() β†’ browser_scroll({ deltaY: -500 }) β†’ browser_screenshot() +\`\`\` + +**Research Workflow (search + browser):** +\`\`\` +browser_navigate("https://google.com/search?q=...") β†’ search engine in a tab +browser_open_urls(urls: ["url1", "url2", ...]) β†’ open result pages in parallel tabs +browser_list_tabs() β†’ see all open tabs with targetIds +browser_switch_tab(targetId: "...") β†’ switch to a tab +browser_screenshot() β†’ visually inspect the page +browser_execute_js("document.querySelector('.main').innerText") β†’ extract content +\`\`\` +`; +}
A src/protocol.ts

@@ -0,0 +1,201 @@

+/** + * Protocol types and low-level socket communication for the browser-harness daemon. + * + * The daemon exposes a Unix socket at /tmp/bu-<NAME>.sock. + * Protocol: one JSON-line request β†’ one JSON-line response per connection. + */ + +import { createConnection } from "node:net"; + +// ── Request ────────────────────────────────────────────────────────────────── + +export interface DaemonRequest { + /** CDP method (e.g. "Page.navigate") β€” absent for meta requests */ + method?: string; + /** CDP parameters */ + params?: Record<string, unknown>; + /** CDP session ID β€” null/absent for browser-level (Target.*) calls */ + session_id?: string | null; + /** Meta command: "drain_events" | "set_session" | "pending_dialog" | "shutdown" */ + meta?: string; +} + +// ── Response ───────────────────────────────────────────────────────────────── + +export interface DaemonResponse { + result?: unknown; + error?: string; + events?: CDPEvent[]; + session_id?: string; + dialog?: DialogInfo; + ok?: boolean; +} + +// ── CDP Event ──────────────────────────────────────────────────────────────── + +export interface CDPEvent { + method: string; + params: unknown; + session_id?: string; +} + +// ── Dialog ─────────────────────────────────────────────────────────────────── + +export interface DialogInfo { + type: "alert" | "confirm" | "prompt" | "beforeunload"; + message: string; + defaultPrompt?: string; +} + +// ── Tab ────────────────────────────────────────────────────────────────────── + +export interface TabInfo { + targetId: string; + title: string; + url: string; +} + +// ── Page State ─────────────────────────────────────────────────────────────── + +export interface PageInfo { + url: string; + title: string; + width: number; + height: number; + scrollX: number; + scrollY: number; + pageWidth: number; + pageHeight: number; +} + +export type PageInfoResult = PageInfo | { dialog: DialogInfo }; + +// ── Daemon Status ──────────────────────────────────────────────────────────── + +export interface DaemonStatus { + alive: boolean; + sessionId: string | null; + pid: number | null; + namespace: string; + socketPath: string; + remoteBrowserId?: string; +} + +// ── Remote Config ──────────────────────────────────────────────────────────── + +export interface RemoteConfig { + cdpUrl: string; + browserId: string; +} + +// ── Internal URL prefixes to filter ────────────────────────────────────────── + +const INTERNAL_PREFIXES = [ + "chrome://", + "chrome-untrusted://", + "devtools://", + "chrome-extension://", + "about:", +]; + +export function isInternalUrl(url: string): boolean { + return INTERNAL_PREFIXES.some((p) => url.startsWith(p)); +} + +// ── Low-level socket send ──────────────────────────────────────────────────── + +/** + * Send a single JSON-line request to the daemon socket and read one JSON-line response. + * Opens a fresh connection per request (stateless, matches helpers.py _send pattern). + */ +export function sendRequest( + socketPath: string, + request: DaemonRequest, + timeoutMs = 15_000, +): Promise<DaemonResponse> { + return new Promise((resolve, reject) => { + const socket = createConnection(socketPath); + + const timer = setTimeout(() => { + socket.destroy(); + reject(new Error(`Daemon request timed out after ${timeoutMs}ms: ${JSON.stringify(request.method ?? request.meta)}`)); + }, timeoutMs); + + let buffer = ""; + + socket.on("connect", () => { + socket.write(JSON.stringify(request) + "\n"); + }); + + socket.on("data", (chunk: Buffer) => { + buffer += chunk.toString(); + // Response is one JSON line ending with \n + if (buffer.includes("\n")) { + clearTimeout(timer); + const line = buffer.split("\n")[0]; + try { + const response: DaemonResponse = JSON.parse(line); + socket.end(); + resolve(response); + } catch (e) { + socket.end(); + reject(new Error(`Failed to parse daemon response: ${line.slice(0, 200)}`)); + } + } + }); + + socket.on("error", (err: NodeJS.ErrnoException) => { + clearTimeout(timer); + socket.destroy(); + reject(new Error(`Daemon socket error (${socketPath}): ${err.message}`)); + }); + + socket.on("close", () => { + clearTimeout(timer); + if (buffer.length === 0) { + reject(new Error(`Daemon socket closed without response (${socketPath})`)); + } + }); + }); +} + +/** + * Quick check if the daemon socket accepts connections. + */ +export function daemonAlive(socketPath: string, timeoutMs = 1000): Promise<boolean> { + return new Promise((resolve) => { + const socket = createConnection(socketPath); + const timer = setTimeout(() => { + socket.destroy(); + resolve(false); + }, timeoutMs); + + socket.on("connect", () => { + clearTimeout(timer); + socket.end(); + resolve(true); + }); + + socket.on("error", () => { + clearTimeout(timer); + resolve(false); + }); + }); +} + +/** + * True health check: send Target.getTargets and verify "result" key in response. + * A stale daemon can accept socket connections but have a dead CDP websocket. + */ +export async function healthProbe(socketPath: string): Promise<boolean> { + try { + const response = await sendRequest( + socketPath, + { method: "Target.getTargets", params: {} }, + 3000, + ); + return "result" in response && response.result !== undefined; + } catch { + return false; + } +}
A src/renderers.ts

@@ -0,0 +1,21 @@

+/** + * TUI renderers for pi-browser-harness. + * + * Registers custom message renderers and tool renderCall/renderResult + * for browser tools. + */ + +import type { ExtensionAPI } from "@mariozechner/pi-coding-agent"; +import { Text } from "@mariozechner/pi-tui"; + +export function registerRenderers(pi: ExtensionAPI): void { + // Register a message renderer for screenshot notifications + pi.registerMessageRenderer("browser-screenshot", (message, _options, theme) => { + const path = (message as { details?: { path?: string } }).details?.path || "unknown"; + return new Text( + theme.fg("accent", `πŸ“Έ Screenshot: ${path}`), + 0, + 0, + ); + }); +}
A src/setup.ts

@@ -0,0 +1,178 @@

+/** + * Setup command β€” guides the user through installing and connecting + * browser-harness to their Chrome browser. + */ + +import type { ExtensionAPI, ExtensionContext } from "@mariozechner/pi-coding-agent"; +import { execSync } from "node:child_process"; +import { existsSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import type { BrowserDaemon } from "./daemon"; + +export function registerSetupCommand(pi: ExtensionAPI, daemon: BrowserDaemon): void { + pi.registerCommand("browser-setup", { + description: "Install and connect browser-harness to Chrome", + handler: async (_args, ctx) => { + await runSetup(ctx, daemon); + }, + }); +} + +async function runSetup(ctx: ExtensionContext, daemon: BrowserDaemon): Promise<void> { + ctx.ui.notify("browser-harness setup: checking installation...", "info"); + + // Step 1: Check for browser-harness installation + const bhInstalled = await checkBrowserHarnessInstalled(); + if (!bhInstalled) { + ctx.ui.notify( + "browser-harness not found. Installing...", + "warning", + ); + const installed = await installBrowserHarness(ctx); + if (!installed) { + ctx.ui.notify( + "Installation failed. Please install manually:\n" + + " git clone https://github.com/browser-use/browser-harness ~/Developer/browser-harness\n" + + " cd ~/Developer/browser-harness && uv sync", + "error", + ); + return; + } + } + + ctx.ui.notify("browser-harness found βœ“", "info"); + + // Step 2: Check Chrome is running + const chromeRunning = checkChromeRunning(); + if (!chromeRunning) { + ctx.ui.notify( + "Chrome/Chromium/Edge not detected. Please start your browser and retry /browser-setup.", + "error", + ); + return; + } + ctx.ui.notify("Chrome is running βœ“", "info"); + + // Step 3: Try to start the daemon + ctx.ui.notify("Connecting to Chrome...", "info"); + try { + await daemon.start(); + ctx.ui.notify("Daemon connected βœ“", "info"); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + const lower = msg.toLowerCase(); + + if ( + lower.includes("devtoolsactiveport") || + lower.includes("remote debugging") + ) { + ctx.ui.notify( + "Chrome remote debugging needs to be enabled.\n\n" + + "Start Chrome with: --remote-debugging-port=9222\n" + + "Then run /browser-setup again.", + "warning", + ); + return; + } + + ctx.ui.notify(`Connection failed: ${msg}`, "error"); + return; + } + + // Step 4: Verify with test navigation + ctx.ui.notify("Testing browser control...", "info"); + try { + await daemon.newTab("https://github.com/browser-use/browser-harness"); + const info = await daemon.getPageInfo(); + if ("dialog" in info) { + await daemon.cdp("Page.handleJavaScriptDialog", { accept: true }); + } + ctx.ui.notify( + `Browser control verified βœ“\nNavigated to: ${"url" in info ? info.url : "github.com"}`, + "info", + ); + } catch (err) { + ctx.ui.notify( + `Browser connected but test navigation failed: ${err instanceof Error ? err.message : String(err)}`, + "warning", + ); + } +} + +// ── Helpers ────────────────────────────────────────────────────────────────── + +async function checkBrowserHarnessInstalled(): Promise<boolean> { + const searchPaths = [ + join(homedir(), "Developer", "browser-harness"), + join(homedir(), "src", "browser-harness"), + join(homedir(), "browser-harness"), + join(homedir(), "dev", "browser-harness"), + join(homedir(), "Projects", "browser-harness"), + process.cwd(), + ]; + + for (const dir of searchPaths) { + if ( + existsSync(join(dir, "daemon.py")) && + existsSync(join(dir, "helpers.py")) + ) { + return true; + } + } + + // Check if uv tool is installed globally + try { + execSync("uv tool list 2>/dev/null | grep browser-harness", { timeout: 5000 }); + return true; + } catch { + // not found via uv + } + + return false; +} + +async function installBrowserHarness(ctx: ExtensionContext): Promise<boolean> { + // Try uv tool install + try { + ctx.ui.notify("Installing browser-harness via uv...", "info"); + execSync("uv tool install browser-harness", { + timeout: 60_000, + stdio: "pipe", + }); + return true; + } catch { + // uv install failed + } + + // Try git clone + try { + const targetDir = join(homedir(), "Developer", "browser-harness"); + ctx.ui.notify(`Cloning browser-harness to ${targetDir}...`, "info"); + execSync( + `git clone https://github.com/browser-use/browser-harness "${targetDir}"`, + { timeout: 30_000, stdio: "pipe" }, + ); + execSync("uv sync", { cwd: targetDir, timeout: 60_000, stdio: "pipe" }); + return true; + } catch { + return false; + } +} + +function checkChromeRunning(): boolean { + try { + if (process.platform === "darwin" || process.platform === "linux") { + const out = execSync("ps -A -o comm=", { timeout: 5000 }).toString().toLowerCase(); + return ["google chrome", "chrome", "chromium", "microsoft edge", "msedge"].some( + (n) => out.includes(n), + ); + } else if (process.platform === "win32") { + const out = execSync("tasklist", { timeout: 5000 }).toString().toLowerCase(); + return ["chrome.exe", "msedge.exe"].some((n) => out.includes(n)); + } + } catch { + // best-effort + } + return false; +}
A src/state.ts

@@ -0,0 +1,77 @@

+/** + * Session persistence for pi-browser-harness. + * + * Persists daemon namespace, tab history, and preferences across + * session reloads and branch navigation. + */ + +import type { ExtensionAPI, ExtensionContext } from "@mariozechner/pi-coding-agent"; + +// ── Types ──────────────────────────────────────────────────────────────────── + +export interface BrowserState { + /** Daemon namespace (BU_NAME) for this session */ + namespace: string; + /** Recently active tab target IDs */ + tabHistory: string[]; + /** Cloud browser ID (for remote sessions) */ + remoteBrowserId?: string; + /** Screenshot output directory */ + screenshotDir: string; + /** Whether debug click overlay is enabled */ + debugClicks: boolean; +} + +// ── Defaults ───────────────────────────────────────────────────────────────── + +export function defaultState(namespace = "default"): BrowserState { + return { + namespace, + tabHistory: [], + screenshotDir: "", + debugClicks: false, + }; +} + +// ── Persist ────────────────────────────────────────────────────────────────── + +export function persistState(pi: ExtensionAPI, state: BrowserState): void { + pi.appendEntry<BrowserState>("browser-harness-state", state); +} + +// ── Restore ────────────────────────────────────────────────────────────────── + +/** + * Find the last browser-harness-state entry in the current branch + * and return the restored state, merged with defaults. + */ +export function restoreState(ctx: ExtensionContext, currentNamespace?: string): BrowserState { + const branchEntries = ctx.sessionManager.getBranch(); + const defaults = defaultState(currentNamespace); + + // Walk branch from newest to oldest to find last persisted state + for (let i = branchEntries.length - 1; i >= 0; i--) { + const entry = branchEntries[i]; + if (entry.type === "custom" && entry.customType === "browser-harness-state") { + const data = entry.data as Partial<BrowserState> | undefined; + if (data) { + // Merge persisted data onto defaults, but always keep the current + // namespace (which comes from the --browser-namespace flag or auto-gen). + return { ...defaults, ...data, namespace: currentNamespace || data.namespace || defaults.namespace }; + } + } + } + + return defaults; +} + +// ── Merge helpers ──────────────────────────────────────────────────────────── + +export function addToTabHistory(state: BrowserState, targetId: string): BrowserState { + const filtered = state.tabHistory.filter((id) => id !== targetId); + return { ...state, tabHistory: [targetId, ...filtered].slice(0, 20) }; +} + +export function setRemoteBrowser(state: BrowserState, browserId: string): BrowserState { + return { ...state, remoteBrowserId: browserId }; +}
A src/tools.ts

@@ -0,0 +1,1200 @@

+/** + * Tool registrations for pi-browser-harness. + * + * Maps browser-harness daemon methods to pi tools with TypeBox schemas, + * prompt snippets, guidelines, and TUI renderers. + */ + +import { + DEFAULT_MAX_BYTES, + DEFAULT_MAX_LINES, + formatSize, + truncateHead, + withFileMutationQueue, +} from "@mariozechner/pi-coding-agent"; +import type { ExtensionAPI } from "@mariozechner/pi-coding-agent"; +import { Text } from "@mariozechner/pi-tui"; +import { Type } from "typebox"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { BrowserDaemon } from "./daemon"; +import type { TabInfo } from "./protocol"; + +// ── Key code map ───────────────────────────────────────────────────────────── + +const SPECIAL_KEYS = [ + "Enter", "Tab", "Backspace", "Escape", "Delete", + "ArrowLeft", "ArrowUp", "ArrowRight", "ArrowDown", + "Home", "End", "PageUp", "PageDown", " ", +]; + +// ── Screenshot path ────────────────────────────────────────────────────────── + +let screenshotCounter = 0; +function nextScreenshotPath(): string { + screenshotCounter++; + return join(tmpdir(), `pi-browser-screenshot-${Date.now()}-${screenshotCounter}.png`); +} + +// ── Output truncation ──────────────────────────────────────────────────────── + +/** Track temp directories for cleanup on session shutdown. */ +const tempDirs: string[] = []; + +/** Best-effort cleanup of all tracked temp directories. */ +export async function cleanupTempDirs(): Promise<void> { + const dirs = tempDirs.splice(0); + await Promise.all( + dirs.map((d) => rm(d, { recursive: true, force: true }).catch(() => {})), + ); +} + +/** + * Truncate output to pi's built-in limits (50KB / 2000 lines). + * When truncated, writes the full output to a temp file so the LLM + * can read it via the read tool and so CTRL+O expand can show it. + */ +async function applyTruncation( + output: string, + prefix: string, +): Promise<{ + text: string; + fullOutputPath?: string; + wasTruncated: boolean; +}> { + const truncation = truncateHead(output, { + maxLines: DEFAULT_MAX_LINES, + maxBytes: DEFAULT_MAX_BYTES, + }); + + if (!truncation.truncated) { + return { text: truncation.content, wasTruncated: false }; + } + + const tempDir = await mkdtemp(join(tmpdir(), `pi-bh-${prefix}-`)); + tempDirs.push(tempDir); + + const tempFile = join(tempDir, "output.txt"); + await withFileMutationQueue(tempFile, async () => { + await writeFile(tempFile, output, "utf8"); + }); + + const omittedBytes = truncation.totalBytes - truncation.outputBytes; + let text = truncation.content; + text += `\n\n[Output truncated: ${truncation.outputLines} of ${truncation.totalLines} lines`; + text += ` (${formatSize(truncation.outputBytes)} of ${formatSize(truncation.totalBytes)}).`; + text += ` ${formatSize(omittedBytes)} omitted. Full output: ${tempFile}]`; + + return { text, fullOutputPath: tempFile, wasTruncated: true }; +} + +// ── Registration ───────────────────────────────────────────────────────────── + +export function registerTools(pi: ExtensionAPI, daemon: BrowserDaemon): void { + // ═══════════════════════════════════════════════════════════════════════════ + // browser_navigate + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_navigate", + label: "Browser Navigate", + description: + "Navigate to a URL. Creates a new tab on first call (safe), navigates current tab on subsequent calls.", + promptSnippet: "Navigate to a URL (creates new tab on first call)", + promptGuidelines: [ + "Use browser_navigate to go to URLs. The first call creates a new tab so it doesn't clobber the user's active tab.", + "Use browser_wait_for_load after browser_navigate to wait for the page to finish loading.", + "For extracting data from a page you already navigated to, use browser_execute_js or browser_http_get (faster for APIs).", + ], + parameters: Type.Object({ + url: Type.String({ description: "Full URL to navigate to (e.g. https://github.com)" }), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + + // Determine if we need a new tab: create one on first-ever call, + // or if the current tab is an internal Chrome page. Otherwise + // navigate the current tab in-place. + let targetId: string; + let isNewTab = false; + try { + const tabs = await daemon.listTabs(false); + if (tabs.length === 0) { + // No real tabs exist β€” create one + targetId = await daemon.newTab(params.url); + isNewTab = true; + } else { + // Navigate the current tab in-place + await daemon.cdp("Page.navigate", { url: params.url }); + const tab = await daemon.currentTab(); + targetId = tab.targetId; + } + } catch { + // Fallback: create a new tab + targetId = await daemon.newTab(params.url); + isNewTab = true; + } + + const info = await daemon.getPageInfo(); + if ("dialog" in info) { + return { + content: [{ type: "text" as const, text: `Navigated to ${params.url}\nTarget: ${targetId}\n⚠️ A dialog is open: ${info.dialog.type} β€” "${info.dialog.message}"\nUse browser_handle_dialog to dismiss or accept it.` }], + }; + } + + const prefix = isNewTab ? "New tab: " : ""; + return { + content: [{ + type: "text" as const, + text: `${prefix}Navigated to: ${info.url}\nTitle: ${info.title}\nViewport: ${info.width}x${info.height}\nTarget: ${targetId}`, + }], + }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Navigation failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + renderCall(args, theme) { + return new Text(`🌐 Navigate to ${(args as { url: string }).url}`, 0, 0); + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_page_info + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_page_info", + label: "Browser Page Info", + description: + "Get current page state: URL, title, viewport size, scroll position, page dimensions. If a JS dialog (alert/confirm/prompt) is open, returns dialog info instead.", + promptSnippet: "Get current page URL, title, viewport, and scroll position", + promptGuidelines: [ + "Use browser_page_info to quickly check what page you're on and whether a JS dialog is blocking interaction.", + "If browser_page_info returns a dialog, use browser_handle_dialog before any other browser actions.", + ], + parameters: Type.Object({}), + async execute() { + try { + await daemon.ensureAlive(); + const info = await daemon.getPageInfo(); + + if ("dialog" in info) { + return { + content: [{ + type: "text" as const, + text: `⚠️ DIALOG OPEN: ${info.dialog.type}\nMessage: ${info.dialog.message}${info.dialog.defaultPrompt ? `\nDefault: ${info.dialog.defaultPrompt}` : ""}\n\nUse browser_handle_dialog to accept or dismiss.`, + }], + }; + } + + return { + content: [{ + type: "text" as const, + text: `URL: ${info.url}\nTitle: ${info.title}\nViewport: ${info.width}x${info.height}\nScroll: (${info.scrollX}, ${info.scrollY})\nPage size: ${info.pageWidth}x${info.pageHeight}`, + }], + }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_go_back + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_go_back", + label: "Browser Go Back", + description: "Navigate back one page in history.", + promptSnippet: "Go back one page in browser history", + promptGuidelines: [ + "Use browser_go_back to return to the previous page after navigating.", + ], + parameters: Type.Object({}), + async execute() { + try { + await daemon.ensureAlive(); + const history = (await daemon.cdp("Page.getNavigationHistory")) as { + entries: Array<{ id: number; url: string; title: string }>; + currentIndex: number; + }; + if (history.currentIndex <= 0) { + return { content: [{ type: "text" as const, text: "Already at the beginning of history." }] }; + } + const targetEntry = history.entries[history.currentIndex - 1]; + await daemon.cdp("Page.navigateToHistoryEntry", { entryId: targetEntry.id }); + return { content: [{ type: "text" as const, text: `Navigated back to: ${targetEntry.url}` }] }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_go_forward + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_go_forward", + label: "Browser Go Forward", + description: "Navigate forward one page in history.", + promptSnippet: "Go forward one page in browser history", + promptGuidelines: [ + "Use browser_go_forward to go forward after navigating back.", + ], + parameters: Type.Object({}), + async execute() { + try { + await daemon.ensureAlive(); + const history = (await daemon.cdp("Page.getNavigationHistory")) as { + entries: Array<{ id: number; url: string; title: string }>; + currentIndex: number; + }; + if (history.currentIndex >= history.entries.length - 1) { + return { content: [{ type: "text" as const, text: "Already at the end of history." }] }; + } + const targetEntry = history.entries[history.currentIndex + 1]; + await daemon.cdp("Page.navigateToHistoryEntry", { entryId: targetEntry.id }); + return { content: [{ type: "text" as const, text: `Navigated forward to: ${targetEntry.url}` }] }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_reload + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_reload", + label: "Browser Reload", + description: "Reload the current page.", + promptSnippet: "Reload the current page", + promptGuidelines: [ + "Use browser_reload to refresh the current page, e.g. after making changes that should be reflected.", + ], + parameters: Type.Object({}), + async execute() { + try { + await daemon.ensureAlive(); + await daemon.cdp("Page.reload"); + const info = await daemon.getPageInfo(); + if ("dialog" in info) { + return { content: [{ type: "text" as const, text: `Reloaded. ⚠️ Dialog open: ${info.dialog.type}` }] }; + } + return { content: [{ type: "text" as const, text: `Reloaded: ${info.title} (${info.url})` }] }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_click + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_click", + label: "Browser Click", + description: + "Click at viewport CSS-pixel coordinates. Compositor-level click works through iframes, shadow DOM, and cross-origin content. Use browser_screenshot first to find coordinates.", + promptSnippet: "Click at pixel coordinates (x, y) on the page", + promptGuidelines: [ + "Use browser_click for all clicks. Coordinates are viewport CSS pixels (not device pixels).", + "Capture a browser_screenshot BEFORE clicking to find the right coordinates.", + "Capture another browser_screenshot AFTER clicking to verify the action worked.", + "Compositor-level clicks pass through iframes, shadow DOM, and cross-origin content β€” no selector needed.", + ], + parameters: Type.Object({ + x: Type.Number({ description: "X coordinate in CSS pixels from left edge of viewport" }), + y: Type.Number({ description: "Y coordinate in CSS pixels from top edge of viewport" }), + button: Type.Optional(Type.String({ description: 'Mouse button: "left", "right", or "middle". Default: "left"' })), + clicks: Type.Optional(Type.Number({ description: "Number of clicks (1 = single, 2 = double). Default: 1" })), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + await daemon.cdp("Input.dispatchMouseEvent", { + type: "mousePressed", + x: params.x, + y: params.y, + button: params.button || "left", + clickCount: params.clicks || 1, + }); + await daemon.cdp("Input.dispatchMouseEvent", { + type: "mouseReleased", + x: params.x, + y: params.y, + button: params.button || "left", + clickCount: params.clicks || 1, + }); + return { content: [{ type: "text" as const, text: `Clicked at (${params.x}, ${params.y})` }] }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Click failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + renderCall(args, theme) { + const { x, y } = args as { x: number; y: number }; + return new Text(`πŸ–±οΈ Click at (${x}, ${y})`, 0, 0); + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_type + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_type", + label: "Browser Type", + description: "Type text into the currently focused element. Use browser_click first to focus an input field.", + promptSnippet: "Type text into the focused element", + promptGuidelines: [ + "Use browser_type to enter text. Click on an input field with browser_click first to focus it.", + "For special keys (Enter, Tab, Escape, arrows), use browser_press_key instead.", + ], + parameters: Type.Object({ + text: Type.String({ description: "Text to type" }), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + await daemon.cdp("Input.insertText", { text: params.text }); + return { content: [{ type: "text" as const, text: `Typed: "${params.text}"` }] }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Type failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_press_key + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_press_key", + label: "Browser Press Key", + description: + "Press a keyboard key. Supports special keys (Enter, Tab, Backspace, Escape, Delete, arrows, Home, End, PageUp, PageDown, Space as ' ') and regular characters. Optional modifier bitfield: 1=Alt, 2=Ctrl, 4=Meta/Cmd, 8=Shift.", + promptSnippet: "Press a key (Enter, Tab, Escape, arrows, or any character)", + promptGuidelines: [ + "Use browser_press_key for keyboard shortcuts and navigation keys.", + "Special key names: Enter, Tab, Backspace, Escape, Delete, ArrowLeft, ArrowUp, ArrowRight, ArrowDown, Home, End, PageUp, PageDown.", + "Use Space as ' ' (a single space character).", + "Modifiers: 1=Alt, 2=Ctrl, 4=Meta(Cmd), 8=Shift. Combine with bitwise OR: Ctrl+Shift = 2|8 = 10.", + ], + parameters: Type.Object({ + key: Type.String({ description: 'Key to press. Special keys: Enter, Tab, Backspace, Escape, Delete, ArrowLeft/Up/Right/Down, Home, End, PageUp, PageDown. Space as " "' }), + modifiers: Type.Optional(Type.Number({ description: "Modifier bitfield: 1=Alt, 2=Ctrl, 4=Meta(Cmd), 8=Shift. Combine with OR." })), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + const k = params.key; + const mod = params.modifiers || 0; + const vk = daemon.getVirtualKeyCode(k); + const code = daemon.getKeyCode(k); + const isSpecial = SPECIAL_KEYS.includes(k); + const text = k.length === 1 && !isSpecial ? k : ""; + + // keyDown + const base = { + type: "keyDown" as const, + key: k, + code, + modifiers: mod, + windowsVirtualKeyCode: vk, + nativeVirtualKeyCode: vk, + ...(text ? { text } : {}), + }; + await daemon.cdp("Input.dispatchKeyEvent", base); + + // char event for printable single characters + if (text.length === 1) { + await daemon.cdp("Input.dispatchKeyEvent", { + type: "char", + text, + key: k, + code, + modifiers: mod, + windowsVirtualKeyCode: vk, + nativeVirtualKeyCode: vk, + }); + } + + // keyUp + await daemon.cdp("Input.dispatchKeyEvent", { + type: "keyUp", + key: k, + code, + modifiers: mod, + windowsVirtualKeyCode: vk, + nativeVirtualKeyCode: vk, + }); + + return { content: [{ type: "text" as const, text: `Pressed: "${k}"${mod ? ` (modifiers: ${mod})` : ""}` }] }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Key press failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_scroll + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_scroll", + label: "Browser Scroll", + description: + "Scroll the page at given coordinates (or center of viewport if not specified). Default scrolls down 300px.", + promptSnippet: "Scroll the page by deltaY/deltaX pixels at coordinates", + promptGuidelines: [ + "Use browser_scroll to scroll pages. Default deltaY=-300 scrolls down.", + "Positive deltaY scrolls up, negative scrolls down.", + "Capture browser_screenshot after scrolling to see the new viewport content.", + ], + parameters: Type.Object({ + deltaY: Type.Optional(Type.Number({ description: "Vertical scroll delta in pixels. Negative = down, positive = up. Default: -300" })), + deltaX: Type.Optional(Type.Number({ description: "Horizontal scroll delta in pixels. Default: 0" })), + x: Type.Optional(Type.Number({ description: "X position to scroll from. Default: viewport center" })), + y: Type.Optional(Type.Number({ description: "Y position to scroll from. Default: viewport center" })), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + const dy = params.deltaY ?? -300; + const dx = params.deltaX ?? 0; + + // Default to center of viewport if not specified + let cx = params.x; + let cy = params.y; + if (cx === undefined || cy === undefined) { + try { + const info = await daemon.getPageInfo(); + if (!("dialog" in info)) { + if (cx === undefined) cx = Math.round(info.width / 2); + if (cy === undefined) cy = Math.round(info.height / 2); + } + } catch { + cx = cx ?? 500; + cy = cy ?? 400; + } + } + + await daemon.cdp("Input.dispatchMouseEvent", { + type: "mouseWheel", + x: cx, + y: cy, + deltaX: dx, + deltaY: dy, + }); + + return { content: [{ type: "text" as const, text: `Scrolled by (${dx}, ${dy}) at (${cx}, ${cy})` }] }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Scroll failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_screenshot + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_screenshot", + label: "Browser Screenshot", + description: + "Capture a PNG screenshot of the current page. Use this BEFORE interactions to find targets, and AFTER to verify results. This is the primary visual feedback mechanism.", + promptSnippet: "Capture a screenshot of the current page (PNG)", + promptGuidelines: [ + "Use browser_screenshot as your FIRST step when exploring a page β€” it shows you what's visible.", + "After every browser_click, browser_type, browser_scroll, or browser_navigate, capture a screenshot to VERIFY the action worked.", + "Screenshots are the fastest way to find click targets, notice hidden blockers, and decide next steps.", + "Never assume an action worked. Always re-screenshot to confirm.", + ], + parameters: Type.Object({ + fullPage: Type.Optional(Type.Boolean({ description: "Capture the full scrollable page (not just viewport). Default: false" })), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + const path = nextScreenshotPath(); + await daemon.captureScreenshot(path, params.fullPage === true); + return { + content: [{ type: "text" as const, text: `Screenshot saved: ${path}${params.fullPage ? " (full page)" : ""}` }], + details: { path }, + }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Screenshot failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + renderCall(_, theme) { + return new Text(`πŸ“Έ Capturing screenshot...`, 0, 0); + }, + renderResult(result, _options, theme) { + const path = (result.details as { path?: string })?.path; + return new Text(path ? `πŸ“Έ Screenshot: ${path}` : `πŸ“Έ Screenshot captured`, 0, 0); + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_list_tabs + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_list_tabs", + label: "Browser List Tabs", + description: "List all open browser tabs.", + promptSnippet: "List all open browser tabs with URLs and titles", + promptGuidelines: [ + "Use browser_list_tabs to see what tabs are open before switching.", + "Use browser_switch_tab to switch to a specific tab by targetId.", + "Set includeChrome: true to also see Chrome internal pages (devtools, settings, etc).", + ], + parameters: Type.Object({ + includeChrome: Type.Optional(Type.Boolean({ description: "Include Chrome internal pages (chrome://, devtools://, etc). Default: true" })), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + const tabs = await daemon.listTabs(params.includeChrome !== false); + const listing = tabs + .map((t, i) => `[${i}] ${t.title || "(no title)"}\n URL: ${t.url}\n ID: ${t.targetId}`) + .join("\n\n"); + return { + content: [{ type: "text" as const, text: `${tabs.length} tab(s):\n\n${listing}` }], + details: { tabs }, + }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + renderResult(result, { expanded }, theme) { + const details = result.details as { + tabs?: TabInfo[]; + fullOutputPath?: string; + } | undefined; + + const tabCount = details?.tabs?.length ?? 0; + + if (expanded) { + // Expanded (CTRL+O): show full tab listing + const content = result.content[0]; + const raw = content?.type === "text" ? content.text : ""; + return new Text(raw || "(no tabs)", 0, 0); + } + + // Collapsed (default): compact count + return new Text( + theme ? theme.fg("dim", `πŸ“‹ ${tabCount} tab${tabCount !== 1 ? "s" : ""}`) : `πŸ“‹ ${tabCount} tab${tabCount !== 1 ? "s" : ""}`, + 0, + 0, + ); + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_current_tab + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_current_tab", + label: "Browser Current Tab", + description: "Get info about the currently active tab.", + promptSnippet: "Get the current active tab's URL, title, and ID", + promptGuidelines: [ + "Use browser_current_tab to confirm which tab you're controlling.", + ], + parameters: Type.Object({}), + async execute() { + try { + await daemon.ensureAlive(); + const tab = await daemon.currentTab(); + return { + content: [{ type: "text" as const, text: `Current tab: ${tab.title || "(no title)"}\nURL: ${tab.url}\nID: ${tab.targetId}` }], + }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_switch_tab + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_switch_tab", + label: "Browser Switch Tab", + description: "Switch to a different browser tab by targetId (from browser_list_tabs).", + promptSnippet: "Switch to a browser tab by its targetId", + promptGuidelines: [ + "Use browser_list_tabs first to find the targetId, then browser_switch_tab to switch.", + ], + parameters: Type.Object({ + targetId: Type.String({ description: "Target ID of the tab to switch to (from browser_list_tabs)" }), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + await daemon.switchTab(params.targetId); + const tab = await daemon.currentTab(); + return { + content: [{ type: "text" as const, text: `Switched to: ${tab.title || "(no title)"}\nURL: ${tab.url}` }], + }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Switch failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_new_tab + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_new_tab", + label: "Browser New Tab", + description: "Open a new browser tab and switch to it. Optionally navigate to a URL.", + promptSnippet: "Open a new browser tab, optionally navigate to a URL", + promptGuidelines: [ + "Use browser_new_tab for the first navigation in a task β€” it preserves the user's active tab.", + "Use browser_navigate for subsequent navigation in the same tab.", + ], + parameters: Type.Object({ + url: Type.Optional(Type.String({ description: "URL to navigate to in the new tab. Default: about:blank" })), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + const targetId = await daemon.newTab(params.url); + const msg = params.url + ? `New tab created and navigated to: ${params.url}\nTarget: ${targetId}` + : `New tab created: about:blank\nTarget: ${targetId}`; + return { content: [{ type: "text" as const, text: msg }] }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_execute_js + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_execute_js", + label: "Browser Execute JS", + description: + 'Execute JavaScript in the current page and return the result. Use for data extraction, DOM inspection, and page manipulation. Expressions with "return" are automatically wrapped in an IIFE.', + promptSnippet: "Execute JavaScript in the page and return the result", + promptGuidelines: [ + "Use browser_execute_js for extracting structured data from the page (text, attributes, JSON from embedded scripts).", + "Use browser_execute_js for DOM inspection when coordinates from screenshots aren't sufficient.", + "For bulk data extraction, prefer browser_http_get for API calls β€” it's 10-50x faster than browser DOM scraping.", + ], + parameters: Type.Object({ + expression: Type.String({ description: "JavaScript expression to evaluate. 'return' statements are auto-wrapped in IIFE." }), + targetId: Type.Optional(Type.String({ description: "Target ID of an iframe to evaluate in (optional)" })), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + const value = await daemon.evaluateJS(params.expression, params.targetId); + const raw = value === undefined ? "undefined" : JSON.stringify(value, null, 2); + const { text, fullOutputPath } = await applyTruncation(raw, "js"); + return { + content: [{ type: "text" as const, text }], + details: { valueLength: raw.length, fullOutputPath }, + }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `JS execution failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + renderResult(result, { expanded }, theme) { + const details = result.details as { + valueLength?: number; + fullOutputPath?: string; + } | undefined; + + const len = details?.valueLength ?? 0; + + if (expanded) { + // Expanded (CTRL+O): show the full JS result + const content = result.content[0]; + const raw = content?.type === "text" ? content.text : ""; + return new Text(raw || "undefined", 0, 0); + } + + // Collapsed (default): size + first-line preview + const content = result.content[0]; + const raw = content?.type === "text" ? content.text : ""; + const firstLine = raw.split("\n")[0] || ""; + const preview = + firstLine.length > 120 ? firstLine.slice(0, 117) + "..." : firstLine; + + const summary = `πŸ“œ JS result (${formatSize(len)})`; + return new Text( + theme + ? theme.fg("dim", summary) + "\n" + theme.fg("muted", preview) + : summary + "\n" + preview, + 0, + 0, + ); + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_http_get + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_http_get", + label: "Browser HTTP GET", + description: + "Make a direct HTTP GET request (no browser). Use for APIs and static pages β€” much faster than browser navigation for data retrieval.", + promptSnippet: "Make a direct HTTP GET request (outside browser, for APIs)", + promptGuidelines: [ + "Use browser_http_get for API calls and static page retrieval β€” it bypasses the browser entirely and is 10-50x faster.", + "Use browser_http_get with ThreadPoolExecutor for bulk data retrieval from multiple URLs.", + "Do NOT use browser_http_get for pages that require JavaScript rendering β€” use browser_navigate + browser_execute_js instead.", + ], + parameters: Type.Object({ + url: Type.String({ description: "URL to fetch" }), + headers: Type.Optional(Type.Record(Type.String(), Type.String(), { description: "Optional HTTP headers" })), + timeout: Type.Optional(Type.Number({ description: "Timeout in seconds. Default: 20" })), + }), + async execute(_id, params) { + try { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), (params.timeout || 20) * 1000); + + const response = await fetch(params.url, { + headers: { + "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36", + Accept: "text/html,application/json,*/*", + ...params.headers, + }, + signal: controller.signal, + }); + + clearTimeout(timer); + + const body = await response.text(); + const contentType = response.headers.get("content-type") || "unknown"; + + // Build header (always shown) and truncate body separately + const headerText = `HTTP ${response.status} ${response.statusText}\nContent-Type: ${contentType}\nLength: ${body.length} chars\n\n`; + const { text: truncatedBody, fullOutputPath, wasTruncated } = await applyTruncation( + body, + "http", + ); + + return { + content: [{ + type: "text" as const, + text: headerText + truncatedBody, + }], + details: { + status: response.status, + contentType, + length: body.length, + fullOutputPath, + wasTruncated, + }, + }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `HTTP request failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + renderResult(result, { expanded }, theme) { + const details = result.details as { + status?: number; + contentType?: string; + length?: number; + wasTruncated?: boolean; + fullOutputPath?: string; + } | undefined; + + const status = details?.status ?? 0; + const ct = details?.contentType ?? "unknown"; + const len = details?.length ?? 0; + const wasTruncated = details?.wasTruncated ?? false; + + if (expanded) { + // Expanded (CTRL+O): show full response body + const content = result.content[0]; + const raw = content?.type === "text" ? content.text : ""; + return new Text(raw || "(empty response)", 0, 0); + } + + // Collapsed (default): compact HTTP response summary + const shortCt = ct.split(";")[0] || ct; // drop charset etc. + const truncNote = wasTruncated ? " (truncated)" : ""; + const summary = `🌐 HTTP ${status}, ${shortCt}, ${formatSize(len)}${truncNote}`; + + return new Text( + theme + ? theme.fg(status >= 200 && status < 300 ? "success" : status >= 400 ? "error" : "warning", summary) + : summary, + 0, + 0, + ); + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_wait + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_wait", + label: "Browser Wait", + description: "Wait for a specified number of seconds.", + promptSnippet: "Wait for N seconds (default 1s)", + promptGuidelines: [ + "Use browser_wait for simple delays. Prefer browser_wait_for_load after navigation.", + ], + parameters: Type.Object({ + seconds: Type.Optional(Type.Number({ description: "Seconds to wait. Default: 1" })), + }), + async execute(_id, params) { + const sec = params.seconds || 1; + await new Promise((r) => setTimeout(r, sec * 1000)); + return { content: [{ type: "text" as const, text: `Waited ${sec}s.` }] }; + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_wait_for_load + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_wait_for_load", + label: "Browser Wait For Load", + description: + "Wait for the page to finish loading (document.readyState === 'complete'). Returns false on timeout. Note: SPAs may need additional waiting after this.", + promptSnippet: "Wait for document.readyState === 'complete'", + promptGuidelines: [ + "Use browser_wait_for_load after browser_navigate or browser_new_tab.", + "This checks document.readyState only. Single-page apps may still be loading data after this returns true.", + "For SPAs, use browser_execute_js to check for specific elements after browser_wait_for_load.", + ], + parameters: Type.Object({ + timeout: Type.Optional(Type.Number({ description: "Max seconds to wait. Default: 15" })), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + const timeout = params.timeout || 15; + const deadline = Date.now() + timeout * 1000; + + while (Date.now() < deadline) { + const state = await daemon.evaluateJS("document.readyState"); + if (state === "complete") { + return { content: [{ type: "text" as const, text: "Page loaded (readyState: complete)." }] }; + } + await new Promise((r) => setTimeout(r, 300)); + } + + return { + isError: true, + content: [{ type: "text" as const, text: `Page load timed out after ${timeout}s. Current state unknown.` }], + }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Wait failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_handle_dialog + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_handle_dialog", + label: "Browser Handle Dialog", + description: + "Handle a JavaScript dialog (alert, confirm, prompt, beforeunload). Use when browser_page_info reports an open dialog.", + promptSnippet: "Accept or dismiss a JavaScript dialog (alert/confirm/prompt)", + promptGuidelines: [ + "Check browser_page_info first β€” if it returns a dialog, use browser_handle_dialog before any other browser actions.", + "JS dialogs freeze the page's JS thread, so no other interaction works until the dialog is handled.", + 'Use action: "accept" for alerts and confirms, "dismiss" to cancel. For prompts, provide promptText.', + ], + parameters: Type.Object({ + action: Type.String({ description: '"accept" or "dismiss"' }), + promptText: Type.Optional(Type.String({ description: 'Text to enter for prompt dialogs (only used with "accept")' })), + }), + async execute(_id, params) { + try { + await daemon.ensureAlive(); + + if (params.action === "accept") { + await daemon.cdp("Page.handleJavaScriptDialog", { + accept: true, + ...(params.promptText !== undefined ? { promptText: params.promptText } : {}), + }); + } else { + await daemon.cdp("Page.handleJavaScriptDialog", { accept: false }); + } + + return { content: [{ type: "text" as const, text: `Dialog ${params.action}ed.` }] }; + } catch (err) { + return { + isError: true, + content: [{ type: "text" as const, text: `Dialog handling failed: ${err instanceof Error ? err.message : String(err)}` }], + }; + } + }, + }); + + // ═══════════════════════════════════════════════════════════════════════════ + // browser_open_urls + // ═══════════════════════════════════════════════════════════════════════════ + pi.registerTool({ + name: "browser_open_urls", + label: "Browser Open URLs", + description: + "Open multiple URLs in new browser tabs in parallel. Useful after web_search to open citation links for visual inspection. Creates real Chrome tabs with full JS rendering support β€” identical to browser_new_tab but for many URLs at once.", + promptSnippet: "Open multiple URLs in new tabs (parallel, full JS rendering)", + promptGuidelines: [ + "Use browser_open_urls to open multiple URLs in parallel β€” much faster than calling browser_new_tab repeatedly.", + "Use browser_open_urls after web_search to open citation links from the synthesized answer.", + "Each URL opens in a real Chrome tab with full JS rendering β€” same as browser_new_tab.", + "After opening, use browser_list_tabs to see all tabs, browser_switch_tab to switch, and browser_screenshot / browser_execute_js to interact.", + "Use browser_wait_for_load on a tab before extracting data from SPAs.", + ], + parameters: Type.Object({ + urls: Type.Array(Type.String(), { description: "Array of URLs to open in new tabs" }), + }), + async execute(_id, params, _signal, onUpdate) { + await daemon.ensureAlive(); + + const total = params.urls.length; + const results: Array<{ + url: string; + targetId: string; + ok: boolean; + error?: string; + }> = []; + + // ═══════════════════════════════════════════════════════════════════ + // Phase 1: Create all tabs in parallel. + // Target.createTarget is a browser-level CDP call β€” no session + // needed, no shared state mutated. Safe to parallelize. + // ═══════════════════════════════════════════════════════════════════ + const createPromises = params.urls.map(async (url: string) => { + try { + const result = (await daemon.cdp("Target.createTarget", { + url: "about:blank", + })) as { targetId: string }; + return { url, targetId: result.targetId, ok: true as const }; + } catch (err) { + return { + url, + targetId: "", + ok: false as const, + error: err instanceof Error ? err.message : String(err), + }; + } + }); + + const created = await Promise.all(createPromises); + + // ═══════════════════════════════════════════════════════════════════ + // Phase 2: Attach a CDP session to each tab and navigate. + // Each tab gets its own sessionId passed explicitly to cdp(), + // avoiding the shared daemon._sessionId. Safe to parallelize. + // ═══════════════════════════════════════════════════════════════════ + let completed = 0; + const attachAndNav = created + .filter((t) => t.ok) + .map(async (tab) => { + try { + // Attach to target to get a session for this tab + const attachResult = (await daemon.cdp( + "Target.attachToTarget", + { targetId: tab.targetId, flatten: true }, + )) as { sessionId: string }; + const sid = attachResult.sessionId; + + // Enable the Page domain so navigation events fire + await daemon.cdp("Page.enable", {}, sid); + + // Navigate to the URL using the tab's own session + await daemon.cdp("Page.navigate", { url: tab.url }, sid); + + return { ...tab, ok: true as const, targetId: tab.targetId }; + } catch (err) { + return { + ...tab, + ok: false as const, + error: err instanceof Error ? err.message : String(err), + }; + } finally { + completed++; + // Stream progress so the TUI shows live per-URL status + const okCount = created.filter((t) => t.ok).length; + onUpdate?.({ + content: [ + { + type: "text" as const, + text: `Opening URLs… ${completed}/${okCount} navigated`, + }, + ], + }); + } + }); + + // Also collect failures from Phase 1 + const failures = created.filter((t) => !t.ok); + + const settled = await Promise.all(attachAndNav); + results.push(...settled, ...failures); + + // ═══════════════════════════════════════════════════════════════════ + // Phase 3: Activate the last successfully opened tab so the user + // sees something meaningful in the foreground. + // ═══════════════════════════════════════════════════════════════════ + const lastOk = [...settled].reverse().find((r) => r.ok); + if (lastOk) { + try { + await daemon.cdp("Target.activateTarget", { + targetId: lastOk.targetId, + }); + } catch { + // best-effort β€” the tabs are still open + } + } + + // Build result text + const okTabs = results.filter((r) => r.ok); + const failedTabs = results.filter((r) => !r.ok); + + const lines: string[] = []; + if (okTabs.length > 0) { + lines.push( + `βœ… ${okTabs.length}/${total} tabs opened:`, + ...okTabs.map( + (t, i) => ` [${i}] ${t.url} β†’ ${t.targetId}`, + ), + ); + } + if (failedTabs.length > 0) { + lines.push( + `❌ ${failedTabs.length}/${total} failed:`, + ...failedTabs.map( + (t) => ` ${t.url}: ${t.error || "unknown error"}`, + ), + ); + } + + const isTotalFailure = okTabs.length === 0; + const fullOutput = lines.join("\n"); + const { text, fullOutputPath } = await applyTruncation(fullOutput, "urls"); + + return { + isError: isTotalFailure, + content: [{ type: "text" as const, text }], + details: { tabs: results, fullOutputPath }, + }; + }, + renderCall(args, _theme) { + const { urls } = args as { urls: string[] }; + return new Text( + `🌐 Opening ${urls.length} URL${urls.length !== 1 ? "s" : ""}…`, + 0, + 0, + ); + }, + renderResult(result, { expanded }, theme) { + const details = result.details as { + tabs?: Array<{ + url: string; + targetId: string; + ok: boolean; + error?: string; + }>; + fullOutputPath?: string; + } | undefined; + + const tabs = details?.tabs || []; + const ok = tabs.filter((t) => t.ok).length; + const fail = tabs.filter((t) => !t.ok).length; + const total = tabs.length; + + if (expanded) { + // Expanded (CTRL+O): show the full URLβ†’targetId mapping + const content = result.content[0]; + const raw = content?.type === "text" ? content.text : ""; + // Strip the truncation footer for cleaner display, show it separately + const truncIdx = raw.indexOf("\n\n[Output truncated:"); + const displayText = truncIdx >= 0 ? raw.slice(0, truncIdx) : raw; + + let expandedText = displayText || "(no output)"; + if (details?.fullOutputPath && truncIdx < 0) { + // Truncated but we stripped the footer β€” add a compact note + expandedText += `\n\n${theme ? theme.fg("dim", `Full output: ${details.fullOutputPath}`) : `Full output: ${details.fullOutputPath}`}`; + } else if (details?.fullOutputPath) { + expandedText += `\n${theme ? theme.fg("dim", `Full output: ${details.fullOutputPath}`) : `Full output: ${details.fullOutputPath}`}`; + } + return new Text(expandedText, 0, 0); + } + + // Collapsed (default): compact summary + let text: string; + if (total === 0) { + text = "No URLs provided."; + } else if (fail === 0) { + text = `βœ… ${ok} tab${ok !== 1 ? "s" : ""} opened`; + } else if (ok === 0) { + text = `❌ All ${fail} URL${fail !== 1 ? "s" : ""} failed`; + } else { + text = `⚠️ ${ok} opened, ${fail} failed`; + } + + return new Text( + theme ? theme.fg(fail === 0 ? "success" : ok === 0 ? "error" : "warning", text) : text, + 0, + 0, + ); + }, + }); +}
A tsconfig.json

@@ -0,0 +1,18 @@

+{ + "compilerOptions": { + "target": "ES2022", + "module": "ESNext", + "moduleResolution": "bundler", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "outDir": "./dist", + "rootDir": "./src", + "declaration": true, + "declarationMap": true, + "sourceMap": true + }, + "include": ["src/**/*.ts"], + "exclude": ["node_modules", "dist"] +}