docs/COMPARISON.md (view raw)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 |
# pi-chrome vs. the rest of the browser-automation landscape
This is the honest "which tool when" page. The browser-automation space has three different layers and people often compare across them — let's not do that.
We benchmark in public — see [`../test-suite/`](../test-suite). Where exact scores matter (WebVoyager, WorkArena++, BrowseComp, Mind2Web 2), check the live leaderboards; they shift monthly.
---
## TL;DR
| You are… | Use… |
| -------------------------------------------------------------- | ------------------------------- |
| A Pi agent operator who wants the agent to use **your real Chrome** (logged-in tabs, cookies, extensions) | **pi-chrome** |
| Building a Pi/LLM agent and want low-level browser primitives | **pi-chrome** |
| Writing deterministic end-to-end tests in CI | Playwright / Cypress |
| Building a hosted scraping/agent fleet on isolated profiles | Playwright / Puppeteer + Browserbase / Steel |
| Want a turnkey "natural-language agent" with built-in loop | Browser Use / Stagehand / Skyvern |
| Want a hosted, vendor-managed agent | OpenAI Operator / Project Mariner / Surfer |
| Debugging your own app from your editor without leaving your real session | **pi-chrome** |
`pi-chrome` is **primitives**, not an opinionated agent loop. Think of it as **"Playwright for the Chrome you're already signed into"** — and pluggable under any agent framework above the line.
---
## The three axes you should compare on
| Axis | Examples | What it gives you |
| --------------------------------- | --------------------------------------------------------------------- | ----------------------------- |
| **1. Driver / transport** | Playwright, Puppeteer, Selenium, CDP raw, `puppeteer-stealth`, **pi-chrome** | low-level tools (click, type, navigate) |
| **2. Agent framework** | Browser Use, Stagehand, Skyvern, Magnitude, Alumnium, LangGraph-with-Playwright, Operator, Mariner, Surfer | LLM loop, planning, NL API |
| **3. Cloud browser provider** | Browserbase, Steel.dev, Hyperbrowser, Anchor, Browserless | managed Chromes, sessions, quotas |
`pi-chrome` is **axis 1**. You can wrap it with any axis-2 agent framework, or run it directly from a Pi agent's `chrome_*` tool calls. It explicitly does NOT compete with axis 3 — it runs locally inside *your* Chrome.
---
## Axis 1 — drivers (where pi-chrome lives)
| Tool | Transport | Profile | Browser input | Banner when controlling | Default detectable as bot |
| --------------------------------- | ------------------------------------------ | ---------------------------------- | -------------------- | ----------------------------------- | ------------------------- |
| Playwright | CDP (own driver) | throwaway by default | always | always ("controlled by test software") | yes (webdriver flag, automation flags) |
| Puppeteer | CDP | throwaway by default | always | always | yes |
| Selenium | WebDriver / BiDi | throwaway | partial (BiDi improves) | always | most detectable |
| puppeteer-stealth / playwright-extra | CDP + patches | throwaway | always | always | medium (patches flags) |
| Raw CDP | direct devtools protocol | either (needs `--remote-debugging-port`) | always | always | yes |
| **pi-chrome** | **Chrome extension bridge → local loopback** | **your real Chrome profile, signed-in cookies, extensions, history** | **always for input tools** | **while Chrome input is attached** | **low (real profile + Chrome input)¹** |
¹ pi-chrome uses `chrome.debugger` for browser input and shows Chrome's banner like other CDP-based tools. The [`test-suite/`](../test-suite) grades browser-control behavior against common detection signals.
### What makes pi-chrome different on this axis
1. **Profile attach, not driver launch.** Every other driver fights cookie persistence, login walls, MFA, and extension state. pi-chrome inherits all of it because it *is* your Chrome.
2. **Chrome input against your real profile.** Interactive tools use CDP input for reliability while still controlling the Chrome profile you already use.
3. **Extension bridge transport.** No `--remote-debugging-port`, no throwaway Chromium. Survives Chrome auto-updates. Works alongside your normal Chrome usage.
4. **Structured action results.** Input tools return target coordinates/tags and can include a fresh snapshot (`includeSnapshot`) so agents can verify state instead of blindly retrying.
5. **Multi-session shared bridge.** Planner + worker + audit Pi sessions all drive the same Chrome concurrently.
6. **Stable element uids.** `chrome_snapshot` returns deterministic uids you can pass to subsequent actions — similar to BrowserGym's `bid`, but built into the snapshot tool itself.
---
## Axis 2 — agent frameworks (built on top of axis 1)
These wrap a driver with an LLM loop. They are **higher-level than pi-chrome** and **complementary**, not competitors.
| Framework | Driver underneath | Approach | Open source |
| ------------------------ | ------------------------------ | --------------------------------------------------------------------------------------------- | --------------- |
| **Browser Use** | Playwright | DOM + a11y tree → LLM → action JSON. Open-source leader; widely cited on WebVoyager. | MIT (Python) |
| **Stagehand** (Browserbase) | Playwright | Natural-language `.act()` / `.observe()` / `.extract()`; deterministic + AI mix. | MIT (TypeScript)|
| **Skyvern** | Playwright + own DOM model | Vision-first + DOM; YAML workflows for form/workflow automation. | AGPL (Python) |
| **Magnitude** | Playwright | NL test authoring; QA-focused. | open |
| **Alumnium** | Selenium / Playwright | Test-author NL → agent. QA-focused. | open |
| **LangGraph / AutoGen + Playwright** | Playwright | Generic agent graph + browser tools. | open |
| **OpenAI Operator** | proprietary | OpenAI's own VLM + browser; ChatGPT-integrated. | closed, hosted |
| **Project Mariner** (Google) | proprietary Chrome integration | Google's own VLM Chrome experiment. | closed |
| **Surfer 2 / Surfer-H** (H Company) | proprietary | Hosted proprietary agent stack. | closed, hosted |
| **Anthropic Computer Use** | OS-level screenshots + mouse/keyboard | Broader than browser; OS-level events. | closed (API) |
**Why pi-chrome is not on this list:** it's intentionally **not an agent**. There's no LLM loop, no `.act("click the blue button")`. Pi handles the loop; pi-chrome provides the primitives. This means:
- You can use pi-chrome under Browser Use, Stagehand, LangGraph, or any other agent framework (see [Interop](#interop)).
- You don't pay for an opinion you don't want.
- Agent improvements compound across all your workflows because the primitives stay stable.
---
## Axis 3 — cloud browser providers (orthogonal)
| Provider | What it sells |
| ----------------- | -------------------------------------------------------- |
| **Browserbase** | Managed browsers; pairs with Stagehand. |
| **Steel.dev** | Managed browsers + public agent leaderboards. |
| **Hyperbrowser** | Managed browsers, session APIs. |
| **Anchor Browser**| Managed browsers. |
| **Browserless** | Managed Chrome, scraping focus. |
**pi-chrome doesn't compete here.** It runs locally in *your* Chrome. The right framing if someone asks: *"no cloud cost, no session-handoff, no rate limits — and the agent runs against your real logged-in state."* If you need fleets of isolated Chromes in CI, you want one of these.
---
## Interop
`pi-chrome` exposes tools that any Pi agent can call. If you want to use it from outside Pi:
1. The local bridge speaks HTTP JSON over `127.0.0.1:17318` (default). The API is internal; use the Pi tool surface unless you are building an adapter.
2. Tool surface mirrors Playwright closely (click/type/navigate/snapshot/screenshot/evaluate/wait_for) so adapter code is short.
3. `includeSnapshot` on input tools lets agent harnesses verify state after actions.
If you want a first-class pi-chrome adapter for Browser Use / Stagehand / LangGraph, file an issue with your use case.
---
## "But Playwright has `storageState` / Puppeteer has user-data-dir"
Yes — you can export cookies and replay them, or point Playwright at your existing profile directory. In practice for agent workflows that breaks down fast:
1. **OAuth + SSO** providers (Okta, Google, GitHub) often pin sessions to TLS fingerprints, device IDs, and browser-extension state that doesn't survive replay or a parallel Chrome instance.
2. **MFA** tokens expire mid-run.
3. **Internal admin tools** hard-pin to your real device.
4. **Pointing Playwright at your real `user-data-dir`** requires closing your normal Chrome (Chrome won't share the profile lock). pi-chrome doesn't fight you for the profile because it lives *inside* it.
5. **Watching the agent work** in your real window is a different UX than a hidden parallel Chrome. Demos, pair-driving, and confidence-building all want axis-1-with-attach.
---
## "Is this safer than CDP?"
Different security boundary, not strictly safer.
- **CDP-based tools** require `chrome --remote-debugging-port=...`. That port is unauthenticated and exposes the whole browser to any local process. Easy to misconfigure.
- **pi-chrome** runs through an extension you install yourself with broad permissions (tabs, scripting, debugger, webNavigation). The bridge listens on `127.0.0.1:17318` loopback only, rejects browser-origin command requests, and keeps chrome_* tools locked until `/chrome authorize` is run in the current Pi session. **Only install the bundled extension if you trust the source you got the npm package from.**
If your threat model excludes extensions with broad permissions, neither approach is a fit — you want a sandboxed CI runner.
---
## Public benchmarks worth knowing (for axis 2 / axis 3 comparison)
Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **43 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering trusted input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, iframes, file uploads, strict-CSP screenshot fallback and CDP eval/snapshot bypass, dynamic waits, tab lifecycle, network observability, fingerprint leaks, and agent-safety honeypots. Scoring tracks expected outcomes per challenge instead of raw PASS count, with `core`, `conditional`, and `quality` gate buckets. That's **driver-level** grading.
For **agent-level** comparison (axis 2), the public benchmarks worth citing:
| Benchmark | What it measures | Notes |
| ---------------- | ------------------------------------------------- | ------------------------------------------------ |
| **WebArena** (CMU) | Hermetic, programmatic graders | Gold standard for reproducibility. |
| **WorkArena++** (ServiceNow) | Enterprise SaaS workflows | Hardest realistic benchmark; <5% frontier. |
| **BrowseComp** (OpenAI) | Hard info-retrieval | Not saturated. |
| **Mind2Web 2** (NeurIPS '25) | Long-horizon, rubric-tree judge | New, well-designed. |
| **WebChoreArena** | Tedious cross-page workflows | Reflects real ops work. |
| **WebVoyager** | Live web tasks | **Saturated** — 90%+ scores common; cite only as smoke test. |
| **VisualWebArena** | Multimodal | |
| **MiniWoB++** | Classic unit-task suite | |
| **BrowserGym + AgentLab** | Research harness covering the above | The de-facto research API; pi-chrome's snapshot uid is comparable to BrowserGym's `bid`. |
Cite live leaderboards rather than hard-coded numbers; agent scores shift monthly.
---
## Reproducing pi-chrome's driver-level claims
Run [`../test-suite/`](../test-suite) against any browser-control tool. Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events`, so any tool (Playwright, Puppeteer, Selenium, Stagehand, pi-chrome) can grade itself deterministically. Headline release scoring should use the `core` gate; `conditional` depends on declared environment capabilities, and `quality` tracks adversarial/humanization regressions.
```bash
cd test-suite && python3 -m http.server 8765
# open http://127.0.0.1:8765/ in the Chrome window the tool controls
```
If you build a competing tool, please open a PR with your scores.
|