test-suite/notes/browsergym-compat.md (view raw)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
# BrowserGym compatibility contract
Pi-chrome benchmark tasks mirror BrowserGym without importing Gymnasium.
## Task fields
Each `task-manifest.json` entry maps to `AbstractBrowserTask` concepts:
- `taskId` / `id` — stable namespaced id
- `seed` — deterministic state seed
- `viewport`, `slowMoMs`, `timeoutMs`, `locale`, `timezoneId`
- `goal` and `goalObject` — text and OpenAI-style message objects
- `setupUrl` — page URL that performs setup from `task`, `run`, `seed`
- `validateScript` — JS validate hook returning reward contract
- `cheatScript` — gold recipe used to sanity-check grader
- `maxSteps`, `humanBaseline`, `difficulty`, `tags`
- `actionSubsets`, `allowedActions` — BrowserGym HighLevelActionSet metadata
## Reset / step return
Runner should expose:
```text
reset() -> (obs, info)
step(action) -> (obs, reward, terminated, truncated, info)
```
`terminated` means task done. `truncated` means max step or timeout.
## Validate return
In-page validators return:
```js
{ reward, done, message, info }
```
`reward` is incremental. Current hermetic tasks use binary terminal reward. `info.rubric` carries per-check details.
## Observation schema target
Runner observations should include BrowserGym-like keys:
- `chat_messages`
- `goal`, `goal_object`
- `open_pages_urls`, `open_pages_titles`, `active_page_index`
- `url`, `screenshot`, `dom_object`, `axtree_object`
- `extra_element_properties`
- `focused_element_bid`
- `last_action`, `last_action_error`
- `elapsed_time`
## BID plan
BrowserGym uses `bid` attributes as stable element handles. Pi-chrome currently returns snapshot `uid`. Compatibility path:
1. During snapshot, assign every interactive element a deterministic `bid`.
2. Return both `uid` and `bid`.
3. Add `visibility`, `bbox`, `clickable`, `set_of_marks` metadata.
4. Maintain uid↔bid mapping so agents can use either addressing mode.
Target attributes:
- `bid`
- `browsergym_visibility_ratio`
- `browsergym_set_of_marks`
## Action subsets
See `../browsergym-action-space.json` for BrowserGym-compatible subsets and function signatures.
|