Tool reference: read-only tools

Read-only tools

URL redaction is default-on. Every surface that returns captured page traffic (ActionResult.network, network_read, ws_read, and URL substrings inside console_read / page-error text) is routed through one centralized sanitizer at the egress boundary: query strings, fragments, user:pass@ userinfo, and token/identity-shaped path segments are stripped (a present-but-stripped query/fragment shows as ?… / #…), while scheme + host + path-pattern + method + status + timing + response-shape are preserved. This is a posture, not an opt-in. browxai output is meant to be shareable and the server is heading public. The raw request/response body remains separately gated behind the off-by-default network-body capability. Internal filtering (beacon detection, ws_read url-substring filter) still operates on the un-redacted value; only what leaves toward an MCP result is sanitized. See Threat model.

snapshot

Compact accessibility-tree snapshot of the current page, augmented by a DOM-walk pass that surfaces interactive elements and any element bearing one of the configured BROWX_TEST_ATTRIBUTES (default data-testid,data-test,data-cy,data-qa). The DOM walk runs every snapshot, which is what makes browxai work on heavy-SPA targets whose accessibility tree is sparse / non-semantic. Nodes only seen by the DOM walk are marked [from-dom]; nodes found by both tiers in the same snapshot are [from-both] and appear once, on the accessibility tier’s line, with the accessibility tier’s ARIA role, accessible name and ref. The two tiers are matched on the CDP backend node id, which is an exact identity, two elements that merely look alike (same role, same accessible name, same test attribute) are never folded together. An element the accessibility tier does not expose, or whose path the DOM sweep cannot resolve exactly (anything inside a shadow root), keeps its own [from-dom] line.

Each interactive node gets a stable [ref=eN] you can pass back to action tools. Refs persist across snapshots within a session (a node that’s still there keeps its eN). Token-efficient: generic / presentational nodes are pruned; states (disabled, checked=…, focused, value=…, [<test-attr>=…]) are inlined. Test-attribute hints emit the actual attribute name that matched (e.g. [data-type="feature-panel-language-input"]) so you can transcribe the selector directly.

When the a11y tree has fewer than 5 interactive descendants under root, a warning is emitted, usually meaning the page is a heavy SPA and the DOM-walk source carried the load.

When the accessibility call fails or returns no nodes, the snapshot has no accessibility tier at all. A warning then names the CDP error (accessibility tree extraction failed (…)) or says Accessibility.getFullAXTree returned no nodes, and stats.a11yInteractive reads 0 whatever the page holds. The low-content warning gives the node count of the tree it read.

stats.tier names which tier supplied the interactive content: a11y, dom-walk, mixed, or empty. dom-walk is the degraded case. The accessibility tier found nothing and the DOM-walk fallback answered on its own. mixed means the DOM walk contributed elements the accessibility tier does not have. Non-Chromium engines and child-frame snapshots report dom-walk by construction: neither runs a CDP accessibility pass.

stats.domWalkCombined counts the DOM-walk entries the accessibility tier already had (the [from-both] nodes) and stats.domWalkNew the ones it did not ([from-dom]); the two sum to stats.domWalkEntries. Both describe the snapshot in hand, not the session’s history.

Nodes CDP marks ignored (presentational wrappers, layout tables, and <html> / <body>, which Chromium marks ignored on essentially every page) contribute no line of their own, and their children take their place in document order. An aria-hidden container’s descendants are themselves ignored, so nothing under it reaches the a11y tier. The DOM walk still reports it, [from-dom]-marked.

Chromium’s text and layout leaves get no line either: StaticText, LineBreak, ListMarker, LayoutTable / LayoutTableRow / LayoutTableCell, Abbr, EmphasizedText, StrongText, Ruby / RubyAnnotation, superscript and subscript. Each carries text its enclosing control, heading, cell or paragraph already names, so emitting both printed the page twice, two thirds of the serialised body across six real pages. One of these nodes bearing a configured test attribute is still emitted, and find will still rank it.

Container text. A paragraph, list item, blockquote, layout table cell or unnamed generic has no accessible name, so Chromium puts its words on a StaticText child. The row carries those words in the name position: paragraph "Shipping takes 3 days." [ref=e12]. Text from a child that has its own row (a link, a button) stays on that row. The words left on the container are trimmed, so See <a>docs</a> gives paragraph "See" [ref=e2] above link "docs" [ref=e3]. Nothing under a button, link, heading, cell or form field carries text, so a password field’s value never lands on a container row. Each row keeps at most 120 characters, cut with …, and one snapshot carries at most 4000 characters in total. Past that, rows come out bare and a last line reads ... [text left off N row(s) past the 4000 character text budget; use text_search or extract to read it]. text_search and extract read the full text.

Inputs (all optional):

• scope: <ref> emits only the subtree rooted at this ref (from a prior snapshot/find). Drops “I asked for one section and got 500 nodes” cost. Falls back to full tree with a warning if the ref isn’t found.
• maxNodes: <N> is a hard cap on emitted nodes; excess is elided with a +N more nodes elided marker pointing the agent at scope or a higher cap.
• omit: ["<pattern>",...] takes case-insensitive substring patterns, matched against each node’s role / name / testId. Matching nodes and their entire subtrees are skipped. Useful for noisy regions: omit: ["timeline-segment-", "clip-thumbnail"].

Output (text): url: / title: / stats: header + (optional) scope: / warnings: block + indented role "name" [ref=eN] [<test-attr>=…] [from-dom|from-both] [state] lines + (when relevant) ... [+N more nodes elided] or ... [omit matched N subtree(s), M nodes total] or ... [text left off N row(s) ...].

Example. Scope to one panel instead of dumping the whole tree:

snapshot({ scope: "e12", maxNodes: 150, omit: ["clip-thumbnail"] })

url: https://app.example.com/records
title: Records
scope: e12
region "Records" [ref=e12]
  table "Q2 records" [ref=e15]
    row "Wed, May 13  Engineering  Reviewed" [ref=e16]
      button "Edit" [ref=e17] [data-testid="row-edit"] [from-both]

find

Find candidate elements by natural-language description and return them ranked, each with a stable ref. Use it to locate one element to act on; use snapshot to see the whole page and text_search to check that text is present or gone.

Inputs: { query: string, maxCandidates?: number (default 5, max 20), confidenceFloor?: number, contextRef?: string, visibleOnly?: boolean }

• visibleOnly: default false. When true, non-actionable candidates (off-screen / clipped / covered / disabled) are dropped entirely and never ranked last. find returns an empty candidates list plus the “no visible candidate” warning. A confident hidden hit otherwise lures agents into coordinate fallbacks despite the warning; an empty result is the safer signal (“the target isn’t actionable yet: wait/renavigate, don’t chase coordinates”).
• Attached/BYOB bbox reliability: the CDP visible-rect path can spuriously null out a rendered DOM-walk node on an attached Chrome (no live backend node, cross-frame quirks), which would wrongly classify it off-screen (and make visibleOnly:true drop a correct hit). find now falls back to Playwright’s own locator bounding box before classifying. A node that is genuinely on the page keeps a real bbox / actionable:true. So visibleOnly is dependable in attached mode as well as managed/incognito.
• confidenceFloor: emit a warnings: ["no candidate scored confidently above N (top score: …)"] block when no top candidate exceeds this score. Default 0 (off). Pass e.g. 0.5 (or any chosen integer) to get a “fall through to snapshot” signal instead of grinding through low-quality results.
• contextRef: limit ranking to descendants of this ref. Lets you say “the X under Y” without encoding the relationship in the natural-language query. Ignored (with a warning) if the ref isn’t in the current snapshot.
• Not candidates: the text and layout leaves snapshot suppresses (StaticText, LineBreak, ListMarker, LayoutTable*, Abbr, EmphasizedText, StrongText, Ruby*, superscript, subscript) and the document root (RootWebArea / WebArea). None is addressable, role=StaticText[name="button"] is not a locator the engine resolves, and ranking one costs a slot the real control then does not get, because the list is cut to maxCandidates before the candidates are probed. A node carrying a configured test attribute is ranked whatever its role.

Output: JSON

{
  "query": "the Save button",
  "candidates": [
    {
      "ref": "e42",
      "role": "button",
      "name": "Save",
      "testId": "save-btn",
      "stability": "high", // high = data-testid; medium = role+name; low = fallback
      "selectorHint": "[data-testid=\"save-btn\"]",
      "selectorTier": 1, // 1..5 preference order
      "bbox": { "x": 12, "y": 200, "width": 80, "height": 30 }, // visible-rect
      "clipped": false, // true → bbox: null (element fully off-screen / clipped)
      "score": 17,
      "context": {
        // structural neighbourhood when this candidate
        "collection": "table", //         lives in a repeated container. Omitted otherwise.
        "rowKey": "Wed, May 13",
        "column": "Type",
        "rowText": "Wed, May 13 Engineering Reviewed PR …",
      },
    },
  ],
}

selectorHint preference order: [<test-attr>="…"], then role=<role>[name="…"], then stable text on stable role, then structural (id/semantic), then positional (last resort). Tier-1 fires on any configured BROWX_TEST_ATTRIBUTES value and does not gate on a role wrapper. A <div data-type="x"> on a heavy SPA gets stability: "high" directly. The emitted selector preserves the matched attribute name. stability: "low" still means the agent should refuse to transcribe into a flow-file and ask a human or push for a test attribute on the app team. A hint that resolves to nothing against the live page falls to the candidate’s positional CSS path (tier 5, stability: "low"): the DOM-walk fallback reports an element’s bare tag in role, so a named <a> on table-shaped markup builds role=a[name="…"], which Playwright’s role engine rejects. The positional path is brittle across deploys but it resolves, so the bbox / actionable probes report what is actually on screen instead of clipped: true for every candidate.

Stability semantics: stability: "high" means “uniquely identifies this element in this snapshot”, i.e. the locator works right now. It does not mean “survives content rotation across deploys.” An asset card with [data-testid="asset-container-12345678"] (a content-keyed numeric suffix) is "high" for this snapshot but rotates with content. For a flow-file that needs to survive day-to-day rotation, prefer a structural/name selector or compose: [data-testid^="asset-container-"]:has-text("…"). The current stability field is honest about per-snapshot uniqueness; “deploy stability” is the agent’s call to make on top of it.

What find() matches against: the query is tokenised on whitespace and matched (case-insensitive substring) against each candidate’s accessible name + role + test-attribute value (whichever attribute matched per BROWX_TEST_ATTRIBUTES) + the candidate’s trimmed text content (a weaker signal that picks up a title tooltip or sr-only label when it surfaced into the node’s text). It does not match raw HTML attribute names, icon glyphs, placeholder=, or off-screen ancestors’ text. For truly icon-only controls, the testid/data-attr value is still the strongest query target.

Phrase ranking. The strongest name signal is an accessible name that equals a contiguous run of query tokens, not one that equals the whole query. The documented input is a natural-language description, so the target’s name sits inside a sentence. find("the past link in the top navigation bar") scores a link named exactly past on that run (+7, plus +3 per content word in it), which is why a multi-word name match ranks above a one-word one. Closed-class function words (the, of, in, to, for, …) score nothing on their own, in the run weighting or in the per-token pass: without that, a long headline sharing the and top with the query out-ranks the element the query actually named.

Name-less / icon-only ranking. For controls with no accessible name, per-test-attribute-token weight is amplified, the trimmed text signal is added, and a control already in a selected / pressed / checked state that also matches the query gets a bonus, so the live feature-panel tab outranks its inert icon-only siblings and unrelated top-nav tabs. The state bonus only ever lifts an existing match; it never fabricates a hit from nothing.

Disambiguation: when the bare selectorHint matches multiple DOM nodes (e.g. a visible button + a hidden DOM sibling sharing the same data-type), the emitted hint is auto-promoted to [<attr>="…"]:visible (or :nth-match(..., 1) last-resort) so mechanical transcription into a flow file doesn’t re-introduce a hidden-duplicate boundingBox hang.

Actionable predicate: each candidate carries actionable: true | "disabled" | "off-screen" | "covered" alongside stability / bbox. Lets a calibration agent reject <input disabled>-shaped halts at write-time instead of run-time. "covered" is reserved for a future check; today the value is true / "disabled" / "off-screen".

Visibility-aware ranking: after scoring, candidates are stable-partitioned so actionable: true ones rank ahead of non-visible (off-screen / clipped / covered / disabled) ones. A slightly-lower-scored visible match outranks a high-scored hidden modal. When there are matches but none are actionable, find() emits a warnings entry (no visible candidate — all N match(es) are off-screen/clipped/covered; usually means the wrong element matched). The recovery it names is re-query or snapshot: the warning’s own premise is that the wrong element matched, and a coordinate derived from a wrong element’s rect turns a miss into a confident click on whatever occupies that point. Coordinates (point_probe to identify what is painted at a point, then coords) are named as the last resort for canvas / painted UI with no DOM element to address. The tool names are capability-aware: coords only when the action capability is enabled, eval_js only when eval is enabled. The tool list never names a disabled tool.

Container demotion. Within the actionable tier there is a second stable partition: non-interactive structural / layout / landmark wrappers (generic, group, region, toolbar, navigation, main, form, …; the things that enclose a control, never the control itself) are demoted below interactive matches, but only when at least one actionable interactive candidate matched. So an aliased / product-facing query (“the X panel in the right tool rail”) returns the button/tab, not its enclosing wrapper. If nothing interactive matched, containers stay put (they may be the best available target). Role-driven and generic, with no query-string heuristics; list / listitem / article / section are deliberately not treated as containers since they can legitimately be the target.

Semantically thin pages. On markup whose accessibility tree is near-empty (table-based layouts, heavy SPAs of unlabelled divs), candidates come from the DOM-walk fallback and their role is the element’s own tag: a, nav, div. Both the interactive bonus and the container demotion resolve that tag to its implicit ARIA role before testing it, so an <a href> counts as a link and a <nav> is demoted like a navigation landmark. An <a> with no href and an <input> are read from their href / type attributes. The reported role stays the tag it was. ref identity is derived from it, so refs handed out by an earlier find or snapshot keep working.

confidenceFloor: pass confidenceFloor: <N> and find() emits a warnings: ["no candidate scored confidently above N (top score: …)"] entry when nothing crosses the bar, giving the agent a clean “fall through to snapshot” signal instead of grinding through a list of low-quality candidates.

bbox semantics: getBoundingClientRect() ∩ each overflow: hidden|clip ancestor ∩ viewport. bbox: null + clipped: true when fully clipped. overflow: auto / scroll are scrollable, not clipping, and do not shrink the rect.

Structural context: candidates that live inside a recognised repeated layout (semantic table/grid row, list listitem, feed article) carry a context: { collection, rowKey, column?, rowText } field. Lets the caller answer “what row/column is this candidate in?” without re-walking the snapshot. column is populated only when the collection has a header row with columnheader cells and the candidate’s index aligns to a header. rowKey is the first non-empty visible text within the row, capped at 80 chars. rowText is the row’s concatenated visible text, capped at 200 chars. Detection is generic, driven by ARIA roles, not by app-specific markers. Nodes outside a repeated layout simply omit context.

frames_list

List every frame in the current page tree with a stable per-session ID (fN; f0 is always the main frame). Pass the returned frameId back as frame: <fN> to snapshot / find to scope observation to a child iframe; refs minted in that frame are bound to it on the registry so subsequent actions (click, fill, etc.) land inside the iframe transparently. Same-origin and cross-origin (OOPIF) iframes both work through Playwright’s frame API.

Inputs: { session? }

Output: JSON

{
  "ok": true,
  "frames": [
    {
      "frameId": "f0",
      "url": "http://example.test/with-iframe",
      "name": "",
      "isMainFrame": true,
      "origin": "http://example.test",
    },
    {
      "frameId": "f1",
      "parentFrameId": "f0",
      "url": "http://example.test/child",
      "name": "same",
      "isMainFrame": false,
      "origin": "http://example.test",
    },
    {
      "frameId": "f2",
      "parentFrameId": "f0",
      "url": "about:srcdoc",
      "name": "data",
      "isMainFrame": false,
      "origin": "",
    },
  ],
  "tokensEstimate": 312,
}

Frame ID stability: within a session, the main frame is always f0. Child frames mint f1, f2, … in first-seen order; identical-fingerprint frames across repeat frames_list calls keep their ID. Intra-iframe navigation (the same <iframe> handle changing URL) preserves the ID. Refs minted while a child frame was attached survive across frames_list calls; if the iframe detaches, calls into the frame return a structured “unknown frame” error and do not throw.

Frame-scoped snapshot / find: both tools accept an optional frame: <fN>. When set:

• snapshot({frame}) returns a tree scoped to that frame. The CDP accessibility-tree path is not run for child frames (rooted at the top target, doesn’t reach into OOPIFs); the snapshot is DOM-walk-sourced only. This is surfaced as a warnings: entry on the result so the agent isn’t surprised by the [from-dom] markers.
• find({frame, query, …}) ranks candidates inside that frame and binds the returned refs to it; passing the ref to click / fill / hover / etc. fires inside the iframe, with no separate action surface needed.

Cross-origin caveats:

• Read works: Playwright’s frame.locator(…) and frame.evaluate(…) span the OOPIF boundary.
• Actions work: frame.locator(…).click() (etc.) cross the same boundary.
• The CDP accessibility-tree skip on child frames means a heavily a11y-driven page in an iframe surfaces less context than the same page would as a top-level document. The DOM-walk pass still surfaces every BROWX_TEST_ATTRIBUTES-bearing element and every interactive control, which is what action targeting needs.
• Frame-scoped bbox is computed via Playwright’s locator .boundingBox(), where main-frame finds use the CDP getBoxModel path; behaviour is identical for visible elements.

Shadow DOM piercing

Modern web components default to shadow DOM; find / snapshot see open shadow content through Playwright’s a11y tree automatically. Two opt-in extensions plus a dedicated read-only tool add direct introspection.

find({ …, pierce? }) and snapshot({ …, includeShadow? }). Both accept a pierce (find) / includeShadow (snapshot) parameter:

• omitted: back-compat. Playwright’s a11y tree already auto-pierces open shadow roots; the DOM-walk fallback does not recurse into shadow content. Earlier, callers see byte-identical output.
• "open": additionally have the DOM-walk recurse through every reachable open shadow root (Element.shadowRoot for each host). Useful on heavy-SPA targets whose a11y tree is sparse and whose interactive controls live behind web-component boundaries.
• "closed": open-walk plus a CDP DOM.getDocument({pierce:true}) pass that harvests interactive / test-attr-bearing elements behind closed shadow boundaries. Closed-shadow candidates carry [from-dom] source marks like any other DOM-walk entry; the result envelope additionally surfaces a warning that closed-shadow elements cannot be actioned through Playwright’s locator engine. Treat them as evidence (“this widget exists at depth N”), not actionable targets.
• false: disables shadow recursion entirely.

Closed-shadow piercing is best-effort by construction. DOM.getDocument({pierce:true}) is a Chromium DevTools facility, not a web-platform guarantee. On older Chromium builds or attached-mode endpoints whose CDP vintage differs from the launcher’s, the call may fail; the result envelope then carries closed-shadow piercing unavailable on this browser/page in warnings[] and falls back to the open-only view. Open shadow is always reachable.

shadow_trees({ ref?, maxHosts?, session? }). Dedicated read-only introspection. Returns:

{
  "trees": [
    {
      "hostRef": "backend:1234", // or "backend:0" when the page-side fallback ran
      "hostTag": "my-widget",
      "mode": "open", // or "closed"
      "children": [
        { "tag": "div", "text": "Hello", "childCount": 2 },
        { "tag": "button", "childCount": 0 },
      ],
      "descendantCount": 12,
    },
  ],
  "closedShadowAvailable": true,
  "warnings": [],
  "tokensEstimate": 142,
}

ref (optional) limits the walk to one host’s subtree (the ref comes from a prior snapshot / find); omit it to walk every shadow root in the document. maxHosts (default 200, max 1000) caps the result with a cappedAt field when hit.

closedShadowAvailable is true when the CDP pierce call returned at least one closed-mode root anywhere in the walked subtree (proves the CDP path is live on this browser); false is informational: the page may simply not contain a closed root, or CDP refused the call.

Capability read (same posture as snapshot / find; no new capability gate).

screenshot

PNG or JPEG of the viewport, optionally cropped to an element, optionally full-page, optionally written to a workspace-rooted file instead of returned inline.

Format / size knobs:

• format: "png" | "jpeg": default "png" (lossless, larger). "jpeg" is dramatically smaller for screenshots dense with content; pairs with quality.
• quality: 0-100: JPEG only; default 80. Ignored for PNG. Lower = smaller payload, more compression artefacts.
• scale: "css" | "device": default "device" (Hi-DPI native resolution). "css" renders at CSS-pixel dimensions; a 2x display drops to ~1/4 the byte size at the cost of detail.

For multimodal agents filling a constrained context window, format: "jpeg", quality: 70, scale: "css" often cuts payload size by ~5 to 10× with minimal impact on a vision model’s ability to read the page. Not OCR-on-the-server: the agent’s own vision capability does the work; F7 just lets the caller tune what it ingests.

Scope / output knobs:

• fullPage: boolean: default false. When true, captures the whole document (Playwright’s page.screenshot({fullPage:true})) and not only the viewport. Mutually exclusive with ref / selector / named, since element-scoped captures are already bounded by the element’s box; combining them returns a structured rejection.
• path: string: workspace-rooted file path. When set, writes the bytes to disk and the result swaps the inline image content part for a JSON envelope { ok, path, bytes, format, fullPage, caption?, tokensEstimate }. Path-traversal is rejected (must resolve under $BROWX_WORKSPACE, the same chokepoint as pdf_save / start_har / dump_storage_state). Parent directories are auto-created. Requires the file-io capability (in addition to the tool’s own read gate); a request with path set against a server without file-io returns a structured requiredCapability: "file-io" rejection. Default mode (no path) is unchanged and needs no extra capability.

Inputs: { ref?, selector?, named?, describe?: boolean, fullPage?: boolean, path?: string } (pass at most one of ref/selector/named; none = viewport unless fullPage:true)

• describe: emit a structured one-line caption alongside the PNG (role "name" [<attr>="…"] bbox=x,y w×h [not-visible|disabled]). Lets the agent skip vision-reading when it just needs to confirm presence. When path is set, the caption rides on the JSON envelope as caption.

Output:

• Default (no path): an MCP image content part (base64 PNG/JPEG), optionally preceded by a text part with the caption. Byte-identical to the v0.3.x shape when path is omitted.
• With path: a JSON envelope { ok, path, bytes, format, fullPage, caption?, tokensEstimate }, with no inline image bytes.

Example.

// Token-cheap evidence for a vision read: ~5–10× smaller than the PNG default.
screenshot({ format: "jpeg", quality: 70, scale: "css" })
// → MCP image content part (base64 JPEG)

// Big full-page capture to disk instead of into context (needs `file-io`).
screenshot({ fullPage: true, path: "shots/checkout.png" })
// → { "ok": true, "path": "/…/.browxai/shots/checkout.png", "bytes": 412380,
//     "format": "png", "fullPage": true, "tokensEstimate": 64 }

screenshot_schedule

Periodic screenshot capture at a fixed interval into a workspace-rooted directory. The “show me what happened across the next N seconds without me babysitting” primitive. Pair it with a long-running interaction or a wait for an async settle.

Inputs: { everyMs, count? | durationMs?, intoDir?, format? }

• everyMs: interval between captures, range [100, 60000] ms.
• Exactly one of:
  • count: integer 1..1000: stop after N captures.
  • durationMs: integer > 0: stop after this wall-clock window. Must be >= everyMs.
• intoDir: workspace-rooted output directory. Defaults to screenshots/<sessionId>-<isoTs>/. A path outside the workspace is rejected.
• format: "png" (default) or "jpeg". JPEGs are written with .jpg.

Files are named <seq>-<offsetMs>.<png|jpg> so the dir-listing alone reproduces the timeline. A belt-and-braces ceiling of 1000 captures per call applies on top of the count/duration bound, surfaced as a warnings[] entry if hit. A single failed snap is logged as a warning and the schedule continues (does not poison the window). The outer action-timeout wraps the whole call: an unbounded screenshot_schedule is refused at validation time, so the deadline is “expected window + slack”.

Output: { ok, intoDir, count, capturedAt: [offsetMs…], paths: […], warnings: […], tokensEstimate }. Paths are absolute, capturedAt is offset-from-start in ms.

Example.

screenshot_schedule({ everyMs: 500, durationMs: 5000, format: "jpeg" })
// → { "ok": true, "intoDir": "/…/screenshots/default-2026-06-12T10-31-04Z",
//     "count": 10, "capturedAt": [0, 500, 1000, …], "paths": ["…/1-0.jpg", …] }

Capability: file-io (same posture as screenshot({path}) / page_archive).

screenshot_on

Event-driven screenshot capture. Arms a trigger for durationMs; every time the trigger fires inside the window, a screenshot is written to a workspace-rooted directory. The “catch the visual state every time X happens” primitive, for after-the-fact debugging of intermittent behaviour where the failure mode is hard to scope to a single action.

Trigger surface (fixed enum):

• navigation: main-frame framenavigated (subframe navigations are noise).
• console-error: page console events with type==="error" OR pageerror.
• network-mutation: write-shaped (POST/PUT/PATCH/DELETE) responses with a 2xx status, same heuristic the ActionResult.network.mutations probe uses.
• dialog: alert / confirm / prompt / beforeunload.

Inputs: { trigger, durationMs, intoDir?, format? }

• trigger: one of the four above.
• durationMs: observation window length, range [1, 600000] ms (10 min ceiling).
• intoDir: workspace-rooted output directory. Defaults to screenshots/<sessionId>-<isoTs>/. A path outside the workspace is rejected.
• format: "png" (default) or "jpeg".

A per-window cap of 50 captures prevents event-storm runaway (e.g. a console-error fired every animation frame), surfaced as a warnings[] entry if reached, and the window closes early. Trigger fires that land while a previous capture is still in flight are dropped (single screenshot per visible state is the useful unit). A snap that errors is logged as a warning; the window keeps observing. The outer action-timeout is at least the observation window plus 1s of slack so the call can run a multi-minute window without aborting.

Output: { ok, intoDir, trigger, capturedAt: [offsetMs…], paths: […], warnings: […], tokensEstimate }.

Example.

screenshot_on({ trigger: "console-error", durationMs: 30000 })
// → { "ok": true, "trigger": "console-error", "intoDir": "/…/screenshots/…",
//     "capturedAt": [1204, 9817], "paths": ["…/1-1204.png", "…/2-9817.png"], "warnings": [] }

Capability: file-io.

text_search

Find nodes whose visible text matches a query. Read-only, distinct from find(): find() ranks actionable targets; text_search verifies presence/absence (“is the bad value gone?”, “did ‘Saved’ appear?”, “no Wrong Type chip in the record grid”).

Args:

• text: string to match.
• exact (default false): when false, case-insensitive substring. When true, case-sensitive equality on the trimmed node name.
• scope: limit the search to descendants of this ref (a prior snapshot/find result).
• includeHidden (default false): only visible (bbox-having) matches are returned by default.
• maxMatches: default 20; hard cap 200.

Returns { count, matches: [{ ref, role, text, context, bbox, clipped }] }. Each match carries the structural context when it lives in a repeated container, so a caller can ask “any Wrong Type left in the record grid?” and get back row-tagged results without re-walking the tree.

count: 0 is the clean absence signal. No more overloading find() for presence/absence.

Example.

text_search({ text: "Wrong Type", scope: "e15" })
// → { "count": 0, "matches": [] }                       // clean absence — the bad value is gone

text_search({ text: "Saved" })
// → { "count": 1, "matches": [{ "ref": "e88", "role": "status", "text": "Saved",
//      "bbox": { "x": 904, "y": 64, "width": 52, "height": 20 }, "clipped": false }] }

extract

Structured, schema-driven data extraction: the primitive every browxai adopter currently rebuilds on top of snapshot(). The schema is the contract: partial matches surface in evidence.partialMisses (or failure.partialMisses when required:true), never silently coerced into a malformed object.

Deterministic, selector-only. Each schema property lowers to a find()-style query or explicit selector scoped to the current subtree. No model-call in the substrate, per the model-agnostic principle.

The mode parameter is RETIRED as of v0.3.2. The deterministic mode is the only supported path, and the typed SDK no longer exposes the field. Setting mode: "llm-assisted" is tolerated (treated as deterministic) for back-compat but will emit a one-shot console.warn at the call site. Drop the mode arg from new code.

Args:

• schema: a JSON-schema-flavoured shape (object/array/string/number/boolean; properties for objects, items for arrays). See the lowering rules below.
• ref: scope to this ref’s subtree (from a prior snapshot/find).
• scope: scope to this CSS selector’s first match. Invalid (zero matches) Returns structured failure, not an empty object. Mutually exclusive with ref.
• mode: RETIRED. Tolerated for back-compat (warn + treated as deterministic). Drop the arg.

Returns {ok:true, data:<schema-shaped>, evidence:{refsUsed,selectorsUsed,partialMisses}, warnings?, tokensEstimate}, or {ok:false, failure:{source,kind,expected,actual,partialMisses?}, tokensEstimate} for misses. evidence.refsUsed lets the agent name_ref / cache the elements the extraction actually drew from. warnings appears when a selector or collection in the schema matched no element, one line per selector naming the first field that used it (selector ".price" matched no element (field price)), up to 10. An ok:true result with empty data and this list means a selector needs fixing.

Lowering rules

Two paths, deliberately layered:

1. Implicit (the simple rule): the property name is the query. A {type:"string"} property "price" looks for a node whose accessible name / testid contains "price" and reads its visible text. This is the path most testid-rich pages take.

2. Explicit (the escape hatch): add x-browx-source per property to override. The fields (first-present wins in source-resolution order):

   • selector: raw CSS / selectorHint, resolved against the current scope. This is the typed escape hatch for per-field targeting.
   • attr: read this HTML attribute ("href", "data-state").
   • prop: read this DOM property ("value", "checked").
   • text: explicit “read visible text” (the default when no read-mode hint is set).
   • value: alias for prop:"value".

The per-field query key is RETIRED as of v0.3.3. The NL tree-scan ranker is unreliable for explicit prose queries (uniform null/0 across rows with no partialMiss surfaced; see CHANGELOG v0.3.3). Use selector for per-field targeting; the implicit property-name lowering still works on testid-rich pages. Setting x-browx-source.query at runtime is tolerated for back-compat: the resolver emits a one-shot console.warn and records a partialMisses entry naming the field, then proceeds with the tree-scan. New schemas should drop the query key.

Set BROWX_EXTRACT_STRICT=1 at server start to turn unknown x-browx-source keys into ok:false invalid-schema failures instead of partialMisses entries. The integer-to-number coercion and the array selector alias stay soft in strict mode.

The implicit rule covers the headline case (testid-friendly pages) without ceremony; the explicit hint covers the cases where the property name carries no signal or the value isn’t innerText.

Per-property modifiers

• required: true: a miss surfaces in failure.partialMisses and fails the extraction. Optional misses (default) only emit evidence.partialMisses.
• default: fallback value applied when an optional miss occurs. The miss is still recorded in evidence.partialMisses.

Lists (type:"array")

{type:"array", items:<schema>, "x-browx-source":{collection:"<selectorOrQuery>"}} finds the container elements and re-runs the inner schema scoped to each. The collection is tried first as a CSS selector; if zero matches, falls back to a tree-scan against the query.

Arrays without an x-browx-source.collection are rejected as a partial miss. There is no defensible implicit default, and an empty list would lie about ground truth.

Examples

Simple object (implicit rule):

{
  "type": "object",
  "properties": {
    "title": { "type": "string" },
    "price": { "type": "number" }
  }
}

List with per-row sub-schema (explicit collection + mixed implicit/explicit fields):

{
  "type": "object",
  "properties": {
    "rows": {
      "type": "array",
      "x-browx-source": { "collection": "tr.product-row" },
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string", "x-browx-source": { "selector": ".name" } },
          "price": { "type": "number", "x-browx-source": { "selector": ".price" } },
          "href": { "type": "string", "x-browx-source": { "selector": "a", "attr": "href" } }
        }
      }
    }
  }
}

verify_visible / verify_text / verify_value / verify_count / verify_attribute / verify_predicate

Assertive read primitives. wait_for is permissive: it returns when satisfied OR when its deadline expires with ok:false as a normal outcome. The verify_* family is the fail-emitting sibling: each tool returns {ok: true} when the assertion holds right now, or {ok: false, failure: {source, kind, expected, actual, evidence?}, tokensEstimate} when it doesn’t, so an agent loop terminates deterministically instead of relying on the LLM eyeballing a snapshot.

Failure shape carries the standard {source} classifier from failure.ts:

• source: "app": the predicate didn’t hold against the page (a real signal the agent should act on).
• source: "browxai": verify itself couldn’t run (ref no longer in the snapshot, malformed input, etc.; the agent should re-snapshot, not file a defect).

All six are read-only (capability read). Coords targets are rejected: verify is structural, and the rare canvas / dismiss-empty-space case stays on click + screenshot.

A third outcome: refusal. The five element-bound verifies need to resolve a ref or selector to one element. On an engine that declares no element sub-interface (safari), they return the engine-refusal envelope, {ok: false, error, engine, hint, tokensEstimate} with no failure key, instead of an assertion result. Key on the presence of failure to tell the two apart: a refusal means the check never ran and says nothing about the page, so it must never be recorded as a defect. verify_predicate evaluates a caller-supplied data bag and runs on every engine.

The refusal envelope is the engine-wide contract, not a verify_* special case. Every tool whose implementation needs a capability the session’s engine does not declare returns the same {ok: false, error, engine, hint, tokensEstimate} shape, with no result-shaped keys at all. Today that covers the network reads (network_read / ws_read / network_body) and the live-emulation setters (set_geolocation / set_color_scheme / set_reduced_motion) on safari, which declares neither network nor emulation; the storage, script and capture families carry the same gate for engines that omit those. export_session_report follows the rule too, on an engine with no network observation it omits the network summary and carries networkUnavailable instead of a zeroed one. An empty result and a refusal are different answers: key on engine being present to tell “this engine cannot” from “nothing happened”.

Example (canonical for the family; the others differ only in the asserted property):

verify_text({ selector: '[data-testid="status-chip"]', text: "Reviewed" })
// → { "ok": true }

verify_count({ text: "Wrong Type", n: 0 })
// → { "ok": false,
//     "failure": { "source": "app", "kind": "count", "expected": 0, "actual": 2 },
//     "tokensEstimate": 52 }

verify_visible({ ref?|selector?|named?, session? })

Asserts the element is currently visible (non-zero box, displayed, opacity > 0). On failure, actual carries a one-word reason: "hidden (display:none)", "hidden (visibility:hidden)", "hidden (opacity:0)", "hidden (zero-sized box)", "off-screen or covered", or "missing (locator matched 0 nodes)".

verify_text({ ref?|selector?|named?, text, exact?, session? })

Asserts the element’s visible text matches. Default: case-insensitive substring on the trimmed innerText. exact: true gives case-sensitive equality. failure.actual carries the first 200 chars of what we saw.

verify_value({ ref?|selector?|named?, value, session? })

Asserts the targeted form-control’s current value (input / textarea / select / contenteditable). Strict equality on the DOM-side value (or innerText for contenteditable). Pairs with ActionResult.element.value from fill, so you can assert the post-fill state without an extra round-trip.

verify_count({ selector?|text?, n, session? })

Asserts exactly n matches. One of selector (raw CSS / Playwright locator) or text (case-insensitive visible-text search over the composed a11y tree) is required. Use for grid/list invariants: “5 rows remain after the delete”, “no Wrong Type chips left in the record grid”.

verify_attribute({ ref?|selector?|named?, attr, value?, session? })

Asserts the element’s HTML attribute. Pass value for strict-equality; omit value to assert mere presence. Use for aria-pressed, data-state, disabled, role state that doesn’t surface as visible text.

verify_predicate({ predicate, data, session? })

Composed predicate check over caller-supplied data. Fixed vocabulary, NOT arbitrary JS. The agent supplies data (which key, which expected value); the vocabulary is server-owned.

The predicate.kind enum:

• Leaves: equals, notEquals, contains, notContains, gt, lt, gte, lte, between, matches (regex string), exists.
• Combinators: and, or, not (recursive; combinators take a predicates array of child predicates).

Each leaf carries {kind, key, value} (or {kind, key, lo, hi} for between). key is a dotted accessor (e.g. "actionResult.element.value", "snapshot.warnings.length") and must start with an allow-listed root: actionResult, snapshot, element, value, expect. The .length suffix over an array or string returns the numeric length.

eval_js (gated behind the eval capability) remains the only arbitrary-JS path in browxai. verify_predicate does not add a second one: it shares the predicate vocabulary with batch.expect (one source of truth lives in src/util/predicates.ts). Use it as a deterministic gate on an already-captured ActionResult / snapshot / metric, the screenshot-judge analogue when chained behind a screenshot.

console_read

Recent console messages (ring buffer). For per-action attribution, use ActionResult.console from any action tool.

Inputs: { limit?: number (default 50, max 500) }

Output: JSON array of { ts, type, text }. A message longer than 8,192 characters is stored cut to that length and ends in … [truncated N chars].

Example.

console_read({ limit: 20 })
// → [{ "ts": 1765540264012, "type": "error", "text": "Uncaught TypeError: x is not a function" }, …]

network_read

Session-wide ring buffer of recent network requests (cap: 500). network_read({ limit?, session? }) returns the most recent limit requests (default 50, max 500). For per-action attribution use ActionResult.network from any action tool, still the primary surface. This is the “what happened across the session” view; useful when an XHR isn’t tied to a specific action. Same noise-folding rules as the action-window tap (Image/Font/Stylesheet/Media/beacons land in summary.byType.other).

sample

Sample a DOM metric over a window and return a time series. Jank / CLS / scroll-drift QA without hand-rolling an in-page loop. sample({ session?, ref?|selector?|named?, metric, durationMs, everyFrame?, intervalMs? }):

• metric is a fixed enum: the agent supplies no JavaScript (arbitrary JS stays eval_js, gated behind eval). With a target: scrollTop/scrollLeft/scrollHeight/scrollWidth/clientWidth/clientHeight/bboxX/bboxY/bboxWidth/bboxHeight. Without a target: the document scroller (bbox* rejected, since it needs an element).
• everyFrame: true: requestAnimationFrame loop; else intervalMs (default 100, min 16).
• Returns { metric, scope, durationMs, mode, count, series?: [{ tMs, value }], summary, autoSummarised?, truncated? }. Caps: 30 s, 2000 points. Read-only (read).
• summary: { count, min, max, first, last, distinctCount, firstChangeTMs }, always included (cheap). The summary arg is tri-state series-omission: true omits the full series; false always includes it; omit the arg for the default, where the series is auto-dropped only for large windows (>300 collected points), with autoSummarised: true on the result so the agent knows to re-request with summary:false if it needs the raw set. Pure server-side reduction; no agent JS.

browxai supplies the fixed in-page rAF/interval loop. The primitive is bounded, not an eval_js variant.

Example.

sample({ selector: ".feed", metric: "scrollHeight", durationMs: 3000, intervalMs: 250 })
// → { "metric": "scrollHeight", "scope": "element", "durationMs": 3000, "mode": "interval",
//     "count": 12, "series": [{ "tMs": 0, "value": 4200 }, …],
//     "summary": { "count": 12, "min": 4200, "max": 6300, "first": 4200, "last": 6300,
//                  "distinctCount": 4, "firstChangeTMs": 750 } }

act_and_sample

Run one action and capture a metric trace across its transition, in a single call. Closes the state-capture-latency blind spot: a separate read after an action lands after the transient UI (spinner / pending button / in-flight counter) has already resolved, so the agent wrongly scores it “fine”. act_and_sample({ session?, action: { tool, args }, ref?|selector?|named?, metric, durationMs, everyFrame?, intervalMs?, summary? }):

• action is a { tool, args } from the batch whitelist (no batch / await_human / recording-control / self). The inner tool’s own capability gate, the confirm hooks, and the anti-wedge deadline all still apply.
• The sampler (sample’s fixed enum, no agent JS) starts, the inner action dispatches concurrently, both are awaited. Sampler self-bounds via durationMs; the action via its deadline. Pick durationMs to cover the expected transition.
• Sample target via ref/selector/named (or omit: document scroller; coords rejected). Same metric enum / caps / summary semantics as sample.
• Returns { action: <inner tool result>, sample: { metric, scope, mode, count, series?, summary, … } }.

No agent JS anywhere: it reuses sample’s fixed-enum sampler + batch’s tool whitelist; eval_js (gated) stays the only arbitrary-JS path.

Example.

act_and_sample({
  action: { tool: "click", args: { ref: "e12" } },
  metric: "clientHeight", selector: ".results-panel", durationMs: 2000,
})
// → { "action": { /* the click's ActionResult */ },
//     "sample": { "metric": "clientHeight", "scope": "element", "count": 20,
//                 "summary": { "first": 0, "last": 480, "firstChangeTMs": 430, … } } }

watch

Observe a fixed time window with no driving action. Samples top-level transient surfaces (dialog/alertdialog/alert/status/tooltip/log/banner/timer) every sampleMs (default 250) so a region that appears and disappears inside the window is caught. Endpoint-only diffs (ActionResult.structure) miss it. watch({ session?, durationMs, sampleMs? }) returns { durationMs, samples, regions: [{ role, name, ref, appearedAtMs, disappearedAtMs }], console, network, wsFrames }. disappearedAtMs: null = still present at window end. Catches double-fire toasts, flash-of-content, “notification never broadcast”. Read-only (read); caps at 60 s.

Example.

watch({ durationMs: 5000 })
// → { "durationMs": 5000, "samples": 20,
//     "regions": [{ "role": "status", "name": "Saved", "ref": "e90",
//                   "appearedAtMs": 750, "disappearedAtMs": 2250 }],
//     "console": { "errors": [], "warnings": 0 }, "network": { … }, "wsFrames": [] }

network_body (gated)

Fetch a full response body by requestId (from network_read or ActionResult.network.requests[].requestId). Off by default: requires the network-body capability in BROWX_CAPABILITIES (loud startup warning when enabled). Returns { ok, body?, base64Encoded?, truncated?, error? }; bounded at 256 KB (truncated:true past that). Best-effort: the renderer discards bodies fast, so fetch right after the request; not retained across navigations. On Firefox and WebKit the body is read when you ask for it, and only if its size is known and within the cap: a response whose content-length (or finished transfer size) is over 256 KB, or unknown, comes back as { ok:false, truncated:true, error } and is not read. A compressed response must be under 32 KB on the wire.

Why gated: full bodies routinely carry PII / auth tokens. The responseShape (top-level keys only) is the safe default for “did the mutation write back the right shape”; network_body is the higher-risk debugging escape hatch for “assert this exact field value” (e.g. a realtime broadcast payload, paired with ws_read).

Example.

network_body({ requestId: "req-41" }) // id from network_read / ActionResult.network.requests[]
// → { "ok": true, "body": "{\"id\":\"rec_1\",\"type\":\"engineering\"}",
//     "base64Encoded": false, "truncated": false }

inspect

Read an element’s whitelisted computed styles + box + overflow/clip state. inspect({ session?, ref?|selector?|named?, styles? }) returns { found, box: {x,y,width,height}, styles, overflowing: {x,y}, visible, childCount }. The layout-break / control-state verification primitive, distinct from find() (ranking) and text_search (presence):

• Default style set: display, visibility, opacity, position, cursor, pointerEvents, overflow{,X,Y}, zIndex, flexDirection, justifyContent, alignItems. styles: [...] appends extra camelCase property names.
• overflowing.{x,y}: scrollWidth/Height > clientWidth/Height (the “label clips / content overflows” signal).
• childCount: direct element children (catch “a flex row lost its 3rd child: misalignment”).
• cursor distinguishes not-allowed vs wait vs pointer (disabled-vs-busy control state).

Read-only (capability read). Coords targets unsupported (no element to resolve); use point_probe for a coordinate.

Example.

inspect({ ref: "e17", styles: ["backgroundColor"] })
// → { "found": true, "box": { "x": 940, "y": 212, "width": 56, "height": 28 },
//     "styles": { "display": "flex", "cursor": "not-allowed", "overflowX": "hidden", …,
//                 "backgroundColor": "rgb(243, 244, 246)" },
//     "overflowing": { "x": false, "y": false }, "visible": true, "childCount": 2 }
// cursor "not-allowed" + visible:true → the control is rendered but disabled.