Tool reference: canvas, diagnostics, human helper and native apps

Canvas-app automation (capability canvas)

Off-by-default. App-agnostic primitives for driving canvas-based editors (Figma, Tldraw, Excalidraw, video editors, drawing apps, anything that paints into a <canvas> instead of laying out DOM). Five MCP tools + a pure-RGBA diff:

• canvas_capture: framebuffer / 2D ImageData / PNG bytes.
• canvas_diff: pixel/region delta over RGBA captures (read capability, pure byte math).
• gesture_chain: multi-step pointer program.
• canvas_world_to_screen / canvas_screen_to_world: affine helpers (explicit or heuristic-discovery).
• canvas_query: dispatcher to a canvas-app adapter plugin.

canvas_capture({ ref?, selector?, format, session? })

Extract framebuffer or 2D ImageData from a <canvas> element. Three formats:

• format:"png": canvas.toDataURL("image/png"). Returns { ok, format:"png", contentBase64, byteLength, width, height }. Suitable for handoff to the host agent’s multimodal vision call (see BYO-vision pattern below).
• format:"2d-imagedata": getImageData(0, 0, width, height). Returns { ok, format:"2d-imagedata", contentBase64 (RGBA, row-major, top-left origin), width, height, channelCount: 4 }. Feed to canvas_diff for pixel math.
• format:"webgl-framebuffer": gl.readPixels(0, 0, w, h, RGBA, UNSIGNED_BYTE, …). Returns the same RGBA shape as 2d-imagedata plus isWebGL: true. The page-side capture flips the result into top-left order so downstream canvas_diff math is consistent across the two RGBA formats.

ref optional (canvas element ref from a prior snapshot() / find()); selector is a fallback selector path; omitting both targets the first <canvas> in the document.

Bounded: canvases larger than 16384×16384 pixels refuse with { ok:false, code:"too-large" }. Defensive cap: most editors stay well below this; a multi-megapixel buffer round-tripped through base64 is genuinely a problem.

Taint: toDataURL / getImageData throw SecurityError on canvases that have drawn cross-origin images without CORS. The page-side function catches and surfaces { ok:false, code:"taint-or-encode" } / { ok:false, code:"taint-or-read" }.

WebGL preserveDrawingBuffer: canvas_capture requests preserveDrawingBuffer:true when it acquires a WebGL context, but it cannot undo a prior context’s choice. Pages that explicitly set preserveDrawingBuffer:false may read back as zero bytes; this comes from the platform.

canvas_diff({ beforeBase64, afterBase64, width?, height?, region?, inputFormat?, session? })

Pure function: pixel/region delta over two RGBA captures. Returns { ok, changedPixelCount, changedBytes, percentageChanged, bboxOfChanges:{x,y,w,h}|null, warnings[] }.

• RGBA inputs require width + height (the byte buffer alone does not carry dimensions). Over-flow region rectangles clamp to image bounds and do not throw.
• changedBytes is the sum of absolute per-channel deltas. Useful for “how much changed” as well as “did anything”.
• bboxOfChanges is the tight bounding box of the changed area in image coordinates. Null when no pixels changed.

PNG-format inputs (deferred): pass inputFormat:"png"; this cycle compares base64 byte equality only and surfaces a warning. Per-pixel diff over PNG is a follow-up; for bbox + per-channel math today, recapture with 2d-imagedata or webgl-framebuffer.

Example (did the stroke land?).

canvas_capture({ format: "2d-imagedata" })          // before — keep contentBase64 + width/height
// …gesture_chain draws the stroke…
canvas_capture({ format: "2d-imagedata" })          // after
canvas_diff({ beforeBase64: "<before>", afterBase64: "<after>", width: 1280, height: 720 })
// → { "ok": true, "changedPixelCount": 1840, "changedBytes": 96214,
//     "percentageChanged": 0.2, "bboxOfChanges": { "x": 210, "y": 80, "w": 120, "h": 60 } }

gesture_chain({ steps, session? })

Multi-step pointer program. Each step is { kind, x?, y?, deltaX?, deltaY?, ms?, pointerId? }. Returns { ok, stepsExecuted, totalDurationMs, warnings[] }.

• kind:"down" | "up" | "move": require numeric x + y. move accepts optional ms pacing delay; values below 5 ms floor to 5 ms with a warning (tighter pacing rarely changes app behaviour and starves the renderer).
• kind:"wait": bounded sleep; ms clamped at 5000 ms with a warning (split longer waits across calls).
• kind:"wheel": requires non-zero deltaX or deltaY; accepts optional x + y to move the pointer first.
• 200 steps max in total, refusing with code:"too-many-steps". Split larger programs across multiple calls.

pointerId is accepted on input but the v1 implementation routes through Playwright’s single-mouse pipeline; multi-pointer fan-out is a future extension. For multi-touch gestures today use touch_* / gesture_pinch / gesture_swipe.

canvas_world_to_screen({ worldX, worldY, ref?, selector?, transform?, session? }) and canvas_screen_to_world({ screenX, screenY, ref?, selector?, transform?, session? })

Affine coord-space translation. Two modes:

• Explicit: caller passes transform: { scale, panX, panY, originX?, originY? }. Math: screenX = (worldX + panX) * scale + originX (and the inverse). Pure function, no page contact.
• Discovery: omit transform to trigger a page-side probe of common app-side globals:
  • app.viewport.zoom + app.viewport.center.{x,y} returns Figma / Excalidraw shape (adapterHint:"figma").
  • app.scale + app.offset.{x,y} returns Tldraw shape (adapterHint:"tldraw").
  • app.transform.matrix (6-element affine [a,b,c,d,e,f]) Returns generic shape (adapterHint:"generic").

On discovery success: { ok, screenX, screenY, transformDiscovered, adapterHint, warnings:["discovery probes are HEURISTIC — …"] }.

On discovery failure: { ok:false, error:"no transform discoverable — pass `transform` explicitly OR use a canvas-app adapter plugin", code:"no-transform" }.

Discovery is HEURISTIC by design. For production, either pass transform explicitly (e.g. read it out of your app’s React state via eval_js, then feed it to the explicit-mode path) or install a canvas-app adapter plugin that owns the transform discovery for your app.

The inverse round-trips with the forward call to within floating-point precision under the same explicit transform.

canvas_query({ adapter, op, args?, session? })

Dispatcher routing to a canvas-app adapter plugin’s handler. adapter is the namespace of a loaded plugin (e.g. "figma"); the tool looks up <adapter>.<op> in the live plugin tool registry and forwards args (with the session passed through).

When no plugin matches: { ok:false, error:"no canvas adapter registered for <adapter>; install @browxai/plugin-<adapter> or pass a registered adapter namespace", code:"no-adapter", requestedAdapter, requestedOp }.

When a plugin matches: the inner plugin tool’s own capability is enforced via the plugin call-graph gate, so a canvas capability turned on alone is not enough to invoke an adapter operation whose plugin declared a different gate.

The dispatcher ships in the host; the canvas-app adapter plugins (@browxai/plugin-figma, @browxai/plugin-tldraw, @browxai/plugin-excalidraw) install separately via browxai plugin install. canvas_query is a forward-compatible API: writing an agent loop against canvas_query({adapter:"figma", op:"…"}) works as soon as the operator installs the matching plugin. The full per-adapter op surface (every op, args, return shape, error codes) is documented in First-party plugins.

Canvas-app automation: BYO vision pattern

browxai is BYO-vision by design. Owner direction 2026-05-30: no bundled OCR, no hosted vision API. browxai’s job is to be a substrate for canvas-app automation: pixels, gestures, transform math, plugin dispatch. Understanding what the pixels mean is the host agent’s multimodal vision call.

The composition loop:

1. Capture: canvas_capture({format:"png"}) returns base64 PNG bytes.
2. Understand: the host agent passes the PNG to its own multimodal-vision call (Claude / GPT-4V / Gemini Pro Vision / etc) with a prompt like “Identify the bounding box of the ‘Delete’ button on this Figma canvas”. The agent returns viewport-space coordinates.
3. Act: gesture_chain({steps:[{kind:"down", x, y}, {kind:"up", x, y}]}) or mouse_* / click to drive the next step.

Worked example, “click the Delete button on the currently-selected Figma node”. Both calls need the canvas capability active on the client (createBrowxai({capabilities:["canvas", …]})) AND on the server (BROWX_CAPABILITIES). The two gate independently:

// 1. Capture the canvas as a PNG.
const png = await client.callTool("canvas_capture", { format: "png" });

// 2. Hand it to your multimodal vision call. (Pseudocode — adopter wires
//    their own model invocation here.)
const { x, y } = await yourVisionAgent.locate({
  imageBase64: JSON.parse(png.content[0].text).contentBase64,
  query: "the Delete button on the top toolbar",
});

// 3. Drive the gesture.
await client.callTool("gesture_chain", {
  steps: [
    { kind: "down", x, y },
    { kind: "up",   x, y },
  ],
});

Why BYO: bundling a vision call into browxai would (a) lock the substrate to a single vision provider (the curator does NOT want to pick winners on the modality side), (b) require browxai to ship model credentials / per-call billing / a configured-provider chain analogous to the captcha and credentials capabilities (additional ops burden, additional posture-broadening surface), (c) collapse a clean composition boundary, where the host agent owns what to do and browxai owns how to do it. The BYO posture preserves the property that browxai is RC-independent and substrate-pure; the vision dimension is the host agent’s choice.

For app-specific understanding without vision: install a canvas-app adapter plugin. An adapter plugin can read scene-graph node bounds / layer ids / frame names directly from the app’s own state (via eval_js or app-specific RPC) and surface them as structured canvas_query({adapter:"figma", op:"getNodeBounds"}) lookups, with no vision call required for the cases the app’s internals already answer.

Diagnostics (capability diagnostics)

Off-by-default per-call recording layer + agent self-feedback. The capability adds three surfaces and one implicit recorder hook:

1. The recorder hook at the MCP-handler dispatch boundary. When the capability is OFF, the hook is a single boolean gate check (no allocations, no file IO, no observable side-effect). When ON, every dispatched tool call lands as a JSONL line. The recorder runs **DOWNSTREAM of the URL sanitiser
   • secrets-masking egress chokepoint**: by the time the recorder sees a result, every egress sink has already rewritten registered secret values back to <NAME> aliases; args are additionally walked through applyMaskDeep so a secret echoed in the call args never lands raw in the store. Capability: diagnostics.
2. diagnostics_note: agent self-feedback.
3. diagnostics_search: read-side query (rides read).
4. diagnostics_report: analysis primitive (rides read).

JSONL store layout + retention

Recorded under $BROWX_WORKSPACE/diagnostics/<sessionId>/<server-start-ISO>.jsonl, one file per session per server-start ISO timestamp, append-only. Retention is config-driven via BROWX_DIAGNOSTICS_RETENTION_DAYS (default 30; 0 disables the sweep). Expired session directories are removed on server start AND on session close; a closed session’s recorded history is discarded along with its other per-session state. Workspace-rooted by construction: a session id that escapes the diagnostics subdir (../escape, an absolute path) is rejected at the path-resolution chokepoint and the dispatch path falls back to a no-op (the call still runs; only the recording is skipped).

Record shapes

Call records (kind:"call"):

{
  "kind": "call",
  "ts": "2026-06-08T12:34:56.789Z",
  "tool": "click",
  "sessionId": "default",
  "argsRedacted": {
    // structural — keys + types + sizes
    "selector": "button[data-testid=save]",
    "value": { "__redacted": true, "sha256": "…", "byteLength": 12345 },
  },
  "resultMeta": {
    "ok": true,
    "sizeBytes": 482, // total result envelope byte length
    "warningsCount": 0,
    "failureKind": "target-not-found", // only present on ok:false
  },
  "durationMs": 12,
  "capabilityDenials": 3, // cumulative across the recorder
  "evalJs": {
    // only present for eval_js / poll_eval
    "exprSha": "…",
    "exprHead": "document.querySelector('#save')",
    "returnType": "string",
    "returnSizeBytes": 24,
    "taxonomy": "dom-query", // dom-query | storage-access | computed-style | callback-trigger | feature-detect | custom
  },
}

Note records (kind:"note"):

{
  "kind": "note",
  "ts": "2026-06-08T12:34:56.789Z",
  "sessionId": "default",
  "insight": "would like an inner_text tool that returns text without eval_js",
  "category": "missing-primitive", // missing-primitive | workaround | perf-concern | ergonomic-friction | other
  "severity": "warn", // info | warn | blocker
  "ref": "eval_js:2026-06-08T12:34:56.000Z", // optional pointer at a prior call
}

failureKind taxonomy (synthesised from the structured error string): one of capability-denied, timeout, target-not-found, bad-arg, internal.

diagnostics_note({ insight, category?, severity?, ref?, session? })

Agent self-feedback. Writes a kind:"note" record carrying a free-text observation plus optional category / severity / ref. Default category other, default severity info. Filing a note implies the recorder is engaged, so this tool sits under the diagnostics capability: a server with the capability OFF returns a structured refusal and never swallows feedback silently. Intended consumer: the curator deciding which primitive to lift next.

diagnostics_search({ since?, tool?, category?, sessionId?, limit?, session? })

Read-side query over the JSONL store. Returns matching records (calls + notes combined) up to limit (default 100, hard cap 1000). since filters by ts (ISO); tool filters by tool name (exact match; applies to kind:"call" only); category filters by note category (exact match; applies to kind:"note" only); sessionId filters by session. The recorder is gated on diagnostics; this query reads whatever lives on disk, so a server with diagnostics OFF but a non-empty workspace history can still surface prior runs. Capability: read. Returns { ok, records, count, truncated }.

diagnostics_report({ format?, since?, sessionId?, session? })

Analysis primitive. format defaults to summary:

• perTool: per-tool { count, failureCount, p50Duration, p95Duration }.
• topEvalJsPatterns: the top 10 eval_js patterns by count, each carrying { exprSha, exprHead, count, taxonomy }.
• capabilityDenials: per-tool denial counts.
• notesByCategory: note-bucket counts.
• missingPrimitiveHypotheses: eval_js taxonomy buckets surfaced as candidates for a curated primitive. Heuristic: any non-custom taxonomy with count ≥ 3, or any custom pattern with count ≥ 5.

format: "full" additionally streams the per-record list capped at 500 records (truncated: true when exceeded). Optional since (ISO) windowing + sessionId filter narrow the rollup. Capability: read.

Example (family).

diagnostics_note({
  insight: "needed three eval_js calls to read one computed style — inspect({styles}) covers it",
  category: "ergonomic-friction", severity: "warn",
})
diagnostics_report({ format: "summary" })
// → { "perTool": { "click": { "count": 41, "failureCount": 2, "p95Duration": 230 }, … },
//     "topEvalJsPatterns": [{ "exprHead": "document.querySelector('#save')", "count": 7,
//                             "taxonomy": "dom-query" }],
//     "missingPrimitiveHypotheses": ["dom-query"], … }

Secrets-masking composability

The recorder hook composes with the per-session secrets registry by construction: args land in the JSONL after applyMaskDeep has rewritten every registered real value back to its <NAME> alias; results land in the JSONL after every egress sink (network, console, ws, snapshot, find, text_search, network_body) has already done the same. Test src/util/diagnostics.test.ts > secrets-masking composability registers a secret, drives a tool call that carries the raw value in args, and asserts the JSONL records the redacted form, never the raw value.

Human and agent helper

await_human({ kind?, prompt, choices?, timeoutMs? })

Blocks the calling agent until the human responds. kind defaults to acknowledge. timeoutMs defaults to 300000 (5 minutes) and is capped at 3600000; an unanswered prompt is denied at the timeout. The prompt is logged to stderr. The prompt names the session’s isolated world (browxai-<random>) and a short ticket. The operator answers from DevTools after switching the console’s context dropdown from top to that world, which the page’s own scripts cannot reach, and passes the ticket as the last argument. An answer without the current ticket, or after its prompt ended, is dropped. The supported kinds:

• acknowledge: __browx.proceed(ticket) (the original site-docs manual-capture use case)
• confirm: __browx.confirm(true, ticket) or __browx.confirm(false, ticket)
• choose: __browx.choose(<index>, ticket) (with choices: ["A", "B", "C"] shown in the prompt; the human responds with 0/1/2)
• input: __browx.input("typed text", ticket)
• pick_element (in-page hover-pick overlay) is not yet available; it needs the shadow-DOM banner UI.

Returns: { kind, value, timedOut }. For typed kinds, value is the user-supplied value (boolean / index / string); for acknowledge, it’s whatever was passed to proceed(…) (often null).

Operator channel. When the operator started the server with the off-by-default operator-channel capability and a socket (BROWX_OPERATOR_SOCKET, BROWX_OPERATOR_TOKEN), the prompt goes to the host daemon on that socket instead, and only the daemon’s answer counts: done (with the value its kind needs), or abort, which returns value: null and an error starting operator-aborted. DevTools answers are ignored while it is on, an unanswered prompt returns timedOut: true at timeoutMs, and no CDP is needed, so it also works on engines that have no isolated world. The call and its result look the same to the agent.

Engines. The world is created over CDP, so the human channel exists on chromium, an attached Chrome, Chrome on Android and Electron. On firefox, webkit, safari and the native engines there is no such world: await_human returns at once with value: null, timedOut: false and an error starting no-human-channel, and the confirm hooks and ask-human policies on those sessions fail closed.

Example.

await_human({ kind: "choose", prompt: "Which account should I use?",
              choices: ["alice@example.com", "bob@example.com"], timeoutMs: 120000 })
// → { "kind": "choose", "value": 0, "timedOut": false }

The __browx helper

Two copies, in two JS worlds.

• In the session’s browxai-<random> isolated world (the answering one). browxai creates the world over CDP on every page the session owns, evaluates the helper in it, and installs a per-session CDP binding scoped to that world name. A call is accepted only when CDP reports it came from one of those world contexts, and never from a context with an extension origin. Re-created on every navigation. The random name keeps an extension named browxai out of it.
• In the page’s main world (window.__browx, display-only). Every method logs where the real channel lives and returns false; status() returns { state: "display-only" }. Page scripts can call, replace or delete it without effect on the server. The data-browx-signal attribute is no longer read.

__browx = {
  signal(name, data?),         // generic; e.g. __browx.signal("paywall-hit")
  proceed(ticket, data?),       // answers await_human({kind:"acknowledge"})
  abort(reason?, ticket?),
  done(what, data?),           // "I did X" — signal("did", { what, data })
  respond(value, ticket) / confirm(yes, ticket) / choose(idx, ticket) / input(text, ticket),
  status(),                    // { state: "ready" } in the browxai world
};

An attached session shares its browser with other sessions, so its bridge wires only its own leased tab and popups opened from it. The shadow-DOM banner UI and the pick_element overlay are not yet available.

Live view (operator channel)

Requires the off-by-default live-view capability, and operator-channel with it. With live-view on and operator-channel off the server refuses to start. It is not a tool and has no tool: the agent cannot start, stop or read it, and nothing an agent can call reports whether a view is running. Only the host daemon on the operator socket controls it, per session, with view.start and view.stop. With the capability off, a view.start gets { "type": "error", "code": "view-disabled" } and nothing starts.

Frames are page pixels and go to that socket only. They are not in any tool result, log, artifact, report, HAR or recording, and secret masking does not cover them: a registered secret that is visible on screen reaches the daemon as drawn, as in screenshot. See Threat model section 7.

Daemon to browxai:

{ "v": 1, "type": "view.start", "session": "default", "maxFps": 5, "maxWidth": 960, "quality": 60, "format": "jpeg" }
{ "v": 1, "type": "view.stop", "session": "default" }
{ "v": 1, "type": "frame.ack", "session": "default", "seq": 412 }

Every field but session is optional. Defaults are 5 fps, 960 px and quality 60. A larger number is clamped to 5 fps, 1280 px and quality 80 (lower bounds 0.2 fps, 160 px, quality 10), and a field of the wrong type or a format other than jpeg is refused with invalid-view. A second view.start for a running session replaces its stream.

browxai to daemon:

// the stream started, with the values that applied
{ "v": 1, "type": "view.started", "session": "default", "format": "jpeg", "maxFps": 5, "maxWidth": 960, "quality": 60 }
// one frame, as parts: the base64 JPEG of every part joins in order
{ "v": 1, "type": "frame", "session": "default", "seq": 412, "at": 1790000000000, "format": "jpeg", "width": 960, "height": 540, "part": 0, "parts": 2, "data": "<base64>" }
{ "v": 1, "type": "view.stopped", "session": "default", "reason": "daemon" }
{ "v": 1, "type": "error", "code": "unknown-session", "session": "default" }

• Parts. A line over 64 KiB drops the connection, and a JPEG of a busy page is often larger than that. So a frame goes as parts lines of at most 40,960 base64 characters (30 KiB of JPEG) each, part counting from 0, all with the same seq, written together. Each part decodes alone, so the daemon can decode and append. A frame needing more than 16 parts is dropped. width and height are the JPEG’s own size.
• Acks. One frame is in flight per stream. Send frame.ack for a seq once its last part is in and handed on. A frame that arrives before the last is acked, or while the socket is backed up, is dropped, never queued. Ack late, or not at all, when the path behind the daemon is slow: browxai takes that as slowness and steps the stream down. A frame unacked after 5 s counts as lost, and a stream with no ack for 30 s ends with reason: "stalled".
• Step-down. From the requested size browxai moves in four steps to 1 fps and 640 px when a 2 s window drops at least 40% of its frames, and back up after 10 clean seconds with room in the ack round trip. The frame’s width shows where it is.
• Stop. view.stopped carries reason: daemon, session-closed, channel-down, channel-closed, stalled, page-closed (the session’s page closed) or source-error. A dropped connection ends every stream, and a redial resumes none.
• Error codes. view-disabled, invalid-view, unknown-session (no open session by that name: a view never opens one), view-unsupported (the engine has no CDP handle, so chromium, an attached Chrome, android and electron stream), view-limit (4 streams at once), view-failed. None carries page content.

Native app control (ios-app, android-app) (gated)

Requires the off-by-default native-device capability. open_session({ engine: "ios-app" }) and open_session({ engine: "android-app" }) both refuse without it, so the gate sits at session creation and no native tool can be reached around it. See Threat model for what the capability grants.

browxai drives two native engines. ios-app drives a native application on the iOS Simulator through its XCUITest accessibility hierarchy. android-app drives an app on an Android emulator over adb, UiAutomator for the view hierarchy, the OS input pipeline for taps, swipes and keys, screencap for frames. Neither is a browser: there is no Page, no DOM, no URL and no Locator. What each has instead is a view hierarchy that composes into the same A11yNode tree snapshot already returns, with the same [ref=eN] refs, so an agent skill written for web transfers and one CI verifier reads every kind of session.

Both share the contract that matters: the tier-1 selector is the accessibility identifier (a React Native testID compiles to iOS’s accessibilityIdentifier and Android’s resource-id), a ref minted on a node that carries one is anchored on the identifier and not on the path, every action re-resolves against a freshly read hierarchy before it dispatches, and an ambiguous match refuses instead of picking the first. Unlike the web engines, which keep a silent first-match pick for compatibility.

Real devices are out of scope by policy. Both engines’ code paths reach a physical handset unchanged.

The platform toolchains are operator-supplied. browxai never bundles Xcode, WebDriverAgent or the Android SDK, never installs any of them, and never creates an AVD or a simulator.

The ios-app engine (iOS Simulator)

Real devices are out of scope for this engine specifically. They need code signing, a provisioning profile, a WebDriverAgent build and attached hardware.

Two operator-supplied pieces, neither bundled nor auto-installed. Xcode supplies xcrun simctl, which does simulator and app lifecycle. WebDriverAgent (Apache-2.0) supplies the XCUITest hierarchy and the input pipeline; you build and run it, and point browxai at its port. With nothing listening, session creation refuses with wda-unreachable and names the fix. browxai takes no npm dependency for either and reaches nothing GPL.

variable	meaning
BROWX_IOS_DEVICE	Simulator udid or exact device name. Omitted means a booted simulator, else the newest runtime.
BROWX_IOS_APP_ID	Bundle id of the app under test. Required.
BROWX_IOS_APP_PATH	Path to a .app to install before launching. Omitted means assumed already installed.
BROWX_IOS_WDA_URL	Where WebDriverAgent listens. Default http://127.0.0.1:8100.

The selector model: accessibility identifier over position

accessibilityIdentifier, what a React Native testID compiles to, is the tier-1 selector, and it changes how a ref is minted. When a node carries one, the structural path is excluded from the ref’s key, so the ref survives a layout change that moves the node. When it does not, the path is included and the ref is snapshot-local: move the node and the old ref reports stale-element and never resolves to whatever now occupies its rectangle. find emits [accessibilityIdentifier="…"] as its tier-1 hint; ~checkout-submit, [testID="…"] and [identifier="…"] all resolve to the same query.

Every action re-resolves before it dispatches. A ref is matched against a freshly read hierarchy immediately before each tap, and the tap point is computed from the element that match returned. No coordinate is cached between calls. Zero matches is a stale-element refusal; more than one match is also a refusal, naming the count. Unlike the web engines, which resolve an ambiguous target to the first match.

What the hierarchy mapping loses. type, then role is a table: Button, then button is exact, Other, then generic is coarse, and the raw XCUITest type is preserved on the node’s tag. iOS StaticText maps to text, deliberately not to browxai’s presentational StaticText, on the web that role is furniture the serialiser drops, and on iOS it is usually the screen’s content. label, then name and value, then value are exact; a placeholder rides on value only when the element has none.

Runs: snapshot, find, text_search, click (tap), fill, press, scroll, gesture_swipe, gesture_pinch, verify_*, screenshot, navigate (deep links only, pass a full URL scheme), name_ref.

Refuses, each naming the missing primitive: touch_start / touch_move / touch_end (XCUITest has taps, drags and pinches, no raw touch pipeline), hover (no pointer), select / choose_option (no <select>), go_back / go_forward (no history stack; iOS has no back key), set_viewport (screen size is the device type), wait_for (unmeasured dump cost), element-scoped and jpeg screenshot (cropping needs a PNG decoder browxai does not bundle), pdf_save, the whole network / storage / script / emulation families, and the CDP-deep tools (deep: false).

Lifecycle: app_list, app_install, app_launch, app_terminate and app_foreground run here through the shared tool family below. app_uninstall and app_reset refuse, naming the engine: simctl uninstall would remove an app the operator may have installed by hand from a device the session leases and does not own, and iOS has no per-app data clear at all, simctl erase wipes every app on the device. device_list reports simulators alongside adb devices; device_boot and device_shutdown are adb/AVD-scoped, because an ios-app session boots the simulator it resolves at session creation and never shuts one down. The session’s own device and app still come from the environment above.

Not yet: segmented video capture and the native/* replay event types (RFC 0008 P3), and console_read over the simulator syslog.

The android-app engine (Android emulator)

engine: "android-app" drives an app on an Android emulator over adb. The Android SDK is operator-supplied: browxai never bundles it, never installs it and never creates an AVD.

The selector model: testID over positional refs

testID (Android: the view’s resource-id) is the only tier-1 selector, and a ref minted on a node that carries one is anchored on the testID, not the path, elementKey is given an empty path, so the ref survives a layout change that moves the node. Without a testID the full index path applies and the ref is snapshot-local, which is the honest status of an unlabelled element. snapshot warns when an app is thin on testIDs, naming the count, so a degraded selector model becomes a fixable list for the app team.

Every action re-resolves before it dispatches. A tap reads a fresh hierarchy, finds the element, and dispatches at the point that read just returned, in the same call. No coordinate is ever cached or replayed. An ambiguous query refuses: a ref or selector matching more than one element taps nothing and reports the count, because a reported tap on the wrong element is worse than a failed tap.

Native selector vocabulary, accepted anywhere selector is:

form	tier	matches
~checkout-submit	testID	Appium’s accessibility-id spelling
testID= / id=	testID	the same tier, spelled out
label=Place order	label	accessibilityLabel (Android content-desc)
text=Place order	text	the rendered string
role=button[name="Save"]	role	role plus label
a bare string	any	testID, then label, then text; reports which

What works, and what refuses

tool	on android-app
snapshot / find / text_search / find_feedback	yes, over the UiAutomator hierarchy, composed into A11yNode
click	yes, a real OS tap at a freshly-resolved point
fill	yes: taps the field, then input text. Registered secrets do NOT materialise. A <NAME> alias is typed literally
press	yes: back / home / menu / appswitch / enter / arrows / any KEYCODE_*; a single character is typed
scroll	yes with to (top/bottom/left/right) or by. Scroll-into-view is not a native primitive and refuses
gesture_swipe, touch_start / touch_move / touch_end	yes, real platform primitives (input swipe, input motionevent). touch_* needs API 30+
gesture_pinch	refuses (native-pinch-needs-multitouch-driver). See below
verify_visible / _text / _value / _attribute / _count	yes, through the element port, over the hierarchy
screenshot, screenshot_region	yes, PNG off screencap; element-scoped captures crop the frame to the bounds just read. format:"jpeg" refuses
navigate	deep links only, navigate({url:"myapp://checkout/42"}) fires a VIEW intent. A native screen has no address bar
go_back	yes, the platform BACK key. go_forward has no Android analogue and refuses
eval_js / poll_eval	refuses: no scriptable context in a release-configuration RN app
network_read / network_body / route / route_queue	refuses: no protocol-level tap without a system proxy or VPN profile, which is the operator’s decision
cookies / localStorage / IndexedDB / Cache API	refuses: web storage. An app’s data lives in its sandbox. Use app_reset
frames_list, the CDP-deep family, pdf_save	refuses: deep:false, no Page, no print surface

Why gesture_pinch refuses. adb shell input has no two-finger primitive. Two concurrent input swipe calls produce two independent single-pointer streams, which an app sees as two unrelated drags and not a pinch; sendevent against /dev/input needs the touchscreen’s device node and ABS ranges read per device and is root-only off an emulator. Real multi-touch needs a device-side UiAutomator2 instrumentation server, which browxai does not ship. A refusal that names this is worth more than a gesture that reports success and did something else.

No video yet. Android’s screenrecord stops at 180 seconds, so a QA session needs the segmented writer RFC 0008 §5 designs. That lands with the native capture path.

Device lifecycle

These take no session. They are what you call before opening one.

• device_list(): every device and emulator adb sees, plus the AVDs defined on this machine, plus every iOS simulator simctl knows about. Each Android device reports {serial, state, model, release, sdk, emulator}; a device that is unauthorized or offline is listed with that state, because the state is the actionable part. Each simulator reports {id, name, state, runtime, platform}. An absent toolchain on either side is an empty list for that platform, never a failed call. Returns { ok, devices, avds, simulators, ready, simulatorsBooted }.
• device_boot({ avd, headless? }): boot an Android emulator and wait until it is usable, sys.boot_completed plus a working package manager, which an adb connection alone does not give (adb wait-for-device returns about a minute before the launcher exists). Headless by default; boots with -no-snapshot-save, so nothing this session installs or grants persists into the operator’s AVD. 30-90 seconds on a cold boot. Returns { ok, device }. There is no iOS counterpart: an ios-app session boots the simulator it resolves at session creation.
• device_shutdown({ serial }): emu kill on an Android emulator. Refuses on a physical device, browxai does not power off the operator’s phone. Closing a session does not shut a device down; a session leases a device, it does not own it. There is no iOS counterpart, for the same reason: an ios-app session never shuts down a simulator it may not have booted.

App lifecycle

These take a session and drive the device it leased. They are ONE surface over both engines: each verb routes through the session handle’s NativeLifecycle, which the adapter implements over adb or simctl. A verb the platform has no primitive for refuses, naming the engine and what is missing. It never answers a plausible empty.

tool	android-app	ios-app
app_list({ includeSystem?, session? })	installed package ids, third-party only by default	simctl listapps; includeSystem has nothing to select on
app_install({ apkPath, reinstall?, session? })	adb install, reinstall keeps existing data	simctl install a .app; it already replaces in place
app_uninstall({ appId, session? })	removes the app and its data	refuses. The session leases the device, it does not own it
app_launch({ appId, session? })	resolves the declared launcher activity first, returns it	simctl launch; no activity exists and none is invented
app_terminate({ appId, session? })	am force-stop	simctl terminate
app_reset({ appId, session? })	clears data and granted runtime permissions	refuses, iOS has no per-app clear; erase wipes every app
app_foreground({ session? })	{appId, activity} plus the app://<package>/<activity> url	{appId} plus the app://<bundle-id>/ url

• app_install is the broadest thing these engines do. The bundle is arbitrary code that then runs on the device. apkPath is workspace-relative and must resolve inside $BROWX_WORKSPACE, symlinks included, like every file-reading tool; a path starting with - or outside the workspace is refused with one message that does not say whether the path exists. On Android it reports adb’s own failure reason (wrong ABI, insufficient storage, unsigned build) alongside the non-zero exit.
• app_launch may return while the app is still on a splash screen. On android-app, follow with snapshot, which surfaces native-hierarchy-not-idle while an indeterminate spinner keeps the window busy.
• app_reset is destructive where it works: logins, local databases and cached files are gone. It is the native answer to incognito, which a native app has no equivalent of.
• app_foreground answers null during a window transition, which is a real state. It returns { ok, device, foreground, url, appUnderTest }, appUnderTest is what the SESSION thinks it is driving, which can differ from what is frontmost when a permission dialog or a share sheet takes focus.

Limits worth knowing before you plan a run

• One UiAutomator owner per device. A session leases a serial; a second session on the same serial gets a structured refusal naming the holder. Two UiAutomator clients on one device make every hierarchy dump fail for both, so this is a correctness gate. Set BROWX_ANDROID_APP_SERIAL to pick a device when several are attached. The engine refuses an ambiguous pick.
• uiautomator dump blocks on window idle and gives up at ~10s. An app with a running animation or an indeterminate progress spinner, a React Native splash screen reproduces this exactly, never goes idle, and snapshot surfaces native-hierarchy-not-idle. Wait for the screen to settle and retry. This is a UiAutomator limitation, not a browxai timeout.
• A dump costs a few hundred milliseconds, and every action pays for one. That is the price of never replaying a cached coordinate.
• Registered secrets do not materialise on either native engine. Secret substitution lives in the Playwright action core, which a native session never reaches, so a <NAME> alias is typed literally. fill says so in its warnings.
• Real devices are out of scope by policy. An adb devices entry is an adb devices entry.