# Agent browser interface

BrowserFabric provides five additional tools alongside the existing API:
`browser_tabs`, `observe_page`, `act`, `browser_screenshot`, and `cancel_action`.
They use the same authenticated REST `/api/v1/services/browseruse/call`
envelope and MCP endpoint. Existing selector tools remain supported.

## Observe, act, verify

Create a session and navigate as before. `observe_page` returns an explicit
`tab_id`, `observation_id`, page text, frame metadata, and element references
with role, accessible name, visibility and form state. It traverses open shadow
roots and frames. Custom cursor/onclick controls supplement semantic controls.
Closed shadow roots and inaccessible frames require visual fallback.

Input values are omitted. URLs omit query strings, fragments and userinfo;
API-key/JWT patterns and passwords entered through `act` are redacted from new
text observations. This is best-effort protection, not a guarantee that arbitrary
page text is free of sensitive data. Screenshots mask password inputs, but other
visible account information remains visible. Legacy raw DOM/JS tools retain their
existing behavior.

```python
state = await session.observe_page()
email = next(e["ref"] for e in state["elements"] if e["name"] == "Email")
submit = next(e["ref"] for e in state["elements"] if e["name"] == "Continue")
result = await session.act(
    state["tab_id"], state["observation_id"],
    [{"type": "fill", "ref": email, "text": "person@example.com"},
     {"type": "click", "ref": submit}],
    wait_for={"text": "Welcome"},
)
print(result["status"])
```

Python async and sync SDKs expose `tabs`, `observe_page`, `act`, and
`browser_screenshot`; TypeScript exposes `tabs`, `observePage`, `act`, and
`browserScreenshot`. The new `act` SDK methods preserve structured failures
including recovery observations instead of raising away that information.
Transport errors still raise. SDK `act` calls generate a request ID once and
reuse it on transport retries.

## Action contract

Every `act` requires an explicit tab and the latest observation ID for that tab.
Actions: `click` (ref or x/y), `fill` (ref/text), `check` (ref/checked),
`select` (ref/value), `press` (key, optional ref), and `scroll` (delta_x/delta_y).
Coordinates are CSS viewport pixels. Reobserve after layout changes; coordinates
do not have the identity guarantees of element references.

`wait_for` accepts exactly one of `text`, `url` (Playwright glob), or `selector`.
Text and selector conditions support visible/hidden/attached/detached states.
The default execution budget is 10 seconds; `timeout_ms` accepts 100–30000.
Observation and optional screenshot capture follow execution and can add latency.
Observation skips stalled frames after one second per frame, with a three-second
collection budget. Skipped frames are reported as `unavailable`; use visual fallback.
The legacy `wait_for` tool's `timeout` is in **seconds**, unlike `act.timeout_ms`.
Up to 20 actions are validated before execution. Do not combine this tool with
an outer batch when other clients may be acting on the same session.

| Status | Meaning |
|---|---|
| `completed` | Requested actions executed; no postcondition was supplied. |
| `verified` | Actions executed and the caller's condition matched. |
| `condition_not_met` | Actions executed, but the condition did not match within the budget. |
| `interrupted` | A navigation, popup or dialog stopped later steps. Check completed indices and events. |
| `stale_observation` / `stale_reference` | No initial actions executed against stale state; inspect the fresh observation. A ref can also become stale mid-batch: inspect completed indices. |
| `action_failed` | Execution failed or timed out. Earlier steps may have completed; inspect before retrying. |
| `request_id_conflict` | The ID was previously used with a different payload. |
| `outcome_unknown` | A prior request was interrupted before its outcome was recorded. Inspect the page. |

Native browser dialogs are dismissed and reported; they interrupt the batch.
A website can trigger navigation after the response; expected transitions should
have explicit postconditions. Browser automation cannot infer arbitrary business
success: choose a condition that actually proves your intended result.

## State, tabs and concurrency

Refs retain DOM node identity rather than reconstructing selectors. A replaced,
detached, renamed or navigated element fails validation. The latest observation
per tab is retained. Another client's observation on the same tab supersedes
yours. `since` requests a diff against that latest observation; unknown bases
return a full observation. Diffs contain `added`, `changed`, `removed` and changed
text. `truncated` indicates the element limit was reached (default 150, max 500).

The first new-interface call enables explicit-tab mode for that session. Popups
are registered without silently switching the selected live-view/legacy page.
`browser_tabs` operations are list/new/select/close. Closing the last tab is
rejected; close the session instead. Proxy routing by legacy `navigate` may
restart the browser, invalidating tab IDs and refs. Relist and observe afterward.

Service calls are serialized per session, including legacy tools, to avoid races
with routing and shutdown. External CDP clients and live-view human input are
outside that lock; always verify the state after handoff.

`cancel_action` interrupts the running service operation without closing the
browser. It returns `cancellation_requested` or `idle`; the interrupted request
returns `operation_cancelled` with `status: outcome_unknown`. SDK sessions expose
this as `cancel()`. Inspect current state: cancellation cannot undo a click that
already took effect. `close_session` interrupts a running operation before closing,
so it cannot be trapped behind a long legacy wait. Ownership is checked before
cancellation; another account cannot interrupt your session.

Request-ID deduplication retains the last 64 act results in the live session.
Identical concurrent requests execute once. Cancellation leaves an unknown-outcome
record rather than allowing automatic replay. IDs are not durable across process
restart or eviction. Replayed observations are historical, not current state.

## Images and compatibility

REST screenshots remain base64 PNG. MCP `browser_screenshot` and the existing
`take_screenshot` now return native image content with text metadata. MCP clients
that previously parsed screenshot JSON text must consume the image content block.
`act` optionally includes a screenshot (`always` or `on_error`); the MCP adapter
converts it to a native image and marks structured errors with `isError`.

## Verification

Install backend dependencies and Chromium, then run:

```bash
python -m playwright install chromium
python -m pytest backend/tests/test_agent_browser.py --asyncio-mode=auto
python backend/tests/benchmark_agent_browser.py
python backend/tests/benchmark_agent_browser_http.py --base-url http://localhost:9000
python backend/tests/smoke_agent_mcp.py --url http://localhost:9000/mcp
```

Tests use deterministic pages intercepted locally, including forms, delayed
navigation, popups, dialogs, frames, shadow roots, concurrency and cancellation.
The benchmark reports call count, JSON response bytes and local elapsed time;
it is not an LLM benchmark or a measurement of public-network latency.