Guide

Human-Agent Handoff Guide

Let humans solve CAPTCHAs, complete 2FA, or handle complex interactions while your AI agent waits.

The problem

AI agents hit walls: CAPTCHAs, two-factor authentication, "prove you're human" challenges, complex multi-step forms. These need a human to step in, solve the blocker, then hand control back to the agent.

How it works with BrowserFabric

BrowserFabric provides the building blocks for human-agent handoff:

1. Agent creates a session and navigates

import browserfabric

async with browserfabric.browser(persist=True) as session:
    await session.navigate("https://app.example.com/login")
    await session.type("#email", "user@example.com")
    await session.type("#password", "secret")
    await session.click("#submit")

    # Agent detects CAPTCHA
    summary = await session.page_summary()
    if "captcha" in summary["text_preview"].lower():
        # Hand off to human
        info = await session.session_info()
        print(f"Human needed! View: /session/{info['session_id']}")
        print(f"CDP: {info['ws_url']}")

2. Share the session URL

Send the session viewer URL to a human operator. They can see a live screenshot of the browser at /session/{id}. The viewer auto-refreshes every 5 seconds.

3. Human connects via CDP

For interactive control, the human connects to the session's CDP WebSocket URL using Chrome DevTools or a Playwright script:

# Human operator connects with Playwright
from playwright.sync_api import sync_playwright

pw = sync_playwright().start()
browser = pw.chromium.connect_over_cdp("ws://api:PORT")
page = browser.contexts[0].pages[0]
# Human solves CAPTCHA manually...
pw.stop()

4. Agent resumes

After the human solves the blocker, the agent polls the page state and continues:

# Agent waits for CAPTCHA to be solved
await session.wait_for("#dashboard", state="visible", timeout=300)
# Continue automation
await session.navigate("https://app.example.com/data")
data = await session.evaluate_js("...")

Using webhooks for notifications

Set up a webhook to notify your system when sessions need human attention:

# Set webhook URL
curl -X PUT https://api.browserfabric.com/api/v1/auth/webhook \
  -H "Authorization: Bearer bf_..." \
  -d '{"url": "https://your-app.com/webhook"}'

# Your webhook receives:
# {"event": "session.created", "data": {"session_id": "..."}}
# {"event": "session.closed", "data": {"session_id": "..."}}

Best practices

  • Use persist=True so the solved state is saved even if the session is interrupted
  • Set long timeouts on wait_for (5-10 minutes) to give humans time to respond
  • Use page_summary to detect blockers like CAPTCHAs or 2FA prompts
  • Save contexts after successful handoffs to avoid repeating the process
← Back to blog