Browser Agents: Capabilities and Pitfalls

Browser agents live in a headless (or headful) Chromium and operate via a mix of DOM inspection and screen reasoning. They're narrower than general Computer Use agents — only the browser — which makes them faster, cheaper, and more reliable on web tasks.

Browser Use (OSS) in 25 lines

from browser_use import Agent
from langchain_anthropic import ChatAnthropic
import asyncio

async def main():
    llm = ChatAnthropic(model="claude-sonnet-4-6")
    agent = Agent(
        task=(
            "Go to arxiv.org. Search for papers on 'prompt injection' "
            "from 2026. Return the top 3 titles with URLs."
        ),
        llm=llm,
        use_vision=True,
        max_actions_per_step=5,
    )
    result = await agent.run(max_steps=20)
    print(result.final_result())

asyncio.run(main())

How browser agents 'see' pages

• DOM tree + accessibility labels (primary) — fast, precise, machine-readable.
• Screenshot + vision (fallback) — works on canvas, SVG-heavy, or JS-rendered content.
• Network requests + console (supplementary) — detect errors, captured forms.
• Element IDs assigned by the agent harness — stable references for 'click element 47'.

Where they break

Ethical and legal traps

• Most sites' Terms of Service forbid automation. Check before running at scale.
• 'Scraping' is legally contested — hiQ v. LinkedIn and Meta rulings are still evolving.
• Ad click fraud, fake account creation, and purchasing bots are likely illegal in your jurisdiction regardless of the technical capability.
• Accessibility nuance: blind users have used similar tools for years. Don't paint all automation as 'bot abuse.'

Production hardening

• Use managed browser infra (Browserbase, Anchor) for stable IPs and bot fingerprint management.
• Record videos of agent runs (most libs support it) — debugging becomes trivial.
• Assert before acting: 'I see a submit button — read its text to confirm before click.'
• Budget caps per task: max actions, max minutes, max dollars.
• Dead-man switch: if no progress for N steps, surface to human.

Next lesson: how we actually measure any of this — benchmarks, evals, and their well-documented failures.