Browser Agents: Capabilities and Pitfalls

Browser agents live in a headless (or headful) Chromium and operate via a mix of DOM inspection and screen reasoning. They're narrower than general Computer Use agents — only the browser — which makes them faster, cheaper, and more reliable on web tasks.

Browser Use (OSS) in 25 lines

from browser_use import Agent from langchain_anthropic import ChatAnthropic import asyncio async def main(): llm = ChatAnthropic(model="claude-sonnet-4-6") agent = Agent( task=( "Go to arxiv.org. Search for papers on 'prompt injection' " "from 2026. Return the top 3 titles with URLs." ), llm=llm, use_vision=True, max_actions_per_step=5, ) result = await agent.run(max_steps=20) print(result.final_result()) asyncio.run(main())

How browser agents 'see' pages

• DOM tree + accessibility labels (primary) — fast, precise, machine-readable.
• Screenshot + vision (fallback) — works on canvas, SVG-heavy, or JS-rendered content.
• Network requests + console (supplementary) — detect errors, captured forms.
• Element IDs assigned by the agent harness — stable references for 'click element 47'.

Where they break

Ethical and legal traps

• Most sites' Terms of Service forbid automation. Check before running at scale.
• 'Scraping' is legally contested — hiQ v. LinkedIn and Meta rulings are still evolving.
• Ad click fraud, fake account creation, and purchasing bots are likely illegal in your jurisdiction regardless of the technical capability.
• Accessibility nuance: blind users have used similar tools for years. Don't paint all automation as 'bot abuse.'

Production hardening

• Use managed browser infra (Browserbase, Anchor) for stable IPs and bot fingerprint management.
• Record videos of agent runs (most libs support it) — debugging becomes trivial.
• Assert before acting: 'I see a submit button — read its text to confirm before click.'
• Budget caps per task: max actions, max minutes, max dollars.
• Dead-man switch: if no progress for N steps, surface to human.

Next lesson: how we actually measure any of this — benchmarks, evals, and their well-documented failures.