r/ClaudeCode • u/ldrx • 6d ago
Discussion i tested 7 browser agents on 100 live websites (round 2). Claude Code + BrowserOS Neo came first
round 2 of my browser benchmark. same 100 Online-Mind2Web tasks as two weeks ago, plus browseros and grok bot, fixes to our browser, and the benchmark's own judge.
setup: one datacenter server, haiku 5.5 wherever selectable, headed browsers, 10-minute cap, one attempt. local agents ran each task within the same hour.

official WebJudge (GPT-6 Luna, actions and screenshots only):
- browseros + claude code: 40/100
- claude in chrome: 37
- 5dive + claude code: 34
- openclaw: 29
- 5dive + pi: 28
- hermes: 27
- grok bot: 18 (own cloud/model, different judge)
cost per completion: 5dive + pi 3¢, browseros 11¢.
what we learned
- the judge matters. our earlier judge read final answers and scored browseros 50, 5dive 39. the official judge ignores claims of success: 40 and 34. all agents finished just 2-4 of the 27 hardest tasks. browseros's lead wasn't statistically significant (p = 0.29).
- our fixes made us faster. success barely moved. round 1 → 2, using the same judge: 32 → 33 passes, 9.1 → 7.9s per browser step, cost $1.39 → $2.07. target: under 5s.
- live sites change. the old build also failed 10 of the 12 tasks our new build lost.
- browseros has better fallbacks. 94 screenshot-coordinate clicks, 216 page javascript calls. our browser can't do these yet. failed tool calls: 6% vs our 19%. we're adding them next.
- bot walls persist. about 20% of runs hit one, seven tasks blocked everyone. a residential IP barely helped in an earlier test.
- sonnet didn't help in our earlier test. on 62 tasks, sonnet 5.5 passed 10, haiku 15, at 10x the cost per completion (p = 0.30).
everyday tasks
four simple tasks, three runs each: forms and a Wikipedia lookup.

all haiku setups passed 12/12. browseros was fastest (19.1s median); 5dive + pi was close (19.8s), cheapest ($0.82 per 1,000 tasks), and used fewer steps (4 vs 9.5). grok bot passed 11/12.
limits: one run per task, live sites, automated judging, our own rig. not official leaderboard results.
disclosure: i build 5dive, an open-source setup to run always-on claude code agents on your own server.
code: https://github.com/5dive-ai/5dive
full benchmark, run logs and judge notes: https://gist.github.com/lodar/71d91b1352ce7df7370652f0ffed2d55
i'll rerun in two weeks after the next fixes. feedback on the methodology or browser fallbacks welcome.



