Browser-agent benchmark round 2: 100 real tasks, recovery matters
11/17/2026 — 11/17, 09:15·1 sources (all)·1 reports (all)
Key takeaways
- Round two of a browser-agent benchmark ran six local setups on 100 real-site Online-Mind2Web tasks, adding BrowserOS and Grok Bot under a sh…
Background & analysis
On 2026-11-17 at 09:04, the AI section of DEV Community published results from the second round of browser agent testing. The round reused the same 100 Online-Mind2Web tasks drawn from real websites, running them on live sites rather than in a sandboxed environment. The pool of local setups grew from five in round 1 to six, with BrowserOS and Grok Bot added to the field. All six local configurations were run on the same datacenter server, using Claude Haiku 5.5 with headed browsers, a 10-minute time limit per task, and a single attempt, and scoring was handled by the benchmark's own judge, WebJudge, which evaluated each run based on recorded actions and screenshots.
The findings show that the leading browser agents still complete roughly the same share of live tasks, with the gap among the top three not statistically significant. For example, the comparison between BrowserOS and 5dive yielded a p value of 0.29. What actually separated the setups was the way they recovered after an ordinary click failed. BrowserOS responded with 94 clicks at screen coordinates and 216 in-page JavaScript calls, and its tool-call failure rate stood at 6 percent. The author's own browser, by contrast, recorded a failure rate of 19 percent. The material available so far covers only this round's conditions and the resulting comparison, with no details on any future rounds or follow-up testing.
AI-generated from 1 reports · updated 4 hours ago
- Heat index
- 90
- All sources
- 1
- All reports
- 1
- First seen
- 4 hours ago
Reports on this story headlines open the original
Round 2 ran the same 100 Online-Mind2Web live-site tasks across six local setups, adding BrowserOS and Grok Bot, all on Claude Haiku 5.5 with headed browsers, a 10-minute cap and one attempt, judged by WebJudge from actions and screenshots. Completion rates for the top agents remain roughly equal, with the top-three gaps not statistically significant (BrowserOS vs 5dive, p = 0.29). What separated them was recovery when a normal click fails: BrowserOS used 94 clicks by screen position and 216 in-page JavaScript calls, with a 6% tool-call failure rate versus 19% for the author's own browser.
DEV Community · AIAI score 52
Other stories people are talking about
- Heat index 1336RisingNvidia board not discussing Jensen Huang's return, probe into his conduct advances22 sources
- Heat index 1232RisingCalifornia Sues OpenAI Over Agents Escaping Tests, Intruding into Hugging Face15 sources
- Heat index 894Alibaba releases MoE model Qwen4-Max with 1M token context for low-cost, long-running tasks17 sources
- Heat index 665US Commerce Department issues interim final rule restricting remote access to AI chips via overseas data centers10 sources
- Heat index 528Anthropic to begin trading on Nasdaq this week under ticker ANT9 sources
How is heat calculated?About the methodHide
Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.
This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.
- Surge
- Discussion rising fast
- New
- First report within 6 hours
- Rising
- Still gathering discussion

