Developer runs 180 calls comparing Claude and GPT, finds single runs unreliable
10/28/2026 — 10/28, 18:19·1 sources (all)·1 reports (all)
Key takeaways
- A developer on DEV Community benchmarked three models from the Claude and GPT families across 12 tasks, repeating each five times for 180 to…
Background & analysis
On October 28, 2026, a developer published a comparison post on DEV Community that put the Claude and GPT families head to head. The experiment covered 3 models and 12 tasks, with each task repeated 5 times, for a total of 180 API calls. Cost per call ranged from 0.0033 to 0.0265 US dollars.
The author's central claim is that a single run proves nothing about a model's stability. A model that survives 3 test queries without breaking anything is not thereby validated. It is simply unnoticed so far. Rather than cycling through different models, the author reversed the usual approach and asked the same question over and over, then looked for a pattern across the 180 calls. That pattern, he argues, would have stayed hidden under a single-run methodology.
The takeaway he draws is methodological: to evaluate Claude and GPT reliably you should rerun every task multiple times instead of trusting one result. The public record of this comparison currently stops at that conclusion, with the 12 tasks, 5 repetitions each, and 180 calls as the evidence base.
AI-generated from 1 reports · updated 10 hours ago
Related AI tools
- Heat index
- 75
- All sources
- 1
- All reports
- 1
- First seen
- 10 hours ago
Reports on this story headlines open the original
A developer compared three Claude and GPT models across 12 tasks, running each five times for 180 calls in total, with a single call costing between $0.0033 and $0.0265. The takeaway: one run proves little about how reliable a model is, and repeating the same task surfaces patterns a single run would hide.
DEV Community · AIAI score 53
Other stories people are talking about
- Heat index 80One prompt makes ChatGPT, Claude and Gemini drop boilerplate1 sources
- Heat index 48Cursor vs Claude Code: the difference is the interaction surface1 sources
- Heat index 49Anthropic engineers discuss Opus 5.5 and Claude Platform1 sources
- Heat index 103chrome-use lets Claude Code and Codex drive a signed-in Chrome2 sources
- Heat index 71Anthropic launches Claude Duet, an always-on proactive agent1 sources
How is heat calculated?About the methodHide
Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.
This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.
- Surge
- Discussion rising fast
- New
- First report within 6 hours
- Rising
- Still gathering discussion