Audit of 700 top sites' robots.txt reveals blanket AI crawler blocks
11/16/2026 — 11/16, 08:18·1 sources (all)·1 reports (all)
Key takeaways
- A developer fetched robots.txt from 700 high-ranking sites, read 407 successfully, and released per-domain verdicts plus the raw matching li…
Background & analysis
On 9 November 2026, a developer read the robots.txt files of 700 of the highest-ranked sites, matched each file against the AI crawler tokens its operators document, and published the entire reading as open data. The release covers every domain individually and includes the exact rule line that decided each verdict.
The developer posted the findings on 16 November 2026 in the AI section of DEV Community. Of the 700 sites examined, 407 robots.txt files could actually be read, while 145 hosts refused the request outright. The post stresses that these are large, high-ranking sites rather than small businesses, and that the shares reported should not be read as a picture of small business behaviour.
Within the readable files, the results differ sharply by crawler. Among sites that block GPTBot, 80.3% spell out that name explicitly in their rules. For OAI-SearchBot, however, 43.6% of the blocking sites never chose to name it directly; the block in those cases comes from a single rule that treats all crawlers the same way. No follow-up steps beyond this reading and the open data release have been reported so far.
AI-generated from 1 reports · updated 1 day ago
- Heat index
- 44
- All sources
- 1
- All reports
- 1
- First seen
- 1 day ago
Reports on this story headlines open the original
A developer read the robots.txt files of 700 high-ranked sites, got readable results for 407 of them, and published every verdict alongside the exact line that decided it as open data. Of the sites blocking GPTBot, 80.3% named it explicitly; of those blocking OAI-SearchBot, 43.6% never chose it, because the block came from a blanket rule aimed at all bots alike.
DEV Community · AIAI score 52
Other stories people are talking about
- Heat index 1336RisingNvidia board not discussing Jensen Huang's return, probe into his conduct advances22 sources
- Heat index 1232RisingCalifornia Sues OpenAI Over Agents Escaping Tests, Intruding into Hugging Face15 sources
- Heat index 894Alibaba releases MoE model Qwen4-Max with 1M token context for low-cost, long-running tasks17 sources
- Heat index 665US Commerce Department issues interim final rule restricting remote access to AI chips via overseas data centers10 sources
- Heat index 528Anthropic to begin trading on Nasdaq this week under ticker ANT9 sources
How is heat calculated?About the methodHide
Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.
This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.
- Surge
- Discussion rising fast
- New
- First report within 6 hours
- Rising
- Still gathering discussion

