HaiAI123

Curated Global AI Tools Directory

Industry update
Flat

Audit of 700 top sites' robots.txt reveals blanket AI crawler blocks

11/16/2026 — 11/16, 08:18·1 sources (all)·1 reports (all)

Key takeaways

  • A developer fetched robots.txt from 700 high-ranking sites, read 407 successfully, and released per-domain verdicts plus the raw matching li…

Background & analysis

On 9 November 2026, a developer read the robots.txt files of 700 of the highest-ranked sites, matched each file against the AI crawler tokens its operators document, and published the entire reading as open data. The release covers every domain individually and includes the exact rule line that decided each verdict.

The developer posted the findings on 16 November 2026 in the AI section of DEV Community. Of the 700 sites examined, 407 robots.txt files could actually be read, while 145 hosts refused the request outright. The post stresses that these are large, high-ranking sites rather than small businesses, and that the shares reported should not be read as a picture of small business behaviour.

Within the readable files, the results differ sharply by crawler. Among sites that block GPTBot, 80.3% spell out that name explicitly in their rules. For OAI-SearchBot, however, 43.6% of the blocking sites never chose to name it directly; the block in those cases comes from a single rule that treats all crawlers the same way. No follow-up steps beyond this reading and the open data release have been reported so far.

AI-generated from 1 reports · updated 1 day ago

updream
24-hour heatHeat index 44 · peak 87 · 23h ago
24 hours agonow

Reports on this story headlines open the original

Yesterday
  1. A developer read the robots.txt files of 700 high-ranked sites, got readable results for 407 of them, and published every verdict alongside the exact line that decided it as open data. Of the sites blocking GPTBot, 80.3% named it explicitly; of those blocking OAI-SearchBot, 43.6% never chose it, because the block came from a blanket rule aimed at all bots alike.

    DEV Community · AIAI score 52

Other stories people are talking about

How is heat calculated?About the method

Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.

This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.

Surge
Discussion rising fast
New
First report within 6 hours
Rising
Still gathering discussion

Back to AI industry updates →