DEV Community explains why robots.txt cannot stop AI crawlers
11/21/2026 — 11/21, 22:17·1 sources (all)·1 reports (all)
Key takeaways
- A DEV Community post explains that under RFC 9309, robots.txt is only a request that crawlers follow the rules, not an access grant or a tec…
Background & analysis
On November 21, 2026 at 22:01, the AI section of DEV Community published a piece titled "robots.txt 拦不住 AI 爬虫,网站还能做什么," examining what robots.txt can and cannot do when AI crawlers come calling. The article's central point, drawn from RFC 9309, the IETF specification for the Robots Exclusion Protocol, is that the rules in robots.txt are "not a form of access authorization." As the article puts it, a robots.txt file blocks no one; it is simply a text file full of polite requests, and the underlying standard says so in plain terms.
The piece argues this limitation matters more now than it used to, because sites are being read by several kinds of automated clients: AI crawlers that collect training data, search bots that influence whether a site appears in chatbot answers, and fetchers that pull pages on demand when a user poses a question. Given that situation, the article advises site owners to find out which crawlers actually follow the rules and to consider adding restrictions that go beyond the polite requests contained in robots.txt.
No concrete mitigation techniques, product recommendations, or follow-up developments are outlined in the available material, so the discussion currently stops at the level of this general recommendation, leaving website operators to decide on their own which additional measures to adopt.
AI-generated from 1 reports · updated 12 hours ago
- Heat index
- 70
- All sources
- 1
- All reports
- 1
- First seen
- 13 hours ago
Reports on this story headlines open the original
A DEV Community article explains that robots.txt is only a request under RFC 9309, not access authorization, so it cannot stop AI crawlers from fetching a site. Site owners need to know which crawlers cooperate and what additional controls to consider for training, search, and on-demand fetchers.
DEV Community · AIAI score 56
Other stories people are talking about
- Heat index 1143RisingAnthropic Reaches $1.6 Trillion Market Value in First Trading Week After IPO21 sources
- Heat index 1011Anthropic Mathematician Says Claude Formalized the Generalized Riemann Hypothesis21 sources
- Heat index 622Anthropic IPO total raise reaches $93.5 billion after greenshoe exercise20 sources
- Heat index 596RisingThinking Machines Lab Open-Sources Inkling 2 Mini with 32GB Model Card Recommendation10 sources
- Heat index 490RisingSEC probes Nvidia accounting practices as shares post seven-week slide9 sources
How is heat calculated?About the methodHide
Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.
This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.
- Surge
- Discussion rising fast
- New
- First report within 6 hours
- Rising
- Still gathering discussion

