Analysis: Blocking GPTBot Won't Remove You From AI Search
11/06/2026 — 11/06, 21:15·1 sources (all)·1 reports (all)
Key takeaways
- An analysis splits AI crawlers into two groups: training bots like GPTBot and ClaudeBot that scrape pages for model training, and citation b…
Background & analysis
On November 6, 2026, an analysis published on DEV Community argued that blocking GPTBot does not keep a website out of ChatGPT search results. The author sorts AI crawlers into two groups. Training crawlers such as GPTBot and ClaudeBot collect pages to train models. Search and citation crawlers, namely OAI-SearchBot, ChatGPT-User, PerplexityBot, and Perplexity-User, fetch a page in real time to answer a user right now and name the site as the source. According to the piece, blocking a training crawler does not remove a page from AI search answers, whereas blocking a citation crawler does. The author also notes that an earlier audit by the same team reached the opposite conclusion, that audit was retracted in writing, and the scoring was rebuilt. The article offers self-test methods for distinguishing each crawler type. Only this single report has appeared so far, and no independent verification or further development has followed.
AI-generated from 1 reports · updated 2 hours ago
Related AI tools
- Heat index
- 93
- All sources
- 1
- All reports
- 1
- First seen
- 3 hours ago
Reports on this story headlines open the original
AI crawlers come in two kinds, the authors write: training crawlers such as GPTBot and ClaudeBot, and search/citation crawlers such as OAI-SearchBot and PerplexityBot that fetch a page to answer a query and credit the source. Blocking a training crawler does not remove a site from AI search answers; blocking the citation crawler does — a conclusion they reached after withdrawing an earlier audit that said the opposite.
DEV Community · AIAI score 67
Other stories people are talking about
- Heat index 136Man sues OpenAI, alleging ChatGPT helped his wife plan poisoning2 sources
- Heat index 52Australian Universities Expand ChatGPT Access to Foster AI Talent1 sources
- Heat index 991OpenAI rolls out GPT-6 Cyber to Daybreak Red users with API, Codex access16 sources
- Heat index 826RisingClaude rolls out Personal Finance to US Pro, Max and Ultra users via read-only Plaid links13 sources
- Heat index 681SurgeThinking Machines Lab releases Inkling 2 Small amid reported $5B funding round9 sources
How is heat calculated?About the methodHide
Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.
This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.
- Surge
- Discussion rising fast
- New
- First report within 6 hours
- Rising
- Still gathering discussion