Diagnosing and fixing GPTBot blocks by Cloudflare WAF rules

On October 4, 2026, DEV Community published a technical article titled "GPTBot blocked by Cloudflare WAF? How to detect it and fix it."

The article describes a sequence that will be familiar to many site owners. You launch a site, write a robots.txt file that welcomes AI search crawlers — "User-agent: GPTBot" followed by "Allow: /" — and move on. Weeks later, you look at your server access logs, or you start wondering why your content never turns up in ChatGPT Search or Perplexity. When you check, there are zero crawler hits.

According to the article, the culprit in these cases is almost never robots.txt itself. The more likely explanation is an edge security rule or a web application firewall (WAF) that blocks GPTBot before its requests ever reach the origin server. That, the piece argues, is why content can stay invisible to ChatGPT Search or Perplexity even when the crawler appears to be explicitly permitted in robots.txt.

The article sets out to help on two fronts: how to determine whether a crawler is being blocked in the first place, and how to allow it through once you have confirmed it is. Beyond that, only this single report is available so far. The material does not include any response from Cloudflare or OpenAI, nor any indication of how common the problem is.