Developer releases Python script to check robots.txt rules for AI crawlers

On November 3, 2026, a post appeared in DEV Community's AI section describing a Python script that uses only the standard library to check, for any given URL path, whether a site's robots.txt allows or blocks AI crawlers such as GPTBot and CCBot. The author explains that most robots.txt files grow by accretion: someone adds a GPTBot block in 2023, someone else adds a wildcard rule, and nobody can say for sure which AI crawlers can fetch which section of the site today. For people working on GEO or AEO, this distinction matters, because a blocked search crawler means pages cannot be retrieved for AI answers, while a blocked training crawler is often a deliberate choice. The script is meant to answer that question for any URL path, using only the standard library. The author also stresses that the script should be run separately on the homepage and on key sections, since the same crawler may be allowed on the homepage yet blocked in directories such as a blog.