Most AI crawlers read HTML only, never running JavaScript
11/15/2026 — 11/15, 23:17·1 sources (all)·1 reports (all)
Key takeaways
- A tutorial explains that most newer AI crawlers fetch only HTML and do not run JavaScript, so script-rendered pricing tables, comment widget…
Background & analysis
On November 15, 2026, at 23:01, DEV Community's AI section published an article titled "What GPTBot Actually Gets From a Page That Looks Fine in Chrome," examining what AI crawlers actually retrieve when they request a web page. The article argues that most newer AI crawlers only fetch HTML and do not run JavaScript, so content rendered by scripts — pricing tables, review components, and client-side tab panels — never appears in the text they read. For many sites, the honest answer to what a crawler sees is "much less than you think."
The piece notes one exception: Google's own AI features inherit its rendering pipeline. Googlebot fetches the HTML, executes the JavaScript in a headless browser, and then indexes the rendered result, so script-driven content remains visible. To support the point, the author cites Vercel's end-of-2024 observation of crawler traffic across the web, which showed how such crawlers actually behave at scale.
The article closes by offering a small script that lets site operators check their own pages in about one minute, so they can compare what a crawler reads against what appears in Chrome. In the material available, this is the only report on the topic, so the discussion currently rests on the technical phenomenon described and the self-check method proposed, with no further developments on record.
AI-generated from 1 reports · updated 2 days ago
- Heat index
- 34
- All sources
- 1
- All reports
- 1
- First seen
- 2 days ago
Reports on this story headlines open the original
Most newer AI crawlers fetch HTML but never execute JavaScript, so pricing tables loaded from an API, review widgets and client-side tabs simply are not in what they read; Google's own AI features are the exception because they ride Google's rendering pipeline. The author cites Vercel's late-2024 observations of crawler traffic and includes a short script for checking your own pages.
DEV Community · AIAI score 60
Other stories people are talking about
- Heat index 1336RisingNvidia board not discussing Jensen Huang's return, probe into his conduct advances22 sources
- Heat index 1232RisingCalifornia Sues OpenAI Over Agents Escaping Tests, Intruding into Hugging Face15 sources
- Heat index 894Alibaba releases MoE model Qwen4-Max with 1M token context for low-cost, long-running tasks17 sources
- Heat index 665US Commerce Department issues interim final rule restricting remote access to AI chips via overseas data centers10 sources
- Heat index 528Anthropic to begin trading on Nasdaq this week under ticker ANT9 sources
How is heat calculated?About the methodHide
Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.
This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.
- Surge
- Discussion rising fast
- New
- First report within 6 hours
- Rising
- Still gathering discussion

