HaiAI123

Curated Global AI Tools Directory

Industry update
Flat

Most AI crawlers read HTML only, never running JavaScript

11/15/2026 — 11/15, 23:17·1 sources (all)·1 reports (all)

Key takeaways

  • A tutorial explains that most newer AI crawlers fetch only HTML and do not run JavaScript, so script-rendered pricing tables, comment widget…

Background & analysis

On November 15, 2026, at 23:01, DEV Community's AI section published an article titled "What GPTBot Actually Gets From a Page That Looks Fine in Chrome," examining what AI crawlers actually retrieve when they request a web page. The article argues that most newer AI crawlers only fetch HTML and do not run JavaScript, so content rendered by scripts — pricing tables, review components, and client-side tab panels — never appears in the text they read. For many sites, the honest answer to what a crawler sees is "much less than you think."

The piece notes one exception: Google's own AI features inherit its rendering pipeline. Googlebot fetches the HTML, executes the JavaScript in a headless browser, and then indexes the rendered result, so script-driven content remains visible. To support the point, the author cites Vercel's end-of-2024 observation of crawler traffic across the web, which showed how such crawlers actually behave at scale.

The article closes by offering a small script that lets site operators check their own pages in about one minute, so they can compare what a crawler reads against what appears in Chrome. In the material available, this is the only report on the topic, so the discussion currently rests on the technical phenomenon described and the self-check method proposed, with no further developments on record.

AI-generated from 1 reports · updated 2 days ago

updream
24-hour heatHeat index 34 · peak 67 · 23h ago
24 hours agonow

Reports on this story headlines open the original

Nov 15
  1. Most newer AI crawlers fetch HTML but never execute JavaScript, so pricing tables loaded from an API, review widgets and client-side tabs simply are not in what they read; Google's own AI features are the exception because they ride Google's rendering pipeline. The author cites Vercel's late-2024 observations of crawler traffic and includes a short script for checking your own pages.

    DEV Community · AIAI score 60

Other stories people are talking about

How is heat calculated?About the method

Heat counts how many independent sources covered a story in the last 48 hours: one source counts once no matter how many posts it published, decaying with a 24-hour half-life. What ranks first is what many people are talking about.

This page aggregates public feeds. Headlines and summaries are machine-organized and remain the property of the original authors; verify important facts at the source. If you believe a headline or summary infringes your rights, email the contact address in the footer with the page URL and basis of your claim, and we will remove or replace it after review.

Surge
Discussion rising fast
New
First report within 6 hours
Rising
Still gathering discussion

Back to AI industry updates →