AI Crawlers and Page Speed: Timeouts, Redirects and Raw HTML

Page speed is a weak factor in AI visibility. The largest study we know of, about 107,000 pages appearing in Google AI Overviews and AI Mode, found only weak correlations with Core Web Vitals, concentrated among the slowest pages. What actually stops an AI crawler from reading your page is more basic: a CDN or firewall rule that refuses it, content that only appears after JavaScript runs, redirects that end on an error, or an origin so slow the request times out. Fix those first and treat speed as hygiene, not a citation lever.

What the speed evidence says

Search Engine Land’s January 2026 analysis tested the common assumption that good Core Web Vitals should go with strong AI visibility. Across roughly 107,000 pages in AI Overviews and AI Mode, it found no reliable advantage for pages that were already acceptable; the weak relationship that did exist came from extremely slow outliers. Our methodology grades load time “weak” for that reason and gives it 3 of the 67 Retrieval points.

SE Ranking’s ChatGPT dataset points the other way: pages with a First Contentful Paint under 0.4 seconds averaged 6.7 citations, against 2.1 for pages slower than 1.13 seconds. But First Contentful Paint is a browser metric no AI crawler measures, and fast sites tend to be well-funded sites that also do everything else well. Both findings are correlational. Neither supports treating speed as a citation lever.

What actually stops a fetch

1. A CDN, WAF or bot rule refuses the request with a 403, a 402 (Cloudflare’s pay per crawl) or a challenge page. Cloudflare changed its default for new domains on 1 July 2025 and added separate Search, Agent and Training settings in 2026; the robots.txt and CDN guide covers what to allow.
2. robots.txt disallows the crawler, or robots.txt itself fails. Under RFC 9309, a robots.txt that is unreachable because of a server error means the crawler “MUST assume complete disallow”, so a 5xx on that one file closes the whole site to compliant crawlers.
3. The content isn’t in the raw HTML. Vercel and MERJ found none of the major AI crawlers rendered JavaScript; see how AI crawlers read your HTML.
4. Redirects end somewhere broken. The same Vercel data showed ChatGPT’s crawler spending 34.82% of its fetches on 404 pages and 14.36% following redirects, against 8.22% and 1.49% for Googlebot.
5. The origin is too slow. No AI vendor publishes its timeout, so the only safe answer is a fast one: cache the HTML at the edge and keep server work before the first byte small.

llms.txt is not in the chain

Compliant crawlers read robots.txt before crawling a site, and sitemaps help them discover URLs. llms.txt plays no part in either. Ahrefs analysed the server logs of 137,000 domains: 97% of llms.txt files received zero requests in May 2026, and no AI bot ever requested an llms.txt file that didn’t exist. Google’s generative-AI guide (May 2026, updated July 2026) says Google Search doesn’t use these files and that keeping one “will neither harm nor help” your visibility in Google Search.

So an llms.txt file can’t slow your pages down or speed them up for AI crawlers. If coding agents are pointed at your developer docs, it may be useful to them; otherwise leave it alone. SEO·AEO·GEO reports whether you have one and scores it zero either way.

How SEO·AEO·GEO measures this

The engine requests the page, robots.txt and llms.txt in parallel, then runs the crawler-identity probes and the remaining lookups alongside each other. The whole audit has a 20-second deadline, set by our hosting runtime rather than by any crawler, and the page request follows up to five redirects, checking each hop. Those are properties of our auditor, not published crawler limits.

• Load time (Retrieval, 3 points): how long the HTML took to arrive, measured server to server, not a browser render. Under one second passes, one to three seconds scores partly, three seconds or more warns.
• Edge access (Retrieval, 4 points): the page requested as OAI-SearchBot, PerplexityBot and Claude-SearchBot, compared with a browser request. Simulated, because real crawlers are verified by IP.
• Raw HTML visibility (Retrieval, 15 points, can cap the score): the words in the HTML before any JavaScript runs.
• robots.txt AI rules (Retrieval, 10 points, can cap the score): including a robots.txt that answers with a server error.

Each finding in the report carries what was measured, such as the HTML delivery time in milliseconds or the HTTP status each simulated crawler received; how to read and export an SEO·AEO·GEO report explains the fields.

Fixes that change whether a crawler gets the page

• Allow verified AI search crawlers at the CDN or WAF, then confirm in your logs that they receive 200 responses.
• Server-render the main content so it is complete in the first HTML response.
• Redirect old URLs in one hop to the final HTTPS address, and point canonicals and internal links straight at it (see canonical URLs and Open Graph).
• Make robots.txt answer 200 with your rules, or 404 if you have none. Never a 5xx.
• Cache HTML at the edge so the first byte arrives quickly for any client.
• Log AI user agents separately. A bot challenge shows up as 403s, or response times far above what a normal visitor gets.

Then keep improving Core Web Vitals for the people who visit. Run a free audit to see Load time, Edge access and Raw HTML visibility for any page.