How AI Crawlers Read Your HTML: Headings, Landmarks and Lists

AI crawlers read the HTML your server sends, not the page your browser paints. What matters most is whether your text is in that HTML at all, because the AI crawlers that have been tested don’t run JavaScript. After that, ordinary semantic structure helps any parser find the answer and keep facts together: one H1, headings that split the page into real sections, lists and tables where the content is a list or a table, and the main content inside <main> or <article>. Google calls this a good idea rather than a requirement, and nothing shows that question-shaped headings or special “AI formatting” earn citations.

What Google actually says

While it's not required to have perfectly semantic HTML (the web in general is not valid HTML, and Google can understand it), it's generally a good idea to try to use semantic HTML when possible, as it helps other types of users, such as screen readers, parse and navigate your web page more easily.

The same guide asks for pages “organized by paragraphs and sections, along with headings that provide a clear structure to navigate content”, says “there's no requirement to break your content into tiny pieces for AI”, and says you don’t need to write in a specific way for generative AI search. Google also renders JavaScript. So for Google, semantic HTML is good practice, not a lever.

The stronger reason to care is everyone else. ChatGPT, Claude and Perplexity send their own crawlers, which don’t render the page, and every answer engine has to work out which part of your page is the content and where one section ends and the next begins.

Raw HTML is the real test

Vercel and MERJ analysed crawler traffic across Vercel’s network and found that none of the major AI crawlers rendered JavaScript: OpenAI’s OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic’s ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. ChatGPT and Claude did fetch JavaScript files (11.5% and 23.8% of their requests) but didn’t execute them. Gemini, which uses Google’s infrastructure, did render. Content that is already in the initial HTML, including JSON in the page, may still be read.

SEO·AEO·GEO’s Raw HTML visibility check is worth 15 points and can cap the whole score: it counts the words in the HTML with scripts removed, and fails pages with fewer than 50 words plus signs of client-side rendering. Speed and redirects can stop the same crawler before it reads anything; that side is covered in what actually stops an AI crawler’s fetch.

An outline any parser can follow

1. Main content first: wrap the story in <main> and <article>, and keep navigation, footers and sidebars outside it. Google’s guide asks for pages where it is easy “to distinguish your main content from other elements on the page.”
2. One H1 that names the page’s topic and matches the <title> (the canonicals and metadata post covers keeping those aligned).
3. H2s for sections and H3s for sub-steps. Don’t skip levels for styling.
4. The first paragraph under the H1 answers the page’s main question in plain words.
5. Lists for steps and sets, tables for comparisons, <blockquote> with a named source for quotes, and links to primary sources for claims.

Why lists and tables? When a system lifts one passage out of a page, a table row still carries its column headers and a numbered step still sits in order. A run of styled divs carries only the words. That is reasoning from how extraction works rather than a measured citation effect; the measured effects are about position and section length, below.

What the data does and doesn’t show

• Position: Kevin Indig’s analysis of 1.2 million ChatGPT results (Growth Memo, February 2026) found about 44% of citations came from the first 30% of the page. Put the answer at the top.
• Section length: in SE Ranking’s dataset of 216,524 pages, sections of 120–180 words between headings averaged 4.6 ChatGPT citations, against 2.7 for sections under 50 words. Don’t chop pages into fragments.
• Question headings: in the same dataset, question-style titles and H1s averaged 3.4 citations against 4.3 for straightforward ones, and pages with FAQ sections 3.8 against 4.1. SE Ranking reads this as page type rather than format. Either way, nothing shows the format helps.
• All three are correlational, and AI answers vary from run to run. Treat them as direction, not formulas.

That is why SEO·AEO·GEO’s model 4.0 replaced its old question-block check with a single Answer first check, and reports passage length without scoring it. The methodology explains what we stopped rewarding and why.

Div soup vs a readable document

Structure doesn’t replace the visible facts either. A fact that exists only in JSON-LD may not be read at all: in one small test, ChatGPT and Gemini returned prices from the visible HTML but not a price that was only in the markup. The JSON-LD post covers what schema still does.

A one-hour semantics checklist

• View source or fetch the page with curl. Can you see the H1 and the first answer without JavaScript?
• One H1, then H2s, then H3s. No skipped levels for decoration.
• Wrap the story in <main> or <article>, with navigation and footer outside it.
• Give every H2 a stable id so readers and other sites can link to a section.
• Replace screenshot-only tables and diagrams with HTML tables, or SVG with a text caption.
• Write headings that describe the section. Don’t rewrite them as questions for AI.

Run a free audit to see Raw HTML visibility, Semantic HTML, Heading depth, H1 and Answer first for any page. If the crawler can’t reach the page in the first place, start with robots.txt and CDN access.