By Zach Wennstedt, founder of Eye To Ad Media · 2026-09-29
Most AI crawlers read the raw HTML a server returns: text, headings, links, metadata and JSON-LD. Many do not run JavaScript, ignore visual styling, and cannot read text baked into images. Content that is present, structured and specific in the initial HTML is what they can use.
Open your website on your phone. Now imagine seeing it with the images gone, the colors gone, the animations gone, and every piece of JavaScript switched off.
What is left? That is roughly what many AI crawlers see.
What they fetch
When a bot like OAI-SearchBot, ClaudeBot or PerplexityBot requests your page, it receives the same HTML your server sends a browser. The difference is what happens next. A browser builds the page, runs scripts and paints pixels. Many crawlers just parse the document.
They read the title, meta description, headings, paragraphs, lists, tables, link text and URLs, image alt attributes, and any JSON-LD structured data in script tags. That is the raw material an answer engine can work with.
What they often skip
- Content injected by JavaScript. If your prices, FAQs or service details only appear after scripts run, some crawlers never see them. Google renders JavaScript, but in a separate, later step.
- Text inside images. That beautiful infographic with your process? To a text crawler it is one alt attribute, if you wrote one.
- Content behind interactions. Tabs and accordions are usually fine when the text is in the HTML. Content that loads only on click is not.
- Anything blocked by robots.txt. If the crawler is disallowed, it reads nothing.
What they are looking for
An answer engine is trying to answer a question with confidence. It prefers passages that are specific, self-contained and attributable. "Drain cleaning in Denver typically takes about an hour" is a sentence it can use. "We deliver excellence with passion" is not.
Five changes that help
- Put the answer first. Open key pages with a one or two sentence direct answer to the question the page targets.
- Ship content in the HTML. Server-render or pre-render important pages. Check with View Page Source.
- Use real structure. One H1, logical H2s and H3s, lists for lists, tables for comparisons.
- Add matching JSON-LD. Describe the business, the page and its FAQs in structured data that matches what is visible.
- Decide your robots.txt on purpose. Know which AI crawlers you allow. Test it with our AI crawler checker.
Then score a page with the machine-readability analyzer and see where you stand.