LLM SEO: how ChatGPT, Claude and Perplexity find your pages

LLM SEO is the work of making your pages easy for language model products such as ChatGPT, Claude, Perplexity and Gemini to fetch, read and cite when they answer with web search. In practice that means letting each company's search crawler in, serving the important text in plain HTML and writing passages that settle one question each, then checking again whenever your site or the engines change.

Each AI company sends more than one bot

OpenAI, Anthropic and Perplexity each document one agent for search and another for pages a user asks the assistant to open, and OpenAI and Anthropic add a training crawler. Turning away a training crawler does not have to cost you search visibility.

OpenAI says a site that opts out of OAI-SearchBot is not shown in ChatGPT search answers, and that a robots.txt change takes about a day to register. Anthropic's crawler page warns that blocking Claude-SearchBot may reduce how visible a site is in Claude's search results. Perplexity says PerplexityBot is not used to collect training data, and that Perplexity-User, which acts for a person, generally ignores robots.txt.

Gate one: can the crawler read the page?

Robots.txt is the first lock, and a firewall or CDN bot rule can be a second one that the file never mentions. JavaScript is the third. In a December 2024 study by Vercel and MERJ, the ChatGPT and Claude crawlers downloaded script files but never ran them, while Gemini, which uses Googlebot's infrastructure, rendered pages in full. A spec table or FAQ that only appears once scripts run may look like an empty page to those two crawlers.

Gate two: is there a passage worth quoting?

Google says its AI features use query fan-out, sending several related searches at once and pulling results for each. A section that opens by answering one of those narrower questions in a single sentence gives a model something it can lift whole. Google also warns that building a separate page for every fan-out variation to game rankings breaks its scaled content abuse policy, so answer the questions on the pages you already have.

Keep watch after you ship

LLM SEO results drift. Models update, the sources they prefer shift, and one deploy can quietly undo last month's fix. A light routine helps:

1. After any robots.txt, CDN or template change, rerun the crawler access and rendering checks.
2. Once a month, put your buyers' main questions to the assistants and note which brands and pages come back.
3. Read the reports you already have: Search Console's Generative AI performance report for AI Overviews and AI Mode impressions, and the AI Performance report in Bing Webmaster Tools for citations in Copilot.
4. If you can export server logs, look at what the AI crawlers actually requested, not just what you allowed.

Running the routine with seogeoaeo.ai

seogeoaeo.ai is the easiest way to run that routine on demand, since each check is a single run priced in credits and the same checks work from Claude Code, Cursor or Codex over MCP. The Robots.txt & AI Crawler Checker, 10 credits, shows which AI bots your rules let through. AI Crawler View, 8 credits, shows what vanishes without JavaScript. The AI Bot Log Analyzer, 8 credits, reads a pasted access log to show what GPTBot, ClaudeBot, PerplexityBot and ChatGPT-User fetched, and flags visits that borrow a crawler's name from an unlisted IP. AI Answer Visibility, 120 credits, samples ChatGPT, Perplexity and Gemini on a single question to see whether your brand comes up.

It runs when you ask and keeps no daily log of answers. If you need a chart of hundreds of prompts over time, a dedicated tracker such as OtterlyAI, which covers ChatGPT, AI Overviews, Perplexity, Gemini, Copilot and Claude, suits that job better. For the reasoning behind each step, see seogeoaeo.ai's longer LLM SEO write-up.