12 minute read

TL;DR: AI answer engines (ChatGPT, Perplexity, Claude, Gemini) sent about 1.08% of website traffic in Conductor’s 2025 benchmark, most of it from ChatGPT. In an August 2025 Ahrefs study, only 12% of URLs cited by ChatGPT, Gemini and Copilot ranked in Google’s top 10 for the same prompt. Structure measurably matters: GEO-SFE (arXiv 2603.29979) reports a 17.3% average citation gain from structural changes across six engines, and the Princeton GEO study reports visibility gains of up to 40%. PerplexityBot requests grew 157,490% between May 2024 and May 2025 (Cloudflare), from a tiny base. Most content is written for a search engine that rewards backlinks; AI visibility tracks brand mentions and structural clarity more closely.

Library index cards in a modular grid, some highlighted, connected to a central orb, representing structured content being indexed by AI crawlers


Search engine optimization rests on a foundational assumption: people type queries, see ranked blue links, click through. Google’s PageRank algorithm and everything built around it (backlinks, domain authority, keyword density) assume this model.

AI answer engines work differently. ChatGPT, Perplexity, Claude, and Gemini don’t return ranked links. They generate answers and cite sources directly. In an August 2025 Ahrefs study of about 15,000 long-tail prompts, only 12% of URLs cited by ChatGPT, Gemini and Copilot ranked in Google’s top 10 for the same prompt. Perplexity’s overlap was higher (28.6%), and Google’s own AI Overviews track rankings closely (a separate Ahrefs study found 76% of their citations come from top-10 pages). Outside Google, AI systems are sourcing content from a different signal set.

If you’ve been optimizing only for Google’s ranked results, you’ve been optimizing for one engine. The others are already crawling your site, and they cite content that can look different from what ranks.

The citation data

Traffic share: AI referrals made up about 1.08% of website traffic in Conductor’s 2025 benchmark of 13,770 domains, and ChatGPT sent 87.4% of that AI referral traffic. Small in absolute terms, but the quality signals look good: Semrush estimated that an AI search visitor is worth 4.4x a traditional organic visitor, based on conversion rates, and SE Ranking’s panel found AI visitors spend about 68% more time on site. Other datasets disagree on engagement, so check your own analytics.

Source independence: In the Ahrefs study, more than 80% of the URLs cited by ChatGPT, Gemini and Copilot did not rank for the target query at all. AI assistants run their own searches with reformulated queries, so they are not simply re-ranking Google results.

Citation position: Industry analyses of ChatGPT citations report that citations cluster near the top of a page. Get your answer out early rather than after a long preamble.

Brand signals, not backlinks: In an Ahrefs study of 75,000 brands (May 2025), branded web mentions correlated 0.664 with brand visibility in Google AI Overviews, versus 0.218 for backlinks. The signal that drives classic Google rankings (backlink authority) correlated far less with AI Overview visibility than being named by others in relevant contexts. Correlation is not causation, and the study measured AI Overviews, not ChatGPT.

What content structure AI actually cites

Two research papers carry most of the weight here. The GEO-SFE Framework (arXiv 2603.29979, “Structural Feature Engineering for Generative Engine Optimization”, March 2026) decomposes content into headings, paragraph chunking, lists, tables and sentence-level emphasis, then rewrites pages structurally while preserving meaning. Across six generative engines it reports an average 17.3% citation improvement (14% to 20% depending on the engine architecture). Its chunking principle targets paragraphs of 150 to 300 words.

The Princeton GEO study (arXiv 2311.09735, KDD 2024) tested content edits on a benchmark of queries. Adding statistics, quotations, and cited sources performed best, improving visibility by up to about 40% over baseline. The effect depends on where a page already ranks: citing sources raised the visibility of fifth-ranked pages by 115% while the top-ranked page lost 30%.

Change What the evidence says
Question-style headings, self-contained sections Part of GEO-SFE’s 17.3% average structural gain
Tables and lists for comparisons Part of GEO-SFE’s structural features; vendor studies claim 2-3x, unreplicated
Statistics and quotations Among the best methods in the Princeton GEO study
Named sources and citations Up to 115% visibility gain for lower-ranked pages (Princeton GEO)
FAQ blocks with FAQPage schema Vendor studies claim large lifts; no peer-reviewed evidence

Why tables? AI systems extract structured data efficiently. A table comparing three tools on five dimensions gives the model a clean, citable fact set. Prose making the same comparison requires inference and loses the citation precision.

Why FAQ sections? FAQ blocks match how AI systems structure answers: as direct responses to specific questions. FAQPage schema makes that structure explicit to crawlers, although Google has shown FAQ rich results only for a small set of authoritative sites since 2023. Vendor studies report large citation lifts from FAQ schema, but their methods are not published, so treat the numbers as marketing until someone replicates them.

Why modular chunks? AI systems retrieve content in chunks and don’t read articles holistically. A section that directly answers a specific question is more citable than a 2,000-word essay that eventually gets to the answer.

The crawler landscape

AI companies now run separate agents for training, search indexing and user-initiated fetches. Cloudflare’s May 2024 to May 2025 comparison shows how fast this changed:

GPTBot (OpenAI): OpenAI’s training crawler. Its share of AI crawler traffic rose from 5% to 30%, and raw requests grew 305%. Respects robots.txt. Blocking it signals that your content should not be used for training; ChatGPT search uses a separate agent, OAI-SearchBot, and user-initiated visits come from ChatGPT-User.

ClaudeBot (Anthropic): Anthropic’s training crawler. Its share of AI crawler traffic fell from 27% to 21%, with requests down 46%. Respects robots.txt. In a separate August 2025 analysis, Cloudflare reported Anthropic’s crawl-to-referral ratio falling from about 286,900:1 in January 2025 to about 38,000:1 in July 2025, still tens of thousands of pages crawled for every visitor sent back.

PerplexityBot: raw requests grew 157,490%, the highest growth rate in Cloudflare’s data, but from under 0.01% of traffic to 0.2%. In August 2025, Cloudflare reported that when PerplexityBot was blocked, Perplexity fetched pages with undeclared crawlers using a generic browser user agent and rotating IP addresses. Perplexity disputed the findings.

The practical split: blocking training crawlers (GPTBot, ClaudeBot) in robots.txt opts you out of training without removing you from AI search, which uses separate agents. Allowing the search agents (OAI-SearchBot, PerplexityBot) keeps you eligible for citations.

# robots.txt pattern: opt out of training, stay in AI search
User-agent: GPTBot
Disallow: /  # OpenAI training crawler

User-agent: ClaudeBot
Disallow: /  # Anthropic training crawler

User-agent: OAI-SearchBot
Allow: /  # ChatGPT search results

User-agent: PerplexityBot
Allow: /  # Perplexity search (compliance disputed, see above)

User-initiated agents such as ChatGPT-User may not follow robots.txt at all, by OpenAI’s own documentation, and Cloudflare has documented crawlers evading blocks. Treat this as ongoing maintenance, not a one-time configuration.

What to change first

Three changes follow directly from the research above:

Restructure prose into modular Q&A sections by adding question-based H2 headers: “What does X do?”, “When does X fail?”, “How do you implement X?” with direct 40 to 60 word answers at the start of each section. The goal is each H2 being citable in isolation, without requiring the surrounding context.

Add data tables for every comparison you currently make in prose. Every benchmark deserves structured rows and columns, with real numbers rather than checkmarks.

Add statistics and named sources. The Princeton GEO study found these among the most effective edits, especially for pages that do not already rank first. FAQPage schema is cheap to add on top (JSON-LD in the head element), but expect less from it than vendor claims suggest.

What not to do: optimize for citation at the expense of accuracy. In a 2025 Nature Communications study of medical questions, between 50% and 90% of LLM answers were not fully supported by the sources they cited. Being cited for a claim your page does not make attaches your name to an error. Precision over volume.

The honest reality check

AI referral traffic is about 1% of all traffic. For most sites, search engines, direct visits and social still drive almost everything. The urgency of AEO optimization depends heavily on your audience: in Conductor’s data, IT was the industry with the highest AI referral share (2.8%).

The contrarian view (that AEO is hype) conflates “not urgent yet” with “not real.” The underlying mechanics are real: crawlers are active and growing, citation patterns are measurable (12% overlap with Google’s top 10 in Ahrefs’ study), and content structure effects show up in controlled research (17.3% average in GEO-SFE, up to 40% in Princeton GEO). The market will grow toward these patterns faster than traditional SEO signals shift.

For AI agent builders, there’s a second angle: your agent’s retrieval pipeline is subject to the same citation dynamics (see agentic RAG for how agents control retrieval). The content your agent retrieves via web search will increasingly be content optimized for AI citation, meaning more structured, more modular, more data-dense. Understanding AEO helps you understand what your pipeline will increasingly encounter.

Key takeaways

  • AI search sent about 1.08% of traffic in Conductor’s 2025 benchmark, mostly from ChatGPT. Only 12% of ChatGPT, Gemini and Copilot citations ranked in Google’s top 10 in Ahrefs’ study.
  • Structure matters in controlled research: GEO-SFE reports a 17.3% average citation gain; Princeton GEO reports up to 40% from statistics, quotations and sources.
  • Vendor claims of 2-3x lifts for tables and FAQ schema are unreplicated. Use them as direction, not forecasts.
  • Brand mentions correlated more strongly with Google AI Overview visibility than backlinks (0.664 vs. 0.218, Ahrefs).
  • PerplexityBot requests grew 157,490% from a tiny base, and GPTBot’s share of AI crawler traffic rose from 5% to 30% (Cloudflare, May 2024 to May 2025).
  • Block training crawlers (GPTBot, ClaudeBot) if you object to training use; allow search agents (OAI-SearchBot, PerplexityBot) to stay eligible for citations.

FAQ

What is AEO (Answer Engine Optimization)? Structuring content to be cited by AI answer systems (ChatGPT, Perplexity, Claude, Gemini). Different from SEO: only 12% of URLs cited by ChatGPT, Gemini and Copilot ranked in Google’s top 10 in Ahrefs’ study. AI systems favor structured, modular, data-dense content.

How much traffic do AI systems currently send? About 1.08% of website traffic in Conductor’s 2025 benchmark, 87.4% of it from ChatGPT. Quality looks high in some studies: Semrush estimated AI visitors are worth 4.4x organic visitors on conversion rates, and SE Ranking found about 68% longer time on site.

What content formats get cited most? Question-style sections, tables and lists, statistics, quotations and named sources. GEO-SFE reports a 17.3% average gain from structural changes; Princeton GEO reports up to 40%. Vendor multipliers for tables and FAQ schema are unreplicated.

How do AI crawlers differ from Googlebot? AI companies split training, search and user-initiated fetching into separate agents. GPTBot and ClaudeBot (training) respect robots.txt; OAI-SearchBot and PerplexityBot handle search. PerplexityBot requests grew 157,490% from a tiny base, and Cloudflare reported Perplexity evading blocks, which Perplexity disputed. Block training crawlers; allow search agents for citation opportunities.

Fastest path to better AI citation rates? (1) Question-based H2s with 40 to 60 word direct answers. (2) Data comparison tables. (3) Statistics and named sources. Princeton GEO: citing sources improved visibility 115% for fifth-ranked pages.


Further reading