How do ChatGPT and Perplexity decide which sites to cite?

They fetch pages with their own crawlers (OAI-SearchBot, PerplexityBot and others) or rely on search indexes like Bing and Google. If a crawler is blocked, can't read the content, or can't tell what the page is about, it cites someone else.

What's the difference between AI search crawlers and training crawlers?

Search crawlers fetch pages to answer questions and link to you — block them and you disappear from that assistant. Training crawlers (GPTBot, ClaudeBot, Google-Extended…) collect content to train models. Blocking those is a reasonable choice and doesn't affect AI search.

Why would my firewall block AI crawlers?

Many CDNs and security plugins now block AI bots by default — for example Cloudflare's "Block AI bots" setting or WordPress security plugins. Your robots.txt can say "allowed" while the server still returns 403. We test that directly.

Does blocking Google-Extended remove me from AI Overviews?

No. AI Overviews and AI Mode use the normal Googlebot index. Google-Extended only controls whether your content trains Gemini.

Do I need an llms.txt file?

It's optional and still emerging — Google doesn't use it — but it's a cheap way to give AI tools a curated map of your most important pages. We draft one for you from your page.

Why does JavaScript matter?

Most AI crawlers read the raw HTML and don't run JavaScript. If your content only appears after scripts load, they see an empty page. Server-side rendering or static generation fixes it.

Is it free? Are there limits?

Yes — no sign-up, no credit card. Each visitor gets 20 checks per day. Our firewall test uses the crawlers' public user agents from our server, so results are a close simulation, not a guarantee.

Free tool

Free AI search readiness checker

Can ChatGPT, Claude, Perplexity and Google's AI Overviews actually read your site? Enter a page and we'll check crawler access in robots.txt and at your firewall, whether your content works without JavaScript, and the signals AI uses to understand and cite you.

100% free
No sign-up
ChatGPT, Claude, Perplexity & Google
Starter llms.txt

Was this tool useful?

What is AI search readiness?

AI search readiness is whether assistants like ChatGPT, Claude, Perplexity, Google AI Overviews and Microsoft Copilot can crawl, read and cite a web page. It depends on three things: your robots.txt and firewall letting their crawlers in, content that's readable without JavaScript, and clear signals — structured data, headings and authorship — about what the page is and who's behind it.

The AI crawlers we check (15)

Each crawler has a robots.txt name. Search crawlers decide whether you appear in AI answers; training crawlers collect content to train models. Google AI Overviews and AI Mode use the normal Googlebot — Google-Extended only controls Gemini training.

AI search and answer crawlersBlock these and you won't be found or cited in that assistant.
robots.txt name	Company	What it powers
OAI-SearchBot	OpenAI	ChatGPT search results
ChatGPT-User	OpenAI	ChatGPT opening pages for users
Claude-SearchBot	Anthropic	Claude search results
Claude-User	Anthropic	Claude opening pages for users
PerplexityBot	Perplexity	Perplexity answers and citations
Googlebot	Google	Google Search, AI Overviews and AI Mode
Bingbot	Microsoft	Bing and Microsoft Copilot
Applebot	Apple	Siri, Spotlight and Safari suggestions

AI training crawlersBlocking these is a choice — it doesn't affect AI search results.
robots.txt name	Company	What it powers
GPTBot	OpenAI	Training OpenAI models
ClaudeBot	Anthropic	Training Claude models
Google-Extended	Google	Training Gemini (not AI Overviews)
Applebot-Extended	Apple	Training Apple Intelligence
CCBot	Common Crawl	Open dataset used to train many models
Meta-ExternalAgent	Meta	Training Meta AI models
Amazonbot	Amazon	Alexa and Amazon AI

Want your site built to be found and cited? See our web development and headless CMS services.

How it works

What AI sees when it visits your site.

Enter a page
Your homepage, a service page or an article — any public page.
We check access
robots.txt rules for 15 AI crawlers, plus a test request as GPTBot, ClaudeBot and PerplexityBot to catch firewall blocks.
We read it like a bot
Content without JavaScript, headings, Q&A sections, structured data, author and dates, llms.txt and sitemap.
Fix the gaps
A score out of 100, a fix for every issue, and a starter llms.txt to copy.

FAQ

AI search readiness questions

Built for people and AI.

We build fast, server-rendered sites on Next.js, Sanity and WordPress with the structure AI assistants need to understand and recommend you.

Talk to us

What is AI search readiness?

The AI crawlers we check (15)

Enter a page

We check access

We read it like a bot

Fix the gaps

AI search readiness questions

Built for people and AI.

More free website tools