AI search readiness is the degree to which an AI assistant — ChatGPT, Perplexity, Google's AI Overviews, or Microsoft Copilot — can crawl, read, and confidently cite your website when a user asks a question you could answer. Many Australian small-business sites are invisible to these tools without their owners realising it, and the causes are almost always technical rather than content-related.
How AI Assistants Actually Find Your Pages
It helps to separate two distinct pipelines: AI training and AI search retrieval. Training is a one-time (or periodic) bulk crawl that bakes knowledge into a model. Search retrieval is what happens the moment someone asks a question — the assistant fetches live results, the same way a traditional search engine does.
Google AI Overviews and AI Mode use the ordinary Googlebot crawl index — there is no separate AI-specific crawler to allow. Microsoft Copilot pulls from Bingbot. ChatGPT's browsing feature sends its own OAI-SearchBot, and Perplexity sends PerplexityBot. Anthropic uses Claude-SearchBot for Claude's search results and Claude-User when someone asks Claude to open a page (both separate from its training crawler).
The table below lists the crawlers you need to know. The 'search' bots determine whether you appear in AI-generated answers today. The 'training' bots determine whether future model versions know about you — blocking them is a legitimate business choice and has no effect on AI search visibility.
- OAI-SearchBot — OpenAI — powers ChatGPT search retrieval (live browsing)
- ChatGPT-User — OpenAI — opens pages when a ChatGPT user asks it to
- PerplexityBot — Perplexity AI — powers Perplexity's AI search answers
- Claude-SearchBot — Anthropic — powers Claude's web search feature
- Googlebot — Google — feeds AI Overviews and AI Mode (same as organic search)
- Bingbot — Microsoft — feeds Copilot and Bing search
- GPTBot — OpenAI — training only, not search retrieval
- ClaudeBot — Anthropic — training only, not search retrieval
- Google-Extended — Google — training opt-out only; does NOT affect AI Overviews
- CCBot — Common Crawl — training data, not search retrieval
Check 1: Your robots.txt Isn't Blocking AI Search Crawlers
This is the most common silent blocker. A robots.txt rule that disallows * (all bots) or names specific AI crawlers will prevent retrieval. The nuance people miss: you can legitimately block training bots while keeping search bots open. Here is a robots.txt that does exactly that:
User-agent: *Allow: / # Allow AI search retrieval botsUser-agent: OAI-SearchBotAllow: / User-agent: ChatGPT-UserAllow: / User-agent: PerplexityBotAllow: / User-agent: Claude-SearchBotAllow: / # Block AI training bots (does not affect AI search visibility)User-agent: GPTBotDisallow: / User-agent: ClaudeBotDisallow: / User-agent: Google-ExtendedDisallow: / User-agent: CCBotDisallow: / Sitemap: https://yourdomain.com.au/sitemap.xmlCheck your live robots.txt by visiting https://yourdomain.com.au/robots.txt and reading it carefully. A broad Disallow: / under User-agent: * blocks everything unless overridden below.
Check 2: Your CDN or Firewall Isn't Silently Refusing AI Crawlers
Cloudflare introduced a one-click 'Block AI bots' toggle in its bot management settings. If your site sits behind Cloudflare, that toggle may be on by default depending on your plan and when it was activated. It returns a 403 or a Cloudflare challenge page to AI crawlers, regardless of what your robots.txt says.
WordPress security plugins such as Wordfence and iThemes Security can do the same thing with custom firewall rules. Because browsers and uptime monitors still get the page normally, this kind of block usually goes unnoticed.
To check: use a user-agent spoofing tool (curl works fine) to send a request pretending to be OAI-SearchBot and confirm you get a 200 response with real content — not a challenge page or error.
curl -A "OAI-SearchBot" https://yourdomain.com.au/ -o /dev/null -w "%{http_code}\n"Repeat for PerplexityBot and Claude-SearchBot. You want 200 back every time.
Check 3: Your Key Content Doesn't Depend on JavaScript to Appear
Most AI search crawlers don't execute JavaScript. If your homepage headline, service descriptions, or pricing information only appear after a React component hydrates or an API call completes in the browser, those crawlers see a blank shell.
This is a known limitation of single-page applications (SPAs) built without server-side rendering. The fix is server-rendered or statically generated HTML — pages where the content exists in the initial HTTP response, before any JavaScript runs. Next.js handles this cleanly with its App Router, rendering content on the server by default. My frontend development work defaults to server rendering precisely because it benefits both traditional Googlebot and AI crawlers.
To check: open your browser's developer tools, disable JavaScript, and reload your pages. If you see empty sections, so do most AI crawlers.
Check 4: Google Can Still Crawl and Index You Properly
Because Google AI Overviews and AI Mode pull from the same index as organic search, your existing Google Search Console health directly predicts your AI Overviews visibility. Check Coverage errors, confirm your sitemap is submitted and processed, and verify that your most important pages are marked index, follow in their meta robots tags.
A page that ranks on page two of organic results is a reasonable candidate for an AI Overview citation. A page that isn't indexed at all will never appear. Fix indexing first; AI visibility follows.
Check 5: Structured Data Gives AI Assistants Clear Attribution
AI assistants parse structured data (Schema.org markup in JSON-LD format) to understand who you are, what you do, where you're based, and whether a page is an article, a product, a FAQ, or a local business listing. Without it, they're guessing from prose alone.
Three schema types give the highest return for a small business:
- Organization or LocalBusiness — your business name, ABN if relevant, address, phone, and primary URL. Add this to every page via your site layout.
- Article with
author,datePublished, anddateModified— signals that your content is maintained and attributable to a real person or organisation. - FAQPage — marks up question-and-answer sections so assistants can lift them directly into conversational answers.
You can generate correct JSON-LD markup with my free schema generator, then paste it into your site's <head> or body. Validate it in Google's Rich Results Test before publishing.
Check 6: Your Page Structure Uses Clear Headings and Question-Style Sections
AI assistants parse heading hierarchies to understand what a page is about and which section answers which query. A page with a single <h1> and five paragraphs of solid text is harder to attribute than one with descriptive <h2> subheadings that mirror how users phrase questions.
Practical habits that help:
- Use descriptive
<h2>headings that answer a specific question ('How much does a website cost in Sydney?' rather than 'Pricing'). - Follow headings with a direct answer in the first sentence of that section — AI assistants often lift that first sentence.
- Use bullet lists and short tables for comparative information; they parse more reliably than prose.
- Keep paragraphs under 80 words where possible.
Check 7: Consider Adding an llms.txt File
llms.txt is an emerging, unofficial standard — proposed in 2024 — that lets site owners publish a structured plain-text map of their content specifically for large language models (LLMs) to read. Think of it as a sitemap.xml written in Markdown that points to your most important pages and explains what they contain.
Important caveats: Google does not use `llms.txt` for AI Overviews. It is not a ranking signal and it carries no guarantee that any AI assistant will act on it. Some AI tools and developer agents do read it. The file lives at https://yourdomain.com.au/llms.txt and costs almost nothing to add.
A minimal llms.txt looks like this:
# WebKingdom > Freelance web development practice based in the Sutherland Shire, Sydney.> Specialities: Next.js, Sanity CMS, WordPress, HTML email signatures, site speed and SEO. ## Services - [Frontend Development](https://webkingdom.com.au/services/frontend-development): Next.js and React sites, server-rendered, performance-optimised.- [WordPress Development](https://webkingdom.com.au/services/wordpress-development): Custom WordPress builds with clean architecture.- [Website Speed Optimisation](https://webkingdom.com.au/services/website-speed-optimisation): Core Web Vitals and load time improvements. ## Resources - [AI Search Readiness Checklist](https://webkingdom.com.au/resources/ai-search-readiness-checklist) ## Contact - https://webkingdom.com.au/get-in-touchAdd it only after the other six checks pass — it amplifies a correctly configured site but can't compensate for a blocked or JavaScript-dependent one.
Run the Full Check in About a Minute
The seven checks above cover the most common failure points, but running them manually across a whole site is tedious. My free AI Search Readiness Checker checks your URL for most of these in one go and returns a prioritised report — no account required. It checks your robots.txt rules, server responses to known AI crawler user-agents, indexability, and whether structured data is present.
If your site is built on WordPress and you're finding security plugin conflicts, my WordPress development work includes reviewing firewall rules and crawler access settings as part of a site audit. If the underlying issue is JavaScript-dependent content, we can talk about moving to a server-rendered stack, something I do regularly as part of my web development work.



