How do you test AI crawlers — isn't robots.txt enough?
No. robots.txt says what a crawler may do; your hosting, CDN or firewall decides what it actually gets. We request your homepage with each crawler's documented User-Agent (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot and 20 more) — the nine key ones also robots.txt and two deeper pages — and compare every answer with what a browser gets for the same URL: blocked, challenged, paywalled with HTTP 402, served noindex, or served different content. A request that times out or is rate-limited is retried once before we give a verdict. Our requests come from our own IP, so a firewall that verifies crawlers by IP may still treat the real crawler differently — the report says so wherever it matters.
What does "AI-ready" mean for a website?
Three things. AI crawlers must be allowed in (robots.txt, meta tags and your CDN or firewall all agree), they must be able to read the content without running JavaScript, and the facts that matter — who you are, what you sell, prices, authors, dates — should be stated in structured data so they are not guessed. That is the "AI search" score. The "AI agents" score adds what agents need to use the site: forms and buttons they can operate, a Markdown version, declared AI usage terms and, for shops, machine-readable shipping and returns and agentic checkout (UCP).
How is the score calculated?
Two parts, both in the grade. "AI search" — crawler access, readable content, structured data and sitemaps — is about three quarters: it decides whether AI answers can find, read and quote you. "AI agents" — usability for browser agents, agent protocols and, for shops, agentic commerce — is the rest, so a site cannot reach an A by ignoring agents. Each check says who backs it (a vendor, a spec or our heuristic). A critical failure, such as Googlebot blocked or content that needs JavaScript, caps the score at 55. Anything we could not check is never counted against you, and "Fix these first" shows how many points each fix adds.
Which AI crawlers should I allow?
Allow the search and answer crawlers — OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), PerplexityBot, plus Googlebot, Bingbot and Applebot, which feed Google AI Overviews, Copilot and Siri. They decide whether you are cited in AI answers. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot only collect data for future models; blocking them is a business decision and does not remove you from answers.
Why does JavaScript rendering matter?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot read the raw HTML and do not execute JavaScript. If your content is rendered in the browser — a typical single-page app — those crawlers see an empty shell. Server-side rendering, static generation or prerendering fixes it.
Do I need an llms.txt file?
It is optional. llms.txt is a Markdown map of your key pages proposed for AI tools, but no major AI vendor has confirmed using it and Google says it neither helps nor harms. It is cheap to add, so this checker offers a draft built from your site. A missing file never counts against you; a valid one adds a little to the AI agents score and a broken one costs a little — the same way Google's Lighthouse treats it.
Does structured data help with AI answers?
Google says AI features need no special markup beyond what already helps Search, while Microsoft explicitly recommends schema.org markup for Bing and Copilot. Either way, Organization, Product with Offer, Article and BreadcrumbList state facts in a form machines read reliably, and they must match the visible page.
What are MCP and agent discovery files?
AI agents increasingly act for users — searching, booking, buying. A Model Context Protocol (MCP) server lets them do that on your site through a proper interface, and discovery files such as /.well-known/ard.json tell agents where to find it. For shops, /.well-known/ucp is what Google's AI Mode and Gemini use to check out on your store (live in the US, Canada and Australia so far). A working MCP server adds to your score; not having one is never counted against you, because assistants only connect to servers their users add.
Can my AI agent run the scan itself?
Yes. Tell it to open https://optimize-for-ai.com/scan?url=your-site.com — it gets the AI fix plan as Markdown (add &format=json for the full report). For Claude, ChatGPT, Cursor or any MCP client, add https://optimize-for-ai.com/mcp as a connector (Streamable HTTP, no sign-in): its tool scan_website runs the scan and returns the same plan. Custom GPT actions can import https://optimize-for-ai.com/openapi.json. The same fair-use limits apply as on this page.
What does the scan do to my site, and is anything stored?
About 80–100 requests over a few seconds: the homepage, robots.txt, sitemaps, five sampled pages and well-known discovery files, plus one homepage request per AI crawler using that crawler's documented User-Agent — so we see exactly what your server and firewall answer it. Every request carries an "OptimizeForAI-probe" marker so it is identifiable in your logs. If you publish an MCP server on your own domain, we connect to it as a client and list its tools. We also look your domain up in the public MCP Registry (registry.modelcontextprotocol.io). Nothing is stored on our side: the report exists only in your browser.
