GEO tutorial
How to Check if AI Crawlers Can Access Your Site
Can AI crawlers access your site? The answer lives in robots.txt, and it is often not the answer the site owner expects. A single wildcard rule, a copied template, or a CDN bot setting can block the very crawlers a site wants to reach — or quietly allow crawlers it meant to refuse. This guide shows how to check access crawler by crawler, read the result correctly, and fix only the rules that are actually wrong.
Step 1: Decide what you actually want
Before touching robots.txt, write down the policy in one sentence. Most sites fall into one of three camps: allow discovery crawlers that may surface the site in answers, but refuse training crawlers; allow both; or refuse both. The distinction matters because crawler names do not all mean the same thing. GPTBot and OAI-SearchBot, for example, are separate user agents with different stated purposes, and ClaudeBot, PerplexityBot and Google-Extended each have their own rules. A result that says “blocked” is not automatically a problem — it is only a problem when it contradicts the policy you just wrote down. Check the current documentation for each crawler before deciding, because names and behaviours change.
Step 2: Read robots.txt yourself first
Open https://your-site.com/robots.txt in a browser. Look for three things: a wildcard block such as “User-agent: * / Disallow: /”, named rules for specific AI agents, and the order of the groups. In robots.txt, the most specific matching group wins for a given crawler, so a named “Allow” for one agent can sit alongside a general block for everyone else. That is why hand-reading is error-prone: the file may look blocked at a glance while a specific agent is in fact allowed, or look open while a later rule quietly closes the door. Read it once to understand the shape, then let the checker evaluate it agent by agent.
Step 3: Run the checker and read it crawler by crawler
Enter your domain in the AI Crawler Access Checker and read the result one crawler at a time, comparing each line against the policy from Step 1. Allowed where you wanted allowed is done; blocked where you wanted blocked is also done. The interesting rows are the mismatches. Common ones include a wildcard block that was copied from a template and never reviewed, a named rule for a crawler that has since been renamed, and a rule that allows the homepage but blocks the directory where the best content lives. Note that this checker tests access to the site root from the public robots.txt; it does not test page-level noindex tags, login walls, or CDN bot-management rules, all of which can block a crawler that robots.txt allows.
Step 4: Fix only the mismatch, in this order
- Fix the wildcard first: if a broad “Disallow: /” for all agents is blocking crawlers you want, replace it with deliberate per-agent rules instead of deleting the file. Deleting robots.txt removes the policy entirely rather than fixing it.
- Add named rules, not comments: comments do not change behaviour. Write an explicit group for each agent whose access you care about, with the Allow or Disallow that matches your policy.
- Check the paths that matter: a crawler that can reach the homepage still cannot use a guide buried under a blocked directory. Check the directories that hold the pages you most want cited.
- Re-check after every edit: one rule change can change several agents at once, because specificity and grouping interact. Re-run the checker after each round of edits, not just at the end.
A minimal example
A site that wants answer-engine discovery but not training use might publish something shaped like this — adapted to its own policy, and only after checking each crawler’s current documentation:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Allow: /
This example is illustrative, not a recommendation: the right set of rules depends entirely on the policy from Step 1. The point of the example is the shape — one deliberate group per agent, no reliance on comments, and no wildcard doing work nobody reviewed.
What access does not prove
Allowed access is a prerequisite, not a promise. It does not prove an AI system will crawl the site, index it, cite it, or recommend it. Those outcomes also depend on clear, answer-shaped content, original evidence, internal links, indexability, and the site’s standing — and on each system’s own decisions about what to use. Treat the checker as the cheapest test in the stack: ten minutes here removes an entire class of silent failures before spending effort on content that a blocked crawler would never see. Pair it with the llms.txt validator so that, once crawlers can reach the site, they also find a clean map of its best pages.