Free tool
robots.txt AI checker
This robots.txt AI checker reads your robots.txt file and tells you whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed to read your website. Enter your address and get the answer in a few seconds, free and without signing up.
Why check robots.txt for AI crawlers
Every site can publish a small text file at /robots.txt. It tells automated visitors which parts of the site they may read. Search engines have followed it for decades, and most of the companies behind AI assistants follow it too. Each one sends its own crawler with its own name, and each name can be allowed or blocked on its own line.
That matters because a rule written years ago for some other reason can quietly shut out an AI crawler today. A broad rule such as “block everything that isn't a search engine” is common on older sites and on sites that were once hit by heavy traffic. A security plugin or a hosting template can add the same kind of rule without anyone on your team noticing. If an AI crawler cannot read your pages, those pages cannot be used as a source when the assistant looks something up for a buyer.
The AI crawlers this tool tests
The check covers 14 crawlers. Five of them matter most for visibility today:
- GPTBot belongs to OpenAI and gathers content that may be used to train its models.
- OAI-SearchBot also belongs to OpenAI. It is the crawler behind search results inside ChatGPT, so blocking it can keep your pages out of those answers.
- ClaudeBot is Anthropic's crawler.
- PerplexityBot builds the index Perplexity uses to find and cite pages.
- Google-Extended is the token that controls whether Google may use your content for Gemini and related AI features. It is separate from Googlebot, so setting it does not change how your site appears in regular search.
The remaining crawlers come from other AI and data companies. The results table lists each one with its owner so you can decide on each separately.
How to read your result
Each crawler gets one of three labels.
- Allowed means your robots.txt does not stop that crawler from reading your pages.
- Blocked means a rule tells it to stay away, either by name or through a general rule that applies to everyone.
- Couldn't check means we could not get a clear answer for that crawler.
The sentence above the table sums it up. “All major AI crawlers can read your site” means the five crawlers listed above are all allowed. “Some AI crawlers are blocked” means at least one of them is not. “We couldn't read your robots.txt” means the file could not be fetched or understood, so no verdict is possible.
Blocking is a choice, and some businesses make it on purpose, for example to keep premium content out of AI training. The aim of this check is to make sure the choice is yours and not an accident.
How to allow an AI crawler
Open your robots.txt and look for a rule that names the crawler, or a rule under “User-agent: *” with a line that says “Disallow: /”. To let a crawler in, remove its block, or add a short group for it:
User-agent: GPTBotAllow: /
Save the file at the root of your domain, so it loads at yourdomain.com/robots.txt, and run the check again. Changes can take a few minutes to appear if your site sits behind a cache. Many website platforms have a setting for robots.txt in their SEO or privacy options, so you may not need to edit a file by hand.
What this check does not tell you
This tool checks your site's setup. It does not show what any assistant says about your business, and an allowed crawler is no promise of being mentioned. Allowing access is a starting point: the assistants still decide which sources to trust and which businesses to name.
Also, robots.txt is a request, not a lock. Well-run crawlers honor it, but it does not technically stop a visitor from fetching a page. If something must stay private, put it behind a login.
What to do after the check
If everything important is allowed, the next question is whether your pages give an assistant something worth citing: clear descriptions of what you sell, who it is for, how it compares, and what it costs to start. The AI readiness checker looks at that side of your setup, including structured data, a sitemap and text that loads without JavaScript. And if you want to know what the assistants actually answer when buyers ask about your market, weekly tracking shows that over time.
Compare
From one check to weekly tracking
| Feature | One snapshot (free tool) | Always-on tracking |
|---|---|---|
| How often | Once, when you run it | Every week, on its own |
| Questions | The one you enter | All the questions your buyers ask |
| Assistants | One answer set | ChatGPT, Gemini, Claude and Google AI |
| Competitors | Not compared | Ranked next to you, week by week |
| What you get | A snapshot | A trend, a score and a to-do list |
Questions
Does this robots.txt AI checker change anything on my site?
No. It reads the public robots.txt file at the address you enter, the same file any crawler can read, and reports what the rules say for each AI crawler. Nothing is written to your site.
If GPTBot is blocked, will ChatGPT stop mentioning my business?
Not necessarily. Blocking a crawler limits what that company's systems may fetch from your site. ChatGPT can still mention your business based on what it already knows and on other sources that describe you. Blocking does remove your own pages as a source it can read.
What does Google-Extended control?
Google-Extended is a token that lets you opt your content out of use for Google's Gemini models and some AI features. It does not affect whether your pages appear in regular Google Search results, which are controlled by Googlebot.
What does "Couldn't check" mean?
We could not get a clear answer for that crawler, usually because the robots.txt file could not be fetched or could not be read. Open yourdomain.com/robots.txt in a browser to see what is there, then run the check again.
My site has no robots.txt file. Is that a problem?
Crawlers treat a missing robots.txt as permission to read everything, so AI crawlers are not blocked. Many sites still add one so they can state a sitemap and make their choices explicit.