Free tool · free account

AI crawler checker

Enter a domain and see, crawler by crawler, whether its robots.txt lets the AI assistants in: ChatGPT's OAI-SearchBot, PerplexityBot, Claude-SearchBot and the rest, separated from the training crawlers you may choose to block. It also looks for an llms.txt, and the generator below writes one. The check takes a free SEO Stuff account, which is what keeps the bots off it.

llms.txt generator

Fill in the four fields and copy the result into a file called llms.txt at the root of your site. No account needed; nothing leaves your browser. Format per llmstxt.org.

Your llms.txt
# Your company

> One sentence on what the company does, for whom, and what makes it different.

Why robots.txt decides your AI visibility

Every assistant that cites web pages fetches them with a crawler that announces its name and obeys robots.txt. If the file says Disallow: / for that name, or for * without an exception, the assistant cannot read the page and will not cite it, whatever the page says. This happens by accident more often than on purpose: a blanket rule added to keep training crawlers out, a staging rule that shipped to production, a plugin's default.

The names matter because the companies split them. OpenAI trains on GPTBot and searches with OAI-SearchBot; Anthropic trains with ClaudeBot and searches with Claude-SearchBot; Google trains Gemini on Google-Extended and runs Search, AI Overviews and AI Mode on Googlebot. You can say no to training and yes to being found, and most sites that have thought about it do exactly that.

What to do with the result

  1. 1

    Any answer crawler blocked: fix it today

    Remove the rule, or add a group for that crawler with Disallow: (empty) above the * group. It takes effect the next time the crawler reads robots.txt, usually within a day.

  2. 2

    Training crawlers: decide once

    Blocking GPTBot, ClaudeBot, Google-Extended and CCBot keeps your text out of future training sets and costs you nothing in citations. Allowing them may help assistants know your brand without searching. Either is defensible; what is not is not knowing.

  3. 3

    Add an llms.txt

    Generate one below, put it at the site root, and keep it current. It is a summary, not a sitemap: what you are, what you sell, your prices, the five pages that matter.

  4. 4

    Then check whether you are cited

    An open door is necessary, not sufficient. The AI visibility checker asks the assistants real buyer questions and shows whether you come up and who does instead.

Questions people ask

Is the AI crawler checker free?

Yes, with a free SEO Stuff account; the account keeps bots from using the checker to scan the web. The llms.txt generator below runs in your browser and needs no account.

What is the difference between answer crawlers and training crawlers?

Answer crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot and the user-triggered fetchers) read pages to answer questions right now and cite them; block those and the assistant cannot recommend you. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended) collect text to train future models; blocking them does not stop you being cited today. Whether to allow training is a policy choice; allowing the answer crawlers is a visibility choice.

My robots.txt blocks GPTBot. Does ChatGPT still cite me?

Yes, as long as OAI-SearchBot and ChatGPT-User are allowed. OpenAI uses GPTBot for training and OAI-SearchBot for search; they obey separate rules. Many sites block the first and allow the second.

What does "partial" mean?

The crawler may fetch the site but some paths are disallowed for it. Usually that is fine (admin pages, search results, cart). It is a problem only when the blocked paths include the pages you want cited, which the list under each crawler will show.

What is llms.txt?

A plain-text file at /llms.txt that describes a site for AI assistants in a few hundred words: what it is, what it sells, the pages that matter. It is a convention (llmstxt.org), not a standard every assistant reads, but it costs nothing, and a clear summary in one place is what an assistant needs when it decides how to describe you. SEO Stuff's own is at seo-stuff.com/llms.txt.

The checker says robots.txt is missing. Is that bad?

No. A missing robots.txt means every crawler is allowed everywhere, which is the right default for most marketing sites. Add one when you have pages to keep out of the index or crawlers to keep out.

Does this check whether my pages are actually in ChatGPT's index?

No; it checks whether the door is open. Whether anyone walked through it is what the AI visibility checker measures: it asks the assistants real buyer questions and reports whether you come up.

Three ways to start

See where you stand

A free, manual SEO + AI search audit of your site, with a report within 48 hours. No credit card, no sales call.

Get the free audit

Talk it through

A free 20-minute call with the founder about whether AI search is a meaningful opportunity for your business.

Book a call

Have it done

The Done-For-You Package: audit, 10 pages of content, 3 DR50+ placements, dashboard. $999 once, delivered in 21 business days.

See the package

Related