Send an enquiry SL

Free tool

Can AI crawlers read your website?

Enter a domain and the tool reads its robots.txt file. For 11 crawlers from OpenAI, Anthropic, Perplexity, Google and Microsoft it shows whether they may read the homepage, and separates search crawlers from model-training crawlers.

Crawlers checked
11
Price
free
Sign-up
not needed

Which crawlers we check

  • OAI-SearchBot (OpenAI): ChatGPT search. If you block it, the site is not shown as a source in ChatGPT search answers
  • GPTBot (OpenAI): model training. According to OpenAI, blocking it does not affect search
  • ChatGPT-User (OpenAI): visits a page when a user asks. OpenAI states robots.txt rules may not apply to it
  • Claude-SearchBot (Anthropic): Claude search
  • ClaudeBot (Anthropic): model training
  • Claude-User (Anthropic): visits a page when a user asks Claude a question
Summary from this page

Check your domain

Enter just the domain, for example company.com. The tool reads the public robots.txt file, checks whether llms.txt exists and shows which rule applies to each crawler.

We only read the public robots.txt and llms.txt files. Results are kept for at most 15 minutes so that repeated checks of the same domain do not load its server.

Which crawlers we check

AI providers use different crawlers for different purposes. Search crawlers matter most for visibility in answers; training crawlers decide whether content is used to train models.

  • OAI-SearchBot (OpenAI): ChatGPT search. If you block it, the site is not shown as a source in ChatGPT search answers.
  • GPTBot (OpenAI): model training. According to OpenAI, blocking it does not affect search.
  • ChatGPT-User (OpenAI): visits a page when a user asks. OpenAI states robots.txt rules may not apply to it.
  • Claude-SearchBot (Anthropic): Claude search.
  • ClaudeBot (Anthropic): model training.
  • Claude-User (Anthropic): visits a page when a user asks Claude a question.
  • PerplexityBot (Perplexity): search and links in Perplexity; according to the provider, not used for training.
  • Perplexity-User (Perplexity): user-requested visits; Perplexity states it generally ignores robots.txt rules.
  • Googlebot (Google): Google Search, including search features.
  • Google-Extended (Google): a robots.txt token, not a separate crawler. It controls use of content for Gemini. It does not affect Google Search.
  • Bingbot (Microsoft): the Bing search engine that Microsoft Copilot builds on.

Sources: Sources: documentation from OpenAI, Anthropic, Perplexity and Google, checked 13 November 2026.

What the result means

  • Allowed

    The crawler may read the homepage and has no specific restrictions.

  • Partly restricted

    The homepage is allowed but some parts of the site are not. This is often intentional, for example the cart or the admin area.

  • Blocked

    A rule forbids reading the homepage. For a search crawler this means that search engine will not cite you as a source.

  • No robots.txt

    The file does not exist (404 response). Under the standard this means every crawler may read the site.

Blocking a training crawler is not a mistake

Whether your content may be used for model training is your decision. OpenAI and Anthropic separate their crawlers: you can block GPTBot and ClaudeBot and still leave the search crawlers (OAI-SearchBot, Claude-SearchBot) open. You stay a possible source in answers without offering your content for training.

The common mistake is the opposite: a blanket "Disallow: /" left over from a staging site, or a plugin that closes every AI crawler with one click, including the search ones.

What 500 Slovenian websites show

In our Digitalni utrip study (data captured 22 September 2026), robots.txt allowed Googlebot on 499 of 500 domains and OAI-SearchBot on 497 of 500. 42 of 497 checked sites (8.5%) had an llms.txt file. Blocking in robots.txt is rare. A crawler can also be stopped by a firewall or bot protection, which robots.txt does not show. The study is published in Slovenian: Digitalni utrip.

What the tool does not check

  • Firewalls and bot protection (e.g. Cloudflare): these can reject a crawler even when robots.txt allows it.
  • Rules for individual pages: we check the homepage, not every subpage, and not noindex tags in the HTML.
  • Whether you are indexed or cited: permission only means a crawler may read. Search Console shows indexing; an AI visibility measurement shows mentions in answers.

Frequently asked questions

More questions and the contact form on the enquiry page.

Should I block GPTBot?

That is a business decision. GPTBot collects content to train OpenAI models; according to OpenAI, blocking it does not affect how you appear in ChatGPT search. Search is handled by OAI-SearchBot, which is worth keeping open if you want to be cited as a source.

Does blocking Google-Extended affect Google Search?

No. Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It only controls use of content for Gemini models.

Does llms.txt improve Google rankings?

No. llms.txt is a proposed format that Google does not use as a ranking signal. The tool shows whether the file exists, but its presence says nothing about visibility in answers.

robots.txt allows the crawlers, but ChatGPT still does not cite me. Why?

Permission is only the first condition. Answers depend on whether the page is indexed, whether it clearly answers the question and whether other sources mention it. A firewall this tool cannot see may also stop the crawler.

Does the tool store my domain?

The result for a domain is kept for at most 15 minutes and then deleted. We do not link the domain to the visitor or use it for anything else.

Access is sorted. What about visibility?

If you want to know whether AI search engines actually cite you and what to fix first, start with a free website check.

Reply
within one working day
Opening hours
Mon–Fri, 8:00–⁠16:00 (Slovenian time)

Happy with our work? Pass it on.

For a small agency a recommendation is the highest praise. Share the link with a business owner who could use a better website.