Check your domain
Enter just the domain, for example company.com. The tool reads the public robots.txt file, checks whether llms.txt exists and shows which rule applies to each crawler.
Which crawlers we check
AI providers use different crawlers for different purposes. Search crawlers matter most for visibility in answers; training crawlers decide whether content is used to train models.
- OAI-SearchBot (OpenAI): ChatGPT search. If you block it, the site is not shown as a source in ChatGPT search answers.
- GPTBot (OpenAI): model training. According to OpenAI, blocking it does not affect search.
- ChatGPT-User (OpenAI): visits a page when a user asks. OpenAI states robots.txt rules may not apply to it.
- Claude-SearchBot (Anthropic): Claude search.
- ClaudeBot (Anthropic): model training.
- Claude-User (Anthropic): visits a page when a user asks Claude a question.
- PerplexityBot (Perplexity): search and links in Perplexity; according to the provider, not used for training.
- Perplexity-User (Perplexity): user-requested visits; Perplexity states it generally ignores robots.txt rules.
- Googlebot (Google): Google Search, including search features.
- Google-Extended (Google): a robots.txt token, not a separate crawler. It controls use of content for Gemini. It does not affect Google Search.
- Bingbot (Microsoft): the Bing search engine that Microsoft Copilot builds on.
Sources: Sources: documentation from OpenAI, Anthropic, Perplexity and Google, checked 13 November 2026.
What the result means
Allowed
The crawler may read the homepage and has no specific restrictions.
Partly restricted
The homepage is allowed but some parts of the site are not. This is often intentional, for example the cart or the admin area.
Blocked
A rule forbids reading the homepage. For a search crawler this means that search engine will not cite you as a source.
No robots.txt
The file does not exist (404 response). Under the standard this means every crawler may read the site.
Blocking a training crawler is not a mistake
Whether your content may be used for model training is your decision. OpenAI and Anthropic separate their crawlers: you can block GPTBot and ClaudeBot and still leave the search crawlers (OAI-SearchBot, Claude-SearchBot) open. You stay a possible source in answers without offering your content for training.
The common mistake is the opposite: a blanket "Disallow: /" left over from a staging site, or a plugin that closes every AI crawler with one click, including the search ones.
What 500 Slovenian websites show
In our Digitalni utrip study (data captured 22 September 2026), robots.txt allowed Googlebot on 499 of 500 domains and OAI-SearchBot on 497 of 500. 42 of 497 checked sites (8.5%) had an llms.txt file. Blocking in robots.txt is rare. A crawler can also be stopped by a firewall or bot protection, which robots.txt does not show. The study is published in Slovenian: Digitalni utrip.
What the tool does not check
- Firewalls and bot protection (e.g. Cloudflare): these can reject a crawler even when robots.txt allows it.
- Rules for individual pages: we check the homepage, not every subpage, and not noindex tags in the HTML.
- Whether you are indexed or cited: permission only means a crawler may read. Search Console shows indexing; an AI visibility measurement shows mentions in answers.