What this article covers: The diagnosis: why ChatGPT doesn’t mention you even though you rank well on Google.
Plenty of businesses sit on page one of Google for their key terms, yet when the same question is put to ChatGPT or Perplexity, it is a competitor that shows up in the answer. That is not bad luck and it is not necessarily a sign of weaker content. AI search engines reach your website by a different route than Google does — and that route has obstacles that classic SEO never had to deal with.
Two different questions
Google answers the question "where is the information". The result is a list of links and the user decides which one to open. Your job is to be as high on that list as possible.
An AI search engine answers the question "what is the answer". It assembles one from several sources and cites only the few it actually used. There is no list — there is an answer and three to five links. So it is not enough to be found: you have to be understood well enough to be usable as a source.
Between those two jobs sits a technical difference worth knowing about.
Three kinds of AI crawlers doing three different jobs
The most common misconception is that there is one "AI bot". In reality every major provider runs separate crawlers for separate jobs, and blocking one does not block another (OpenAI crawler documentation):
- Training crawlers (GPTBot at OpenAI, ClaudeBot at Anthropic) collect content that may be used when building future models. They have no direct bearing on today's answers.
- Search index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index an AI search engine draws on when it needs current information. These matter most for your visibility.
- On-demand fetchers (ChatGPT-User, Claude-User) visit a page only when a user asks for something from it during a conversation. This is where it is decided whether your page is reachable at that exact moment.
Tip: Google-Extended is a special case: it is not a crawler but a switch that tells Google whether your content may be used in its AI products. It has no effect on your ordinary Google ranking — those are two separate things.
Trap one: a robots.txt that does not do what you think
robots.txt has a rule that catches out even experienced people: a named group inherits nothing from the wildcard group. If you have general rules for all crawlers at the top and then address GPTBot separately, only the rules written inside its own group apply to it. Everything else simply does not.
In practice this cuts both ways. If you meant to restrict one crawler and gave it its own group, you may have accidentally opened up everything you had restricted generally. And the other way round: if you meant to allow AI crawlers but never listed them, they fall under general rules that were possibly written for an entirely different purpose.
This single detail accounts for a surprising share of cases where a company believes everything is fine and it is not.
Trap two: content that is not in the source
Modern sites often assemble their content in the browser, with JavaScript. Google's crawler can execute it; many AI crawlers cannot — they read only what the server sends in the first HTML response.
Checking takes thirty seconds: on your own site press Ctrl+U (Cmd+Alt+U on a Mac) to view the source and search for a distinctive sentence from the page. If it is not in the source, chances are an AI cannot see it either.
The same applies to content hidden behind tabs, modals or a "read more" button when it is only loaded on click.
Trap three: the security system that quietly shuts the door
This is the sneakiest one, because everything in your own settings looks correct. Hosting platforms run firewalls and anti-abuse systems that automatically block traffic they consider suspicious — and to many of them, a visit from a crawler they do not recognise is exactly that.
The result: robots.txt grants access, the site opens perfectly in your browser, and the AI crawler gets a 403 instead of your content, or is simply ignored. You never find out, because no warning appears anywhere.
This is not theoretical. While publishing this very service, our own server rejected a completely legitimate request because it did not come from a browser. For forms that kind of protection is welcome — for AI crawlers it can be fatal.
Tip: The only reliable proof that AI crawlers actually reach your site is the server access log. It shows the crawler name, the URL requested and the response code. Everything else is guesswork.
What an AI needs in order to cite you
Once access is sorted, the second half begins: can the system understand and summarise you? Three things help.
- A clear structure. Headings that say what a section is about, and short paragraphs. It is hard to extract an answer to a specific question from a page that is one solid block of text.
- Answers to the real questions. If customers ask what it costs and how long it takes, put that on the page in the words they use — not in the phrasing you use internally.
- Structured data. schema.org markup states unambiguously and machine-readably who you are, where you operate, what you offer and when you are reachable. Without it the system has to infer all of that from prose, and sometimes it gets it wrong.
How to check it yourself in five minutes
None of this needs special tools. Do the following:
- Ask ChatGPT or Perplexity about your line of work in your town, the way a customer would. See who gets named.
- Open yourdomain.com/robots.txt and look for the crawler names from this article. If they are absent, the general rules apply to them — check what those rules say.
- View the page source and search for a sentence from your site. If it is not there, the content is being built in the browser.
- Ask your hosting provider for last month's access log and look for the crawler names in it. That tells you the truth about access.
What can and cannot be promised
Here honesty matters. Nobody can guarantee that ChatGPT will recommend you. Answers are produced by models we do not control, their behaviour shifts from month to month, and providers do not disclose how answers are assembled.
What can be guaranteed is the prerequisite: that your site is reachable by the crawlers you want to let in, that the content is in the HTML, that your business facts are machine-readable, and that the content is structured so an answer can be lifted from it. Without that there is no chance. With it, there is — and that part is genuinely within your control.
If you would like to know where you stand, our quick check of AI search accessibility is free: send us your address and we will tell you what we found within one working day.
Frequently asked questions
Does blocking AI crawlers hurt my Google ranking?
No. Google-Extended and the other AI crawlers are separate from Googlebot, which handles classic search. Blocking AI crawlers does not lower your position in Google search — it only means your content will not be used in AI answers.
Do I have to allow AI crawlers at all?
That is your call. If you want AI search engines to cite you as a source and send you visitors, access is a prerequisite. If you do not want that, it is better for crawlers to be blocked deliberately and correctly than by accident — and right now, for most websites, it is by accident.
Is llms.txt of any use?
It is a proposed convention rather than an established standard, and no provider guarantees it is read. But since the file is short, cheap and cannot do harm, it is worth having — the worst case is that nobody reads it. Google confirmed (June 2026) that it is not required for Google Search and has no effect on visibility.
How often should this be checked?
Crawler access can change with any hosting or security update, and AI answers shift on their own. It is worth checking after major site changes and a few times a year.