Do you know whether ChatGPT can read your website? Probably not, and it's not your fault: almost nobody checks. And checking sometimes brings a surprise: someone closed the door to AI engines years ago, by accident, and nobody had noticed.
AI bots don't work the way you picture them
There's a very common misconception about how ChatGPT, Perplexity, or Claude actually work. Most people assume that when you ask one of these engines a question, the bot dashes out to search the internet in real time — like a Google search happening on the spot. It doesn't always work that way: much of what they answer comes from what they already collected.
What these engines do is much more like how an encyclopedia works. Before you arrive with your question, someone already went out and collected the information, organized it, and made it available for quick retrieval. That «someone» is the crawlers — automated bots that continuously travel the web, read pages, and build a kind of knowledge map that language models can consult when generating a response.
OpenAI has its own: GPTBot. Google has several for its models. Anthropic has ClaudeBot. Perplexity has PerplexityBot. Each one has a name, and almost all of them introduce themselves by that name when they visit a page.
And the ones that follow the rules, before entering your site, read one file. The same file.
The sign on the door nobody re-read
That file is called robots.txt. It exists on a huge number of websites (not all), lives at a very specific address — yourdomain.com/robots.txt — and its job is exactly what it sounds like: telling each bot that arrives at your door what it can see and what it can't.
The syntax is simple. There's an instruction called «User-agent» that addresses a specific bot (or all of them, using an asterisk), and an instruction called «Disallow» that marks off blocked paths. Something like this:
- User-agent: * — addresses all bots
- Disallow: / — blocks them from the entire site
- User-agent: GPTBot — addresses OpenAI's bot specifically
- Disallow: /blog/ — blocks only the blog folder from that bot
Bots that follow the rules — and most of the major ones do — read this file before crawling. If it says they can't come in, they turn around. That simple. That quiet.
The problem isn't the technology. The problem is who wrote that file and when.
In many cases it was written by the developer who set up the site three or five years ago. Sometimes it was auto-generated by a WordPress plugin. Sometimes it was inherited from a template. Sometimes there's a security rule someone added «just in case» that nobody revisited. And when GPTBot first knocked on that door — well after that file was written — it was already locked. Without anyone having consciously made that decision.
Or, at the other extreme: the site has no robots.txt at all. Also very common. And in that case bots interpret that they can access everything — which may not be what you want either, if there are private sections or areas you'd rather not have indexed.
My opinion, not necessarily a gentle one:
I get that robots.txt doesn't keep anyone up at night. It's technical, it's boring, and it sounds like something only developers care about. But there's a perception error here worth correcting: many people investing in content, in SEO, in social media — in building a digital presence — haven't verified whether AI engines can even read what they're producing.
It's like paying for a spectacular advertising campaign and forgetting to open the store. The content work exists. The effort exists. But if the door is closed to AI crawlers, that work doesn't feed the models that more and more people consult before making a decision.
What strikes me most isn't that people haven't checked — it's that many didn't know this existed. The robots.txt has been part of how the internet works for decades. Google has read it from the beginning. What changed is that now there are many more «readers» of the sign: GPTBot, ClaudeBot, PerplexityBot, and those still to come.
Opening the door doesn't guarantee citations — but closing it can leave you out
There's an important distinction worth making clearly here. Allowing AI crawlers access is a necessary but not sufficient condition for appearing in these engines' responses.
GPTBot being able to crawl your site doesn't mean ChatGPT will cite you the next time someone asks about your industry. That takes more: content that answers specific questions, technical structure that models can interpret, brand authority that models recognize as a trustworthy source. All of that is what GEO and AEO optimization works on, and it's a process that takes time.
But closing the door to crawlers complicates everything else: if the bot can't read your content, it can't learn from it directly.
The robots.txt is the first door, not the only one. And many companies haven't even checked whether that first door is open.
AI visibility starts earlier than you think
Before talking about content strategy, schema markup, Entity Authority, or any other GEO tactic — you need to make sure AI crawlers can actually reach the content. The robots.txt is step zero. Not the only step, but the first.
What checking your robots.txt actually tells you
To be clear about what reviewing this file involves — because «it's just a text file» might sound like you just open it and read it — it's worth seeing what specific information it gives you and what it doesn't.
What robots.txt does tell you:
- Which bots have access to your site (or to which parts of it)
- Which bots are explicitly blocked
- Whether there's an inherited rule applying to «all» bots that might be closing the door unintentionally
- Whether the file exists at all (also useful information)
What it does NOT tell you:
- Whether the bots you've allowed are actually crawling your site
- Whether your content is good enough for the models to cite it
- Whether there are other technical barriers preventing crawling (JavaScript blocking content, very high load times, etc.)
- Whether bots are ignoring the robots.txt — some do
It's the first door, not a complete view of the house. But if the first door is closed, the rest carries much less weight.
How to check yours right now
The most direct way is to open a browser tab and type yourdomain.com/robots.txt. If the file exists, you'll see plain text with the rules. If nothing loads or you get a 404 error, the file doesn't exist — which has its own implications.
What to look for specifically:
Look for «User-agent: *» followed by «Disallow: /»
This combination blocks all bots from the entire site. If you see it, that's the clearest sign that something is closed that probably shouldn't be. It may have been intentional — some sites don't want to be crawled — but in many cases it's an inherited rule from a development configuration that was never changed.
Look for explicit mentions of GPTBot, ClaudeBot, or PerplexityBot
If your file has these lines with «Disallow», those specific bots are blocked. If they're not mentioned but there's an open rule for all bots, you should verify whether that rule includes or excludes them.
Check whether you have an llms.txt file in addition to robots.txt
The llms.txt is a newer standard — not yet universal — that some sites use to give language models more specific instructions about how to interpret their content. It doesn't replace robots.txt, but if you're already thinking about AI visibility, it's worth knowing about.
If you don't understand what you're reading, ask someone who does
The robots.txt format is technical but not complicated. Any developer can read it in two minutes. If you don't have one available, there are also tools that interpret it for you.
At AEON42 we have a free tool for this — you enter your domain and email, and within seconds it tells you which AI bots can read your site and which can't. Just a browser, nothing to install. You can check it with [AEON42's crawler tool](https://aeon42.com/herramientas/crawlers-ia), and here's what AEON42 is.
What this has to do with a real AI visibility strategy
The robots.txt is the starting point, but AI visibility — appearing in ChatGPT, Perplexity, Google AI Overviews when someone asks about what you do — is a broader discipline. It means working on content so it answers specific questions, not just so it «exists». It means technical structure that models can interpret. It means building what in technical SEO is called entity authority: getting language models to recognize your brand as a reliable source within your industry.
None of that works if the first step isn't handled. And the first step is as simple — and as overlooked — as reviewing a text file on your own server.
If you want to understand the full picture of how optimization for AI engines and organic search works, there's more context on the site. But start with the basics: open your robots.txt today.
And if you'd rather have someone review it and fix it for you — configure it properly for the AI crawlers that already exist and those still to come — the Play.Interactive team can handle it. Start with a brief and within 3 to 5 business days you have a detailed proposal, no charge. You can subscribe to the newsletter to hear about new tools when they're available, or reach out to us directly.
Frequently asked questions
Most of the major ones do — GPTBot from OpenAI, ClaudeBot from Anthropic, and PerplexityBot are all documented as bots that follow robots.txt instructions. But not every bot on the internet does, and it's important to understand that the file is not a technical barrier — it's a voluntary compliance signal.
Not necessarily a problem, but an uncontrolled variable. When there's no robots.txt, most bots interpret that they can crawl the entire site. That might be fine if you want full visibility, but it can also mean that private sections or work-in-progress areas get exposed to crawling. The ideal is to have a file that's deliberately configured, not simply absent.
Not automatically. Allowing crawler access is the minimum requirement for models to be able to learn from your content, but it doesn't guarantee citations. Whether an AI engine recommends you also depends on content quality, how it's structured, your domain's authority, and how well your site answers the questions users are actually asking. An open robots.txt is step zero, not the whole game.
At minimum, every time someone touches your site's configuration: a redesign, a migration, a new plugin installation, a platform change. It's also worth checking when you adopt a new content or SEO strategy — to make sure the bots you want to let in actually can get in. It's not a file that changes on its own, but it is one that can change without anyone telling you.
The robots.txt is the decades-old standard for telling any bot what it can crawl and what it can't. The llms.txt is a more recent proposal specifically aimed at language models: it doesn't control access, but gives them context about how to interpret a site's content. It doesn't replace robots.txt and isn't universally adopted yet, but it's part of the AI visibility vocabulary that will keep growing.