How AI crawlers work: GPTBot, ClaudeBot, PerplexityBot and your robots.txt

Before an AI engine can cite you, it has to reach your content, and it reaches it through crawlers — automated fetchers with names like GPTBot, ClaudeBot, PerplexityBot and Google-Extended. How you treat those crawlers in your robots.txt is a quiet but consequential decision: handled thoughtlessly, it can make you ineligible to appear in the very answers you are trying to show up in, or leave you exposed to uses you would rather avoid. Yet most sites either ignore AI crawlers entirely or block them in a panic, neither of which is a deliberate choice. This is a practical guide to what the major AI crawlers actually do, how robots.txt controls them, and how to set a crawl policy on purpose rather than by accident — because in the AI era, who you let in partly determines who can cite you.

Meet the AI crawlers

The AI ecosystem has introduced a new generation of crawlers, each tied to a company and a purpose. OpenAI operates GPTBot and related fetchers; Anthropic operates crawlers including ClaudeBot; Perplexity runs PerplexityBot; Google has introduced Google-Extended as a control for its AI uses, separate from its classic search crawler. Each identifies itself with a user-agent string, and each can be addressed individually in your robots.txt. The first step toward a deliberate policy is simply knowing they exist and that they are distinct from the traditional search crawlers you may already allow. Treating ‘AI crawlers’ as one undifferentiated mass is the mistake that leads to careless blanket decisions; they are distinct actors with distinct purposes.

Crawling, training, and retrieval are different

The single most important distinction to grasp is that AI crawlers serve different functions, and conflating them leads to bad decisions. Some crawlers gather content to help train or update models; others fetch pages live, in real time, to answer a specific user question right now. These are fundamentally different activities with different consequences for you. Blocking a training crawler affects whether your content informs a model's general knowledge; blocking a live-retrieval fetcher affects whether you can be cited in answers to current questions. A policy that fails to distinguish the two can accidentally cut you out of live citations while trying to opt out of training, or vice versa. Understanding which crawler does which is the foundation of a sensible policy.

GPTBot and OpenAI's fetchers

GPTBot is OpenAI's crawler associated with gathering web data, and OpenAI also operates fetchers involved in retrieving content to answer questions in its products. The practical point is that OpenAI's crawling is not monolithic: there are distinct user-agents for distinct purposes, and your robots.txt can treat them separately. If your goal is to be available as a source when someone asks ChatGPT a question your content answers, you need to understand which of OpenAI's agents handles that retrieval and ensure you are not blocking it. Blocking everything OpenAI operates in order to opt out of training may also opt you out of being cited, which is usually not what a site trying to grow AI visibility actually wants.

ClaudeBot and Anthropic's crawlers

Anthropic operates crawlers including ClaudeBot, which gather web content, alongside fetchers associated with retrieving pages to support answers. As with OpenAI, the key is that these are identifiable, separable agents you can address individually in robots.txt. A site that wants to be eligible for citation in Claude's answers should ensure it is not inadvertently blocking the relevant fetchers while intending only to limit training use. The recurring theme across every provider is the same: broad, reflexive blocking of a company's entire crawler footprint tends to sweep up the retrieval access you may actually want to keep, so it pays to know which agent does what before writing a rule.

PerplexityBot and retrieval-first engines

Perplexity is built around live retrieval and citation, which makes its crawler especially relevant to AI visibility. PerplexityBot fetches content to support the answers Perplexity generates, and because Perplexity so prominently cites its sources, being accessible to it is directly tied to being a named source in its answers. For a site whose goal is citation, blocking a retrieval-first engine's crawler is close to opting out of a channel designed to send attribution your way. This is the clearest illustration of the general principle: the more an engine is built around citing live sources, the more directly your crawl policy toward it determines whether you can appear as one of those cited sources.

Google-Extended and the search crawler

Google complicates the picture usefully by separating its classic search crawler from its AI uses. Google-Extended is a control that lets you manage whether your content is used for certain Google AI purposes without affecting how Googlebot crawls you for traditional search indexing. This separation matters: you can remain fully indexed for search while making a distinct decision about AI uses, or vice versa. The existence of Google-Extended is a reminder that these controls are increasingly granular, and that a thoughtful policy treats search indexing and AI uses as separate decisions rather than assuming a block or allow applies to everything a provider does.

How robots.txt controls crawlers

Robots.txt is the long-standing mechanism for telling crawlers what they may and may not access, and it is how you address AI crawlers too. You specify rules per user-agent: a block of directives for GPTBot, another for ClaudeBot, another for PerplexityBot, and so on, each allowing or disallowing paths. The mechanism is simple and well understood, but it depends on crawlers choosing to respect it — robots.txt is a widely-honoured convention among reputable operators, not an enforced access control. For the major, reputable AI crawlers, robots.txt is the right and respected tool for expressing your policy, and getting its user-agent rules right is how you turn a decision into an actual configuration.

User-agent-specific rules

The power of robots.txt for AI crawlers lies in per-user-agent rules, which let you make different decisions for different bots rather than one blanket choice. You might allow retrieval fetchers that enable citation while disallowing crawlers you associate purely with training, or allow some providers and not others. Getting this right requires knowing each crawler's user-agent string and purpose, then writing targeted rules rather than a single sweeping directive. The common failure is a lazy wildcard block or allow that treats every bot identically, which throws away exactly the granularity that lets you keep citation access while managing the uses you care about. Targeted, per-agent rules are how a deliberate policy is actually expressed.

What blocking actually costs you

Blocking AI crawlers is a legitimate choice, but it has a direct cost that is easy to underestimate: content a retrieval crawler cannot fetch cannot be cited in that engine's answers. For a site trying to build AI visibility, blocking the relevant fetchers is close to self-defeating — you are opting out of the channel you are trying to win. There are real reasons a site might choose to block (concerns about training use, server load, content protection), and those reasons can be valid. But the decision should be made with the cost in full view: you are trading away eligibility to be a cited source. Too many sites block reflexively without realising they have just excluded themselves from the answers they wanted to appear in.

Why some sites still choose to block

Blocking is not always wrong, and it helps to understand the legitimate reasons. Some publishers object to their content being used to train models they receive nothing from. Some have genuine concerns about server load from aggressive crawling. Some have content they consider proprietary or sensitive enough to keep out of AI systems entirely. These are real considerations, and for some businesses the trade is worth it. The point is not that everyone should allow every crawler; it is that the decision should weigh the specific benefit of blocking against the specific cost of lost citation eligibility, per crawler, rather than being a reflexive blanket reaction. A deliberate block is defensible; a thoughtless one usually is not.

Beware fake crawlers

Not every request claiming to be GPTBot or ClaudeBot actually is one. User-agent strings are trivial to spoof, and malicious scrapers sometimes impersonate reputable AI crawlers to evade blocks or blend in. Reputable operators publish ways to verify their crawlers — typically through the IP ranges they crawl from — so that you can distinguish genuine traffic from impersonators. If you are making decisions based on crawler activity, or seeing unexpected load attributed to an AI bot, verifying that the traffic genuinely comes from the operator it claims to be is worth the effort. Configuring your policy around a spoofed user-agent, or blaming a reputable crawler for a scraper's behaviour, leads to the wrong conclusions.

AI crawlers and crawl budget

AI crawlers consume the same finite resource that traditional crawlers do: your site's crawl capacity and server resources. A large, well-structured site generally absorbs this without trouble, but a site already struggling with crawl efficiency can feel the additional load. This connects to the resource logic we described in our piece on why thin programmatic pages get deindexed: every page and every crawler competes for finite attention and capacity. A clean, efficient site with good internal structure makes it easy for all crawlers — search and AI alike — to fetch what matters, while a bloated one wastes capacity. Crawl efficiency is good hygiene regardless of which crawlers you allow.

Managing server load

If AI crawler traffic is genuinely straining your server, the answer is usually not a blanket block but targeted management. Reputable crawlers generally respect crawl-rate guidance and robots.txt directives, so you can often moderate their behaviour without excluding them entirely. Standard infrastructure measures — caching, a content delivery network, sensible rate controls — handle crawler load for most sites without sacrificing citation eligibility. Reaching for a full block as the first response to load is usually an overcorrection that costs you more than it saves. Diagnose the actual load, verify it is genuine crawler traffic, and manage it with the normal tools before concluding that exclusion is the only option.

Read your logs

Your server logs are the ground truth about which AI crawlers are fetching your content, how often, and which pages they favour. Reading them tells you whether the engines are actually paying attention to your site, whether your robots.txt rules are having the intended effect, and whether changes you make affect crawler behaviour. As we noted in our guide to measuring AI traffic, crawler activity in your logs is a leading indicator — evidence the engines are reaching you, upstream even of citation. For any site serious about AI visibility, periodically reviewing which AI crawlers appear in the logs, and on which pages, turns your crawl policy from a guess into something you can actually observe and tune.

robots.txt and llms.txt are different tools

It is worth distinguishing robots.txt from the newer llms.txt convention, because they do different jobs. Robots.txt controls access — which crawlers may fetch which paths. Llms.txt, as we discussed in our guide to whether you should add an llms.txt, is a curated guide pointing models toward your best content, not an access control. They are complementary: robots.txt decides who gets in, llms.txt (where honoured) suggests what to prioritise once in. Confusing the two — expecting llms.txt to block anything, or robots.txt to curate — leads to misconfiguration. Use robots.txt for access decisions and treat llms.txt as the separate, optional curation layer it is.

Make sure your content is actually fetchable

Allowing a crawler is necessary but not sufficient; the crawler also has to be able to read your content once it arrives. Content that depends heavily on client-side rendering, sits behind interactions, or requires executing complex scripts may be partially or wholly invisible to a crawler that does not render it fully. If your most important content is not present in the HTML a crawler receives, allowing the crawler achieves little. Ensuring your genuinely important content is served in a form crawlers can actually read — present in the markup, not locked behind rendering a bot may not perform — is a technical prerequisite for being retrievable and therefore citable, and it is frequently overlooked.

Common crawler-configuration mistakes

A handful of mistakes recur. Blanket-blocking all AI crawlers without realising it forecloses citation. Wildcard rules that treat training and retrieval identically. Blocking a provider's entire footprint to opt out of training while also cutting off its retrieval fetcher. Configuring policy around spoofed user-agents. Assuming robots.txt enforces rather than requests. Forgetting that content hidden behind rendering is unfetchable even when the crawler is allowed. Letting robots.txt drift out of date as crawlers change. Each of these turns an intended policy into a different, usually worse, actual outcome. The cure is the same throughout: know what each crawler does, write targeted rules, and verify the result in your logs rather than assuming the configuration does what you meant.

A sensible default policy

For most sites whose goal is AI visibility, a sensible default is to allow the retrieval fetchers that enable citation, make a deliberate decision about training crawlers based on your own view of that trade, keep search indexing fully open unless you have a specific reason not to, and manage load through infrastructure rather than exclusion. This keeps you eligible to be cited — the thing you are presumably trying to achieve — while letting you express any genuine reservations about training use through targeted rules. The exact policy depends on your business, but the default posture for a site that wants to show up in AI answers should lean toward accessibility, with any blocks made deliberately and for stated reasons rather than reflexively.

Monitor and adjust over time

Crawler policy is not a set-and-forget configuration, because the crawlers themselves change: new bots appear, user-agents change, providers add controls like Google-Extended, and the relationship between crawling and citation evolves. A policy that was sensible a year ago may be leaving citation access on the table or failing to express a decision you now care about. Reviewing your robots.txt and your crawler logs periodically — checking which AI crawlers are reaching you, whether your rules still do what you intend, and whether new crawlers warrant a decision — keeps your policy aligned with both the evolving ecosystem and your own goals. Treat it as a living configuration, not a one-time setup.

The bottom line

AI crawlers are how the engines reach your content, and your robots.txt is how you decide who gets in — which means crawl policy quietly shapes whether you are even eligible to be cited. The key is to be deliberate: understand that training crawlers and live-retrieval fetchers do different things with different consequences, know the major crawlers and their user-agents, write targeted per-agent rules rather than blanket ones, keep your genuinely important content fetchable, and verify the result in your logs. For most sites chasing AI visibility, the sensible default leans toward allowing the fetchers that enable citation, with any blocks made on purpose and for reasons. The worst policy is the accidental one — and most sites have exactly that.

Frequently asked questions

What are GPTBot, ClaudeBot and PerplexityBot?

They're AI crawlers: GPTBot is OpenAI's, ClaudeBot is Anthropic's, PerplexityBot is Perplexity's. Each identifies itself with a user-agent string and can be addressed individually in robots.txt. They gather web content and/or fetch pages live to support answers, and they're distinct from traditional search crawlers like Googlebot.

Should I block AI crawlers in robots.txt?

It depends on your goal, and the decision should be deliberate. Blocking a live-retrieval fetcher means your content can't be cited in that engine's answers — close to self-defeating if you want AI visibility. There are valid reasons to block (training objections, load, sensitive content), but weigh them against the lost citation eligibility per crawler rather than blanket-blocking reflexively.

What's the difference between training crawlers and retrieval fetchers?

Training crawlers gather content to help train or update models; retrieval fetchers fetch pages live to answer a specific question right now. Blocking a training crawler affects whether your content informs a model's general knowledge; blocking a retrieval fetcher affects whether you can be cited in live answers. Conflating them leads to bad policy.

Does robots.txt actually stop AI crawlers?

For reputable operators, yes — robots.txt is a widely honoured convention, and the major AI crawlers respect it. But it's a request, not an enforced access control, and malicious scrapers can ignore it or spoof crawler user-agents. Reputable operators publish ways (usually IP ranges) to verify their crawlers are genuine.

How is llms.txt different from robots.txt?

They do different jobs. Robots.txt controls access — which crawlers may fetch which paths. llms.txt is a curated guide pointing models toward your best content, not an access control that blocks anything. They're complementary: robots.txt decides who gets in, llms.txt suggests what to prioritise once in.