llms.txt, AI Crawlers and robots.txt: What Owners Should Know

By PDS Team

An llms.txt file is optional, and Google says its Search features, including AI Overviews and AI Mode, do not use it. What actually decides whether AI tools can read your pages is robots.txt, where each AI company has its own crawler names you can allow or block. This guide explains the main AI crawlers, what blocking each one does, and where llms.txt fits.

What is llms.txt and does Google use it?

llms.txt is a proposed plain-text file, placed at the root of a site, that lists a site’s key pages in Markdown for large language models, but Google Search does not use it.

Google’s guide to optimising for generative AI features in Search, published on Search Central in May 2026, has a mythbusting section that says you don’t need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search. It adds that doing so will neither harm nor help your visibility or rankings in Google Search. In June 2026 Google added a note to the same guide stating plainly that Google Search does not use llms.txt files.

Google’s “AI features and your website” page makes the same point from another angle: there is no special schema.org markup needed for AI Overviews or AI Mode. Pages simply need to be indexed and eligible to show a snippet.

That does not make llms.txt harmful. Google’s guide notes that sites can still create such files for other services that use them. Treat it as a small extra, not an SEO task. If you are unsure what AI systems can and cannot see on your site today, our AI visibility audit checks crawler access, indexing and how your brand appears in AI answers.

Which AI crawlers should you know about?

The main AI platforms each run separate bots for training, for search and for fetching pages when a user asks, and each one can be controlled separately in robots.txt.

In short: training bots feed future models, search bots decide whether you can be cited in AI answers, and user-triggered fetchers act for a person in a chat.

What does blocking each type of crawler actually change?

Blocking a training bot keeps your future content out of training data, while blocking a search bot can remove you from that platform’s AI search answers, so the two should be decided separately.

Training bots

Anthropic’s crawler documentation says that if a site blocks ClaudeBot, its future materials should be excluded from Anthropic’s training datasets. OpenAI describes disallowing GPTBot as a signal that your content should not be used for training.

Search bots

OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot prevents its system from indexing your content for search, which may reduce your visibility in Claude’s results. Perplexity recommends allowing PerplexityBot if you want to appear in its search results.

User-triggered fetchers

These behave differently. OpenAI says robots.txt rules may not apply to ChatGPT-User because the visits are initiated by a user. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic, by contrast, says blocking Claude-User prevents its system from retrieving your content in response to a user query.

How does Google-Extended work, and does it affect Google Search?

Google-Extended lets you opt out of your content being used for Gemini training and grounding, and Google says it has no effect on Google Search inclusion or ranking.

Google’s list of common crawlers describes Google-Extended as a standalone product token. It is not a separate bot: Google still crawls with its usual user agents, and the token tells Google how it may use what it crawled. Google states that Google-Extended does not impact a site’s inclusion in Google Search and is not a ranking signal.

The key point for business owners is that Google-Extended does not control AI Overviews or AI Mode. Those features are part of Google Search. Google’s AI features page says the controls there are the ones you already know: Googlebot rules in robots.txt, plus nosnippet, data-nosnippet, max-snippet and noindex. Blocking Googlebot to avoid AI Overviews would also remove you from normal search results.

For more on how pages get picked for those summaries, see our guide to SEO for Google AI Overviews.

How do you write robots.txt rules for AI crawlers?

Add a separate group for each user agent you want to treat differently, and leave everything else to your normal rules.

A common choice for a business site is to stay visible in AI search while opting out of model training. That looks like this:

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /

Others simply allow everything. It is a business decision more than a technical one.

Practical points from the official docs

• OpenAI says it can take around 24 hours after a robots.txt update for ChatGPT search to adjust. Perplexity says up to 24 hours.
• Anthropic says rules must be set for each subdomain, and that blocking IP addresses may not reliably opt you out. It also supports the non-standard Crawl-delay rule for ClaudeBot.
• OpenAI notes that if you allow both OAI-SearchBot and GPTBot, it may use one crawl for both purposes.
• Check that a security plugin, CDN or firewall is not blocking the bots you meant to allow. A robots.txt “Allow” does nothing if the firewall returns an error.

Robots.txt is only one part of crawling and indexing. Our technical SEO checklist covers sitemaps, canonicals and the rest.

What about the robots meta tag and X-Robots-Tag?

The robots meta tag controls how a page appears in search results, while robots.txt controls crawling, and you usually need both working together.

A <meta name="robots"> tag applies to all search crawlers that support it. A <meta name="googlebot"> tag applies only to Google. For files such as PDFs, the same rules can be sent as an X-Robots-Tag HTTP header. In Google, nosnippet and max-snippet limit how much text can be shown, which also limits use in AI Overviews and AI Mode.

One common mistake: if robots.txt blocks a page, Google cannot see a noindex tag on it. To remove a page from Google, allow crawling and use noindex.

Should you add an llms.txt file anyway?

Only if a specific tool or service you rely on reads it; for Google Search it makes no difference either way.

If you do add one, keep it short, accurate and in sync with your sitemap. It should never replace clear pages, good internal links and correct robots.txt rules.

AI crawlers and llms.txt FAQs