Perplexity’s own documentation says its fetcher ignores robots.txt

“this fetcher generally ignores robots.txt rules”

That is Perplexity, describing Perplexity-User on its own crawler documentation page. Not a critic’s characterisation. The vendor’s.

It draws the line most writing about AI crawlers smudges. Crawlers that gather training data are governed by robots.txt and their operators say so. Fetchers that run because a human has just typed a question and is waiting for an answer largely are not, by design, and their operators say that too. Same documents, a few paragraphs apart. Carry that distinction and most advice about “blocking AI” sorts itself into the part that works and the part that cannot.

Short piece. Three things are documented, one is not, one is proposed.

Documented: who fetches what

OpenAI documents four agents on Overview of OpenAI Crawlers. GPTBot is “used to make our generative AI foundation models more useful and safe.” OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features.” Of the third, the page says OpenAI “also uses ChatGPT-User for certain user actions in ChatGPT and Custom GPTs,” and that it “is not used for crawling the web in an automatic fashion.” OAI-AdsBot validates the safety of pages submitted as ads. One operational figure: “it can take ~24 hours from a site’s robots.txt update for our systems to adjust.” IP ranges are published as JSON at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json, all three of which returned 200 when I fetched them on 5 August 2026.

Anthropic documents three: ClaudeBot for training data, Claude-User for user-initiated fetches, and Claude-SearchBot for search indexing. The support page states that “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt,” gives a Crawl-delay: 1 example, and links claude.com/crawling/bots.json for source-IP verification (Does Anthropic crawl data from the web?).

Google-Extended is the one people misread, and the misreading has a consequence. It is not a crawler. Google’s own documentation: “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity” (List of Google’s common crawlers, last updated 14 July 2026). You will never find Google-Extended in your access logs, because there is nothing there to find. It is a switch, not a visitor. Block it, then check your logs for a change. You will see none, and conclude, wrongly, that it was ignored.

Perplexity documents PerplexityBot, which is “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models,” alongside the Perplexity-User line at the top of this page (Perplexity Crawlers).

Documented: what robots.txt is not

The Robots Exclusion Protocol is a real standard: RFC 9309, Koster, Illyes, Zeller and Sassman, September 2022. Three provisions are worth knowing before you lean on it.

It is not a security control. The Security Considerations section says so directly: “The Robots Exclusion Protocol is not a substitute for valid content security measures.” Anything reachable without authentication is reachable.

It is cached. “Crawlers SHOULD NOT use the cached version for more than 24 hours,” which lines up with OpenAI’s stated ~24-hour adjustment window. Deploying a robots.txt change and expecting immediate effect is a category error.

It is size-limited. “The parsing limit MUST be at least 500 kibibytes.” Rules past that boundary have no guarantee of being read at all.

Not documented: whether any of them run JavaScript

The question I get asked most, and none of the operators has answered it. On 5 August 2026 I fetched the four crawler documentation pages named above and searched their rendered text for the string “JavaScript”. Anthropic’s, Google’s and Perplexity’s contain it zero times. OpenAI’s contains it once, in the left-hand navigation tree, as a link to an unrelated “JavaScript Pixel” page in the advertising docs.

The only party documenting rendering is Google Search, and it documents it for Googlebot rather than for anything AI-specific: “Googlebot queues all pages with a 200 HTTP status code for rendering, unless a robots meta tag or header tells Google not to index the page. The page may stay on this queue for a few seconds, but it can take longer than that” (Understand the JavaScript SEO basics, last updated 4 March 2026).

So the defensible statement is a narrow one: rendering is documented for Googlebot and undocumented everywhere else. My practical conclusion (inference, not documentation) is to put the substance in the initial HTML response, since that is the only path every fetcher demonstrably has.

Proposed, not adopted: llms.txt

llms.txt was published by Jeremy Howard on 3 September 2024, and it describes itself accurately, which is the part people skip. The site’s own subtitle: “A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.” Further down: “the llms.txt specification is open for community input.” It specifies a Markdown file with an H1 project name, an optional summary blockquote, and H2 sections containing lists of links.

A proposal. Rather than assert that no major provider has adopted it, here is the check, which takes about two minutes: of the four crawler documentation pages cited above, Anthropic’s and Google’s contain the string “llms.txt” zero times. OpenAI’s and Perplexity’s do contain it. In both cases every occurrence is a link to that documentation site’s own /llms.txt index, offered to agents reading the docs. developers.openai.com/llms.txt and docs.perplexity.ai/llms.txt both return 200.

That distinction carries the whole section. Two providers publish an llms.txt for their documentation. Neither one’s crawler documentation says its crawlers read yours. Serving a file and consuming a file are different behaviours, and the claim “OpenAI supports llms.txt” collapses them.

None of which is an argument against publishing one. It costs almost nothing, and a Markdown index is genuinely useful to an agent that fetches it. The argument is against believing that filing it does anything by itself, and against the second failure mode, which is writing claims into a machine-readable file that your pages do not support. Two of mine did that. /.well-known/ai-plugin.json pointed at an openapi.yaml of 189 bytes whose paths object was empty: a plugin manifest advertising an API with no operations in it, filed at the well-known path where machines go looking. Searching OpenAI’s current Plugins documentation on 5 August 2026, the strings ai-plugin.json and well-known appear zero times. The page describes something built on skills and the Model Context Protocol instead. A dead spec, implemented badly. The ai.txt and llms.txt next to it were worse, because they were live and false: they advertised “19 products tracked, 171 comparison pages, updated hourly” and “public review data from 10+ sources,” about a benchmark that fails most of its own published promises.

The direction of travel on access runs the opposite way from manifests. On 1 July 2025 Cloudflare announced it was “changing the default to block AI crawlers unless they pay creators for their content”, citing referral ratios: “With OpenAI, it’s 750 times more difficult to get traffic than it was with the Google of old. With Anthropic, it’s 30,000 times more difficult.” Whatever you make of the policy, “what will read my site” is now answered partly by your CDN and not only by your robots file.

Inference: what actually gets you cited

Argued, not documented. Read it accordingly.

Nobody outside the labs knows how retrieval weights or selects a page. Every provider above documents which agent fetches what and how to block it. None documents what happens next, so anyone offering you the ranking factors for LLM citation is describing a system whose operators have published nothing whatsoever about its ranking factors.

What can be reasoned about is narrower. To repeat a page without hedging, a model needs a specific claim, attached to a named source and a date, that resolves to something a reader can open. A manifest describing you favourably supplies none of that: it is self-assertion, and self-assertion is the cheapest text there is: the exact output a language model generates without limit at no cost, which is why it carries no weight coming from you either.

So the durable move is to be the primary source of a fact somebody needs. Publish the number you measured, the method you used, the date. A fact available only from you can be cited; a restatement of what a hundred other pages already say cannot, and no file in your web root changes that. I am fairly confident about that and I cannot prove it. Inference, labelled as inference. That is the arrangement I should have used the first time.