How does robots.txt work?

A recent shift by a membership platform from polite crawler instructions to active AI bot blocking highlights a core web concept: robots.txt was never a lock. It is a coordination mechanism, and that distinction matters when creator archives, documentation, and community posts become valuable AI data.

Why this matters now

For years, many website owners treated robots.txt as the standard way to tell crawlers what should and should not be fetched. That worked reasonably well for cooperative search engines because both sides benefited: sites got discoverability, and search systems got structured permission signals.

AI changes the stakes. A crawler is no longer only indexing pages so people can find them later. It may be collecting text, images, code, transcripts, or discussions that can be used to train models, build retrieval systems, or enrich commercial products. Once content is absorbed into a dataset or embedding pipeline, the original owner may have little visibility into where it travels.

So the practical question has shifted from can I express my preference? to can I enforce access? Robots.txt helps with the first question. It does not solve the second.

How it works

Robots.txt is a plain text file placed at the root of a website, usually at a path like robots.txt. It contains crawl rules for bots, often grouped by bot name. A cooperative bot requests the robots txt file, reads the crawl rules, decides which allowed URLs it may fetch, and then crawls page content accordingly.

  Bot requests robots txt file
     │
     ▼
  Reads crawl rules
     │
     ▼
  Chooses allowed URLs
     │
     ▼
  Fetches page content

The key word is cooperative. Robots.txt is part of the Robots Exclusion Protocol, a web convention rather than an access control system. It does not authenticate the bot, encrypt the content, block network traffic, or prove that a crawler obeyed the rule. A bad actor can ignore it. A poorly configured bot can misunderstand it. A scraper pretending to be an ordinary browser may never check it at all.

That is why robots.txt is often compared to a sign on a door, not the door itself. It can say do not enter, but it cannot stop entry. Real enforcement usually requires additional controls: login requirements, paywalls, rate limits, bot detection, network firewalls, token based APIs, contractual terms, or content delivery rules that deny suspicious requests before the page is served.