llms txt vs robots txt, llms.txt, robots.txt, AI SEO, answer engine optimization

Llms Txt vs Robots Txt: A Practical Guide

Written by LLMrefs Team • Last updated November 22, 2026

Robots.txt is a voluntary crawl-access file and llms.txt is an optional Markdown context file, so neither one replaces the other.

The scale gap shows how different their roles are. Google's robots.txt cache tracks more than 4 billion hosts, according to the IETF Robots Exclusion Protocol presentation. A valid llms.txt file appeared on 5.61% of the top 10,000 websites by June 2026, according to the llms.txt adoption survey, leaving roughly 94.39% of that sample without one.

That comparison is useful, but it can mislead. A widely adopted file still does not answer the same question as a newer, optional one. robots.txt tells crawlers which paths they may fetch. llms.txt gives an AI system a short, curated map of pages worth considering. Treat the first as access policy and the second as extra context, and the implementation choices become much clearer.

The most repeated advice in AI SEO is that llms.txt is "robots.txt for AI" and that publishing one is how you earn citations in ChatGPT answers. Both halves of that claim waste effort if taken at face value. The files are read by different software, at different moments, for different purposes. A llms.txt file may help an agent find relevant public pages. It does not grant or deny access, and it does not guarantee that a model cites those pages.

The practical question is narrower and more useful. Which named crawler requests llms.txt on your domain, and do the URLs you list start appearing in AI answers after you publish it? Server logs answer the first part. Citation tracking across engines answers the second. Without those two checks, llms.txt is easy to overcredit and just as easy to dismiss.

Robots.txt deserves the stronger operational stance. If a crawler should not fetch a path, that decision belongs in robots.txt, expressed in a syntax crawlers already understand. llms.txt belongs on sites that want to make their most useful public pages easier for an AI system to identify, especially when documentation, product pages, or editorial hubs are scattered across a large site.

The wrong conclusion is that every site now needs a second policy file. The better conclusion is that each file has a separate job, and each job needs separate evidence. Robots.txt governs access. llms.txt supplies optional guidance. Citations depend on how AI systems retrieve, read, and ground answers, not on the mere presence of a file at /llms.txt.

Llms.txt Versus Robots.txt Side by Side

Reach for robots.txt when the job is declaring which crawlers may fetch which paths. Reach for llms.txt when the job is handing an AI system a short, curated map of the pages worth reading. Google's own robots.txt specification frames the first file as a set of rules for crawlers, which is the right home for every access decision you make.

The two aren't versions of each other, and the standards history explains why. RFC 9309 formalized the Robots Exclusion Protocol in September 2022, approximately 28 years after Martijn Koster proposed it in 1994 to stop crawlers from overloading sites. llms.txt arrived in September 2024 as a machine-readable summary an agent may consult on demand. One governs crawler access policy. The other supplies an optional discovery and context layer.

Llms.txt versus robots.txt on the decisions that matter

Criterion robots.txt llms.txt
Purpose Voluntary pre-crawl access preferences Curated on-demand context and a map of useful public pages
Origin and standard Proposed in 1994 by Martijn Koster; RFC 9309 published September 2022 Proposed September 2024; informal, no finalized RFC or media type
Location /robots.txt at the host root /llms.txt at the root, or a subpath such as /docs/llms.txt
Format Plain text with user-agent groups and Allow or Disallow rules Markdown with a required H1, optional summary, and H2 lists of described URLs
Permission effect Allow and Disallow are preferences, not authentication or a security boundary Listing a URL does not grant access; omitting a URL does not block it
Who is expected to read it Crawlers, before they fetch paths Language-model agents, if they choose to request the file while answering
Compliance in 2026 Broadly recognized and still voluntary Emerging and uneven; major AI platforms have not committed to consistent support
SEO job Control declared crawl access, including named AI bots Optional prioritization of public pages, not a citation or ranking switch

Three mix-ups the table prevents

A listed URL is not a permission slip. Listing a page in llms.txt does not grant permission to fetch it, and leaving a page out does not block access. The permission question stays with the access file, every time.

A Disallow doesn't unlearn anything. A rule stops a crawler from fetching a path going forward. It doesn't remove a page from a model trained on an older crawl, and it may not stop a different product crawler or a licensed index from retrieving the same public URL.

An HTTP 200 is an availability metric. A successful response proves the file is served. It doesn't prove an answer engine requested it, read it, or cited your site.

Practical rule: label every change you're planning as either an access rule or a content hint, then put it in the file that owns that job.

A publisher can disallow /private/ for a named crawler in robots.txt while pointing llms.txt at /docs/overview/ and /research/whitepapers/. The first sentence restricts fetching. The second says which public pages deserve an agent's attention. Pairing them lets you communicate access rules and editorial priorities without pretending they're the same instruction.

How Widely Each File Is Actually Used

Robots.txt has had roughly three decades to become a convention. The 2025 IETF presentation on the Robots Exclusion Protocol reports that the protocol is used by more than 4 billion hosts tracked in Google's robots.txt cache, which is infrastructure-scale deployment rather than a niche habit. Published robots.txt files are also common on heavily trafficked pages, while explicit crawler rules are less common than broad wildcard policies.

The newer file operates at a different order of magnitude, and the surveys don't agree with each other because they aren't measuring the same population.

A timeline graphic showing the evolution of robots.txt from a 1994 practical solution to a standard.

Adoption signals for both files, 2025 to 2026

Measurement Sample Finding Year
Google robots.txt cache Hosts tracked in the cache More than 4 billion hosts 2025 IETF presentation
High-traffic landing pages 11,100 pages Robots.txt is widely published, while named crawler rules are less common than wildcard-only policies 2026
Valid llms.txt on large sites Top 10,000 websites Early adoption in 2025, with higher but still uneven adoption by 2026 2025 to 2026
Readable llms.txt and AI blocks 1,000 top reachable domains 18.7% served readable llms.txt; 21.6% blocked at least one AI crawler (SEO Vision survey) 2026
Llms.txt and llms-full.txt 10,000 high-ranking websites 15.2% served llms.txt; 2.5% also served llms-full.txt (SEO Score API analysis) 2026
Broad domain crawl Nearly 300,000 domains 10.13% had an llms.txt file (SE Ranking study) 2026
Cloud software companies 94 reachable firms in the 2025 Forbes Cloud 100 62%, or 58 companies, served a valid llms.txt (Flux adoption study) 2026
AI-bot rules inside robots.txt Top 10,000 domains that already have robots.txt About 14% had rules targeting AI bots (Cloudflare figures) June 2025

Read the rows separately: these are different samples with different validation rules and dates, so subtracting one percentage from another produces a number that means nothing.

The spread inside the llms.txt rows is the useful signal. Reachable cloud companies show much stronger adoption than the web at large, where broad crawls put llms.txt near one site in ten. Practice concentrates where documentation matters commercially: API platforms, developer tools, and technical SaaS. That's a description of who benefits from curation, not proof that the file earns citations.

For your own site, the split resolves cleanly. Robots.txt has a mature parsing contract and a long history of crawler use. Llms.txt is an additional, evolving signal about how your content is organized, and treating it as a replacement for crawl controls confuses a hint with a rule. The useful question is narrower and more practical: which named crawler fetches llms.txt, and do citations change after you publish it?

Syntax Rules and a Worked Example

The two files sit at the same URL level and solve different problems. Robots.txt declares crawler access rules. Llms.txt describes a curated set of pages in Markdown so language models can understand what matters on a site.

The robots.txt format in practice

A minimal robots.txt file is a set of user-agent groups. Each group starts with User-agent: and carries Allow: or Disallow: path rules. A wildcard group sets default policy, and any bot that needs different treatment gets its own group. A named group replaces the wildcard for that crawler rather than adding to it.

Parsing follows RFC 9309, which matters when a rule appears valid but behaves differently in logs. A file at the root is a preference declaration, not a lock. If a directory needs real protection, keep it behind authentication and avoid publishing its URLs. For a walkthrough of path syntax and the mistakes that block pages by accident, this guide on creating a robots.txt file correctly covers rule order and common failure modes.

A minimal group set: User-agent: * with Disallow: /private/ and Disallow: /draft/, then a second group for a named training crawler that disallows the same two paths and adds Allow: /docs/.

The llms.txt format in practice

The llms.txt proposal defines a Markdown discovery file, not an access-control mechanism. It belongs at /llms.txt, though a subpath such as /docs/llms.txt can describe content beneath that path. The format requires one H1 with the site or project name. It then allows a short summary, additional context, and H2 sections whose list items pair a URL with a description.

The distinction is easy to miss because both files are plain text and both are read by machines. The difference shows up in how strict they are. A crawler that honors robots.txt can treat Disallow as a rule. A model or retrieval system that reads llms.txt gets a suggested map, with no guarantee that it fetches the file, follows its structure, or changes the answers it produces after doing so.

For a fictional Acme API documentation site, the two files look like this:

  • Robots.txt lines: User-agent: *, Disallow: /private/, Disallow: /draft/, then User-agent: GPTBot, Disallow: /private/, Disallow: /draft/, Allow: /docs/.
  • Llms.txt opening: # Acme API, followed by two sentences on what the API does and who it's for.
  • Llms.txt sections: ## Authentication, ## Rate limits, ## SDKs, ## Troubleshooting, each with one canonical page and a one-line description of why it matters.

That example exposes the practical division of labor. The robots.txt lines tell named crawlers which paths they may request. The llms.txt lines identify the pages a model should understand first. If /draft/ is disallowed, listing a draft URL in llms.txt creates a contradiction for any crawler that follows the access rule. If /docs/ is allowed but absent from llms.txt, the page remains available for discovery through ordinary crawling.

Write each description so it stands on its own. The point of the file is to let a model quote a compact summary without ingesting an entire page. A Markdown list cannot override a Disallow, and a Disallow never writes a summary. Some sites also serve an llms-full.txt companion that bundles full page content. Treat that as a rarer extra, not a second required standard.

Before you upload, check five things.

  1. One H1 only, containing the site or project name.
  2. No private or draft URLs anywhere in the file.
  3. Descriptions a model can quote without needing the full page.
  4. Every link resolves to a live canonical URL, not a redirect chain.
  5. Robots.txt still allows each of those public paths for the crawlers you care about.

The fifth check is where syntax becomes measurement. Valid Markdown can still point to pages a target crawler refuses to fetch, and a permissive robots.txt group can still fail to produce a citation. The file pair can be technically correct while the answer-level outcome stays flat.

Training Search and Citation Are Separate Choices

The useful question is not which file replaces the other. It is which named crawler fetches llms.txt and whether citations change after you publish it. "Robots.txt blocks, llms.txt recommends" hides four independent decisions. AI vendors often run separate bots for training, search retrieval, dataset construction, and user-triggered browsing, so a single Disallow is not an organization-wide AI-use policy. Research on the one million most visited websites found that 55.6% deployed robots.txt at the root or subdomain level, and per the same 2026 study of robots.txt gatekeeping on arXiv, more than 25% of the top 1,000 sites blocked OpenAI's web crawler. That study argues the gatekeeping creates an accessibility asymmetry that can shape LLM training data and raises transparency and data-ethics concerns.

The asymmetry a block hides

Blocking a training crawler can reduce future inclusion in that dataset without stopping a search product from retrieving and citing the same public page through another crawler or a licensed index. The reverse is also true: allowing access doesn't guarantee attribution, an accurate summary, or a favorable mention. This split is why llms.txt should be treated as optional context, not a second permission file.

Policy matrix for training, retrieval, and citation

Objective robots.txt action llms.txt action Neither file guarantees
Keep a training crawler off selected paths Disallow those paths for that named user-agent Do not list the paths A different bot or a licensed index may still retrieve public pages
Allow search retrieval Allow the paths or leave them uncovered Optional; listing does not create permission Inclusion in an AI answer or a citation
Steer agents to preferred public pages Leave those paths allowed H1, short summary, and described links to canonical URLs Accurate summary, attribution, or rank
Protect private or draft URLs Disallow the path; do not rely on obscurity Omit the URL entirely Security, because robots.txt is not authentication
Express a full AI-use policy One group per crawler and purpose Cannot encode permission or licensing Consistent behavior across every AI product

Walk the rows once and the work becomes concrete. Excluding a named training bot is a robots.txt edit and nothing else. The honest expectation is that a different bot may still fetch the page. Keeping pages retrievable for search usually means leaving them uncovered rather than adding an explicit Allow, since a broad Allow can accidentally re-permit a path you blocked elsewhere. Steering agents toward canonical explainers is the one row where llms.txt does useful work, and the outcome you can't promise is an accurate summary or a mention.

"llms.txt cannot replace those controls because its proposed role is curatorial rather than permission-based, and major AI systems have not established consistent adherence."

The record you keep for each row should name the crawler, its purpose, the path scope, and whether the goal is exclusion, discoverability, or citation. That record matters because the same page can be blocked for one bot, retrievable through another, and summarized in an answer with no citation at all. When you audit whether those claims hold up in practice, keep the record close to the crawl evidence: crawler name, request path, response code, and whether the answer changed after publication. A team can use a source-checking guide for MUN delegates as a training exercise in tracing claims back to primary records, then apply the same discipline to crawler logs, file responses, and cited URLs.

When Each Kind of Site Should Publish Both

Five site types, five calls, no new syntax to learn.

A diagram comparing robots.txt and llms.txt files for managing web crawler access to documentation and APIs.

Documentation or API product: publish both. Robots.txt keeps /draft/ and private endpoints out of reach. A short llms.txt maps authentication, rate limits, and SDKs so an agent is less likely to quote a changelog entry from 2021 as your current behavior.

Publisher with original research: publish both, with a narrow file. Allow search crawlers on public articles, disallow a named training crawler only where your policy requires it, and list a handful of cornerstone explainers rather than every post you've ever written. A 400-URL file is a sitemap with extra steps.

Local service or small brochure site: robots.txt only, for now. Ship a correct robots.txt and skip llms.txt until you have a stable set of pages worth summarizing. An empty or auto-generated file adds maintenance with no reader.

Enterprise with many country sites: publish both, one curated file per property. Access rules belong on each host. If you use llms.txt, keep a single curated file per property instead of dumping a global sitemap across locales, because the file's value is editorial judgment about what matters.

A site that already blocks all AI crawlers: don't expect citations to return. A guidance file cannot override an access rule, so publishing llms.txt behind a full block changes nothing.

The shared test is the same for all five: which named bots later request the file, and whether the cited URLs are the ones you listed. Before you rely on either file, run the target URLs through an AI crawlability checker and then watch logs for the actual user-agents, because the useful evidence is a fetch pattern, not a publish date. An IETF draft on LLM context discovery notes there's no universal discovery mechanism, so a consumer has to already know or guess where the file lives. Server-log observations so far put /llms.txt requests at very low rates, and they have not shown that highly cited domains are reliably the ones publishing it.

Treat llms.txt as optional context that can help routing and documentation, while robots.txt remains the access mechanism. Publishing may clarify what you want surfaced. It cannot promise training exclusion, citation placement, or a visibility lift.

How to Publish Both Files Together

Sequence matters more than syntax. Access rules land before the content map, because a map that points at blocked URLs is worse than no map.

A two-step infographic showing how to publish files by inventorying URLs and editing robots.txt for AI access control.

  1. Inventory URLs into three buckets. Public canonical, private, and thin. Write the AI-use objective for each bucket before you touch a file, because the objective decides the rule.
  2. Edit robots.txt with one group per crawler that needs a policy different from the wildcard. Disallow private paths and confirm your important public URLs aren't caught by a wildcard rule you inherited.
  3. Draft llms.txt in Markdown with a single H1, a summary under 100 words, and only the public URLs a human would hand to a new teammate. Every entry gets a one-line description.
  4. Generate or validate the draft with a dedicated llms.txt generator rather than pasting a sitemap, then read every link yourself.
  5. Place the files at the host root. Use a subpath llms.txt only when the file should describe that section and nothing else.
  6. Fetch both URLs and record status code, content type, and link resolution. Confirm each listed link returns 200 and is allowed by robots.txt.
  7. Watch server logs for named AI user-agents on /robots.txt and /llms.txt for several weeks before you claim any effect.

Two conflicts cause frequent trouble. A disallowed URL must not appear in llms.txt, and a sitemap-length llms.txt recreates the problem the file exists to solve. A clean file is still a controlled test, since there's no formal RFC for llms.txt and no commitment from major AI platforms to read it.

For teams documenting agent-facing conventions, the Just Cook It agent guidelines are a useful reference for how a public site can describe itself clearly to automated readers without overstating what those readers will do.

Drafting gets faster with the right tools. LLMrefs offers free resource tools including an LLMs.txt Generator, Reddit Threads Finder, AI Crawlability Checker, ChatGPT Prompts Database, Query Fan-Out Generator, AI Content A/B Tester, ChatGPT Query Extractor and ChatGPT Search Audit, so a team can build the file and then verify which important URLs the crawlers you intend to allow can reach.

Measure Mentions Before You Credit the File

Treat the launch as a controlled test, not a ranking change. Before upload, freeze a keyword set and record citations, brand mentions, and the URLs answer engines show, split by model, country, and language, along with the publish date. That baseline is the only thing that makes the after picture readable.

After launch, rerun the same query set and read server logs for named AI user-agents on /llms.txt. Status codes, response headers, timestamps, and link resolution belong in that log as plumbing detail. Watch four fields instead of a single number.

  • llms.txt fetch rate: requests from named AI user-agents, separated from generic crawlers.
  • Target-page citation rate: how often the pages you listed appear as sources in answers.
  • Citation share versus the control period: your share of citations in the same answer set, before and after.
  • A brand-visibility change that survives a before-and-after check: a shift that holds when you rerun by model, geography, and language.

LLMrefs tracks keywords rather than one-off prompts across ChatGPT, Google AI Overviews, AI Mode, Gemini, Perplexity, Claude, Grok and more, aggregates responses, citations, and brand mentions, and turns them into share-of-voice comparisons against competitors, so the readout stays stable and comparable instead of becoming a screenshot of one chat. A null result is useful news, and a method for measuring AI visibility that you repeat on schedule beats any single launch review. Review citation share on the same keywords at your next scheduled update and let the trend answer the question the file alone can't.


The two files do different jobs, and only one of them can be measured in a dashboard you trust week to week. LLMrefs monitors brand visibility across the AI search engines your customers use, benchmarks your share of voice against competitors, and shows which sources get cited so you can tell whether a published llms.txt changed anything. Start free at LLMrefs and set your first keyword set before you upload the file.