llms.txt: What the File Does and How to Set It Up Correctly

The llms.txt file is a Markdown file in the root directory of your website that guides large language models (LLMs) to your most important content. It was proposed by Jeremy Howard (Answer.AI) on September 3, 2024. It does not control crawler access, but instead provides a curated content map for inference.

What is llms.txt and what is the file for?

The llms.txt file is a machine-readable Markdown file that helps LLMs understand and use a website at inference time. Jeremy Howard of Answer.AI published the proposal on September 3, 2024 on llmstxt.org. The idea: instead of making a model navigate through nested HTML, navigation bars, and advertising, it gets a curated list of the most important pages in clean Markdown.

The background is a technical problem. A language model's context window is limited, and full HTML pages are often too large and too cluttered to process. llms.txt aims to solve this problem by pointing to the most relevant content in a format that is easy for models to digest. Important: this is a community proposal, not an officially adopted web standard.

How does llms.txt differ from robots.txt and sitemap.xml?

According to the specification, llms.txt differs fundamentally from robots.txt and sitemap.xml: robots.txt governs crawler access, sitemap.xml lists all pages as a complete index, and llms.txt curates content for inference. Unlike the other two, llms.txt may also link to external pages.

File	Purpose	Audience	Format	Content
robots.txt	Control crawl access	Crawlers/bots	Text directives	Allow/disallow
sitemap.xml	Complete URL index	Search engines	XML	All pages
llms.txt	Inference aid	LLMs/answer engines	Markdown	Curated selection

The decisive difference lies in the function. A robots.txt file tells a bot where it may go. A sitemap.xml file tells it what exists. The llms.txt file suggests what is relevant. It does not replace either file, but complements them on a different level.

How is an llms.txt file structured?

A valid llms.txt file sits at the root path under /llms.txt and consists of Markdown with exactly one required element: an H1 with the project name. According to the specification, this is followed by a blockquote with a short description and optional H2 sections that list Markdown links in the format [name](url).

The structure is deliberately kept minimal. A typical skeleton looks like this: first the H1 with the name of your project or company. Directly below it, a blockquote paragraph that explains in one or two sentences what it is about. H2 headings then group thematic areas, each listing links to the most important pages. A section titled "Optional" marks content that a model may omit when context is limited.

The specification additionally recommends providing a clean Markdown version of every page at the same URL with .md appended. Extended context files such as llms-ctx-full.txt can be generated with the CLI tool llms_txt2ctx described in the specification. A real-world example: Anthropic provides a valid llms.txt file for its developer documentation with an H1, blockquote, and H2 sections, available at docs.anthropic.com/llms.txt.

How do I set up an llms.txt file correctly?

You set up an llms.txt file by curating your core content, formatting it as Markdown according to the specification, and placing the file under /llms.txt in the root directory. For a small website, the total effort is typically under an hour.

1. Select core content: Decide which pages a model should know about, such as product documentation, key service pages, and important guides. Not everything belongs in it; the curation itself is the value.
2. Write the Markdown: H1 with the project name, blockquote description, then H2 sections with [name](url) links and short explanations for each link.
3. Place it at the root: The file must be reachable at your-domain.de/llms.txt.
4. Provide .md versions: Optional but recommended: clean Markdown versions of the linked pages at their respective URL plus .md.
5. Automate it: Documentation hosting platforms and CMS plugins can generate the file automatically on request, so it stays current with every content update.

Do AI crawlers and answer engines even use llms.txt?

Currently, no provider has confirmed that its AI system evaluates llms.txt. According to Search Engine Journal, Google's John Mueller put it this way: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)." In other words, according to Mueller, server logs show that AI services do not even fetch the file. This is the most important and most honest assessment of the topic, also picked up by Search Engine Roundtable.

The known AI crawlers from OpenAI and Anthropic follow robots.txt; no provider has officially confirmed that they evaluate llms.txt. Even Anthropic, which maintains an exemplary llms.txt for its documentation, makes no official commitment that its own crawlers evaluate the standard. So far, the file's spread has mainly come from automation: the documentation platform Mintlify announced automatic support for /llms.txt and /llms-full.txt on November 20, 2024, which means hosted documentation sites get the file without any manual effort.

Does llms.txt bring SEO or GEO benefits?

There is currently no evidence that llms.txt improves AI retrieval, increases traffic, or raises model accuracy. This is the conclusion reached by Ahrefs (Ryan Law) after reviewing the available data: "There's no evidence that llms.txt improves AI retrieval, boosts traffic, or enhances model accuracy."

John Mueller also compared llms.txt to the outdated keywords meta tag: a signal controlled by the site operator and therefore easy to manipulate, which search engines have ignored for over a decade. This skepticism is justified, because a signal that anyone can set arbitrarily is not very trustworthy for an answer engine. Anyone who wants to be cited in AI answers is far better off with substantial, well-structured optimization than with an llms.txt file. Our article on Generative Engine Optimization explains how that works in practice.

Who does llms.txt still make sense for today?

Today, llms.txt is worthwhile mainly for documentation and developer sites, because platforms like Mintlify have been generating the file automatically since November 2024. The effort is low, the risk is practically zero, and search interest is measurably present.

Demand for the topic is real: according to Ahrefs Keywords Explorer (as of 2026), "llms.txt" gets 1,100 searches per month in Germany; the Ahrefs global demand summed across all countries is around 17,000, plus related keywords such as "llms.txt generator" (150) or "create llms.txt" (30). This shows that site operators are actively engaging with the file. So treat llms.txt as a cheap experiment, not a ranking guarantee. If a generator or plugin maintains the file anyway, there is little reason not to provide it, and some reason to, in case answer engines evaluate the standard in the future. For sustainable visibility in search and AI answers, however, what still counts is substantive content, which is what we build in SEO marketing.

Frequently asked questions about llms.txt

Is llms.txt an official standard?

No. llms.txt is a community proposal that Jeremy Howard published on llmstxt.org on September 3, 2024. It is not a web standard adopted by any standards body, and no search engine or AI evaluates it as binding.

Does llms.txt replace robots.txt?

No. robots.txt governs crawler access and remains essential, while llms.txt merely curates content for inference. Both files serve different purposes and are not mutually exclusive.

Does an llms.txt file hurt my SEO?

Based on current knowledge, no. Search engine crawlers do not use the file for ranking, and it sits separately at the root. A benefit, however, is just as unproven.

Where does the llms.txt file need to be placed?

In the root directory of your domain, reachable at your-domain.de/llms.txt. According to the specification, placing it in a subpath is optionally possible too, but the root path is the convention.

What is llms-full.txt?

llms-full.txt is an extended variant that contains the full content directly instead of just linking to it. Mintlify announced it as a platform feature together with /llms.txt support on November 20, 2024. This is distinct from llms-ctx-full.txt, described in the official specification, which can be generated from an llms.txt file with the CLI tool llms_txt2ctx.

Sources

• llmstxt.org: The /llms.txt file (official specification), Jeremy Howard, September 3, 2024
• Anthropic: Developer Documentation llms.txt
• Mintlify Blog: Simplifying docs for AI with /llms.txt (November 20, 2024)
• Search Engine Roundtable: Google Says No AI System Currently Uses LLMs.txt (John Mueller)
• Search Engine Journal: Google Says LLMs.txt Comparable To Keywords Meta Tag
• Ahrefs Blog: What Is llms.txt, and Should You Care About It? (Ryan Law)
• Ahrefs Keywords Explorer (country=de, accessed 2026-06)

As of: June 2026. This article is general professional information on the topic of llms.txt and does not replace individual technical advice. The llms.txt proposal is a young community standard not adopted by any standards body; a concrete benefit for visibility or ranking is not supported by the current data. Author: Alexander Weipprecht.