llms.txt explained: what it is and how to write one

llms.txt is a markdown file placed at the root of your website, at /llms.txt, that tells AI tools what your site is about and links to your most useful pages. It is a proposed convention, documented at llmstxt.org. It does not block or allow crawling. It is a short, curated reading list for language models.

This guide covers the format, a full example, how it differs from robots.txt and sitemaps, and an honest view on whether it is worth your time.

Why llms.txt exists

Web pages are built for people and browsers. They carry navigation, scripts, cookie banners and layout code. A language model reading a page has to dig the useful text out of all that, often with a limited context window.

The llms.txt proposal, published by Jeremy Howard in 2024, suggests a simple fix: give models a clean markdown file that summarizes the site and links to the pages that matter, ideally with markdown versions of those pages too. The llmstxt.org spec describes it as information to help LLMs use a website at inference time, meaning when a user asks a question, not during training.

Typical uses:

• Developer documentation, where coding assistants need accurate API details.
• Product sites that want AI tools to describe them correctly.
• Knowledge bases and help centers with many pages.

The llms.txt format

The spec defines a small, ordered structure in markdown:

1. An H1 with the project or site name. This is the only required part.
2. A blockquote summary. One or two sentences with the key facts.
3. Optional detail paragraphs or lists. Context a model should know, without headings.
4. H2 sections of links. Each item is a markdown link, optionally followed by a colon and a short note.
5. An optional "Optional" section. Links here can be skipped when a tool needs a shorter context.

Here is an example for a hypothetical scheduling app for clinics called Slotwise:

# Slotwise

> Slotwise is appointment scheduling software for small clinics. It handles online booking, SMS reminders and waitlists.

Slotwise is sold per clinic location. It integrates with Google Calendar and Stripe. It does not store medical records.

## Product

- [Features](https://slotwise.example/features.md): booking, reminders, waitlists
- [Pricing](https://slotwise.example/pricing.md): plans and what each includes
- [Integrations](https://slotwise.example/integrations.md): calendar and payment connections

## Docs

- [Getting started](https://slotwise.example/docs/start.md): set up a clinic in 15 minutes
- [Reminder settings](https://slotwise.example/docs/reminders.md): timing and templates

## Optional

- [Changelog](https://slotwise.example/changelog.md)
- [Company](https://slotwise.example/about.md)

Keep notes short and factual. The goal is a model reading only this file and still describing your product correctly.

llms.txt vs robots.txt vs sitemap.xml

These three files are often confused. They do different jobs.

robots.txt is a formal standard, RFC 9309. If you want to stop an AI crawler from fetching your pages, robots.txt is the tool, not llms.txt. Our robots.txt generator and sitemap generator cover the other two files.

How to write your llms.txt in six steps

1. Pick your 10 to 30 most useful pages

Choose pages that explain what you do and answer common questions: features, pricing, key docs, comparison pages and core guides. Skip legal boilerplate and thin pages.

2. Write the summary first

Write one or two sentences that a stranger could repeat accurately. Include the category, the customer and the main job. If your homepage does not say this plainly, fix the homepage too.

3. Group links into clear sections

Use H2 headings people would expect: Product, Docs, Guides, Company. Put each link on its own line with a short note after a colon.

4. Offer markdown versions where you can

The spec suggests serving clean markdown copies of pages, for example by adding .md to the URL. Docs platforms often support this. If yours does not, linking normal HTML pages is still fine.

5. Publish it at the root

Upload the file so it loads at https://yourdomain.com/llms.txt. Check that it returns a 200 status and is not blocked by robots.txt.

6. Keep it current

Update the file when you add a major page or change pricing. An out-of-date summary does more harm than no file.

Our free llms.txt generator builds a valid file from a form, and the llms.txt skill includes a checklist and a prompt for writing the summary.

Does llms.txt actually help?

The honest answer: it depends on who reads it.

• Support is uneven. As of this writing, major AI search providers have not publicly committed to reading llms.txt for ranking or citations. Google's guide to AI features and your website says the usual SEO practices apply and that no special files are required.
• Agents and coding tools benefit most. When a developer points an assistant at your docs, a clean index with markdown pages makes answers more accurate.
• The cost is low. For most sites it takes under an hour, and it forces you to write a clear summary of what you do, which helps everywhere.

So treat llms.txt as a small, cheap extra. Do not expect it to fix visibility on its own. Crawlable pages, answer-first content and third-party mentions matter more, as covered in our guide on how to get cited by AI assistants.

llms.txt vs llms-full.txt

Some sites publish a second file, llms-full.txt, alongside the main one. The two serve different readers.

llms-full.txt is a convention that grew around the proposal, not a required part of it. It suits documentation sites, where a coding assistant may want every API page at once. For a typical marketing site with a dozen key pages, the main llms.txt is usually enough.

If you do publish a full file, generate it automatically from your source pages during your build. Editing it by hand means it will drift out of date.

How to check that your file works

After publishing, run four quick checks:

1. Open the URL in a browser. It should show readable markdown, not a 404 or an HTML page.
2. Check the response headers. A text/plain or text/markdown content type is a safe choice.
3. Click every link. Each should return a 200 status, not a redirect chain.
4. Paste it into an assistant. Ask "Based only on this file, what does this company sell and to whom?" If the answer is wrong or vague, your summary needs work.

That last test is the most useful. It shows you exactly how a model reads your summary without the rest of your site.

Common mistakes

• Listing every URL. That is what sitemap.xml is for. Curate.
• Marketing language in the summary. "The leading platform" tells a model nothing. State facts.
• Broken or redirected links. Check every URL returns a 200.
• Contradicting your site. If llms.txt says one price and the pricing page says another, models get two answers.
• Using it to block crawlers. llms.txt has no blocking power. Use robots.txt.

How Thig helps

Thig's GEO agent checks how AI assistants describe your product and flags missing pieces such as an llms.txt file, unclear summaries or blocked crawlers. The Coding agent can then add the file and related fixes to your site as a pull request for you to review.