llms.txt is a Markdown file at the root of a site that tells AI tools what the site is and which pages matter. The pitch is simple: hand language models a clean map instead of making them parse your menus and cookie banners. Plenty of plugins and agencies now sell it as AI SEO. The search box tells the other half of the story: "does llms.txt work", "is llms.txt actually used", "is llms.txt worth it".
I went to the sources instead of the opinions. I read Google's current guidance, then the source code of the
Lighthouse audit that now checks the file. I requested /llms.txt from 53 well-known sites on 19 November
2026 to see who actually publishes one. This site has had one since September, so I can show how it's built too.
The short answer
- For Google Search, no. Google's own guide says Google Search "ignores" llms.txt files. Adding one neither helps nor hurts rankings, AI Overviews or AI Mode.
- Chrome's Lighthouse does check it, in the new Agentic Browsing category. It checks the format, not whether anything reads the file. A missing file is "not applicable". A broken one fails.
- Adoption is real but narrow. 25 of the 53 sites I checked have a valid file. Most are developer platforms. OpenAI, Anthropic, Perplexity and Google don't have one on their main domains.
- No AI company says its crawler uses it. I found no statement from OpenAI, Anthropic or Perplexity that their crawlers read llms.txt.
- Worth it? For documentation and developer products, yes: it's cheap, and coding tools and people pasting your docs into an assistant can use it. For a typical business site, add it only if it's generated automatically, and never ship a broken one.
What llms.txt is
Jeremy Howard proposed the format on 3 September 2024 at llmstxt.org. It's plain Markdown with a fixed order:
- An H1 with the site or project name. This is the only required part.
- A blockquote summary in one or two sentences.
- Optional paragraphs or lists with more detail (no other headings).
- H2 sections that list links as
[name](url): notes. A section called Optional marks links a tool can skip when it is short on context.
# Example Studio
> Web design studio in Leeds building fast WordPress and Next.js sites for
> independent shops. Projects start at four weeks.
We work with UK retailers only.
## Services
- [Services](https://example.com/services): what we build and how a project runs
- [Case studies](https://example.com/work): six shop rebuilds with before/after numbers
- [Contact](https://example.com/contact): email and booking form
## Guides
- [Shopify vs WooCommerce](https://example.com/blog/shopify-vs-woocommerce): cost and lock-in compared
## Optional
- [About](https://example.com/about): team and history
The proposal also suggests Markdown copies of your pages at the same URL plus .md. It allows the file
on a subpath, such as /docs/llms.txt, which applies to everything under that path. Some sites also publish
an llms-full.txt, with the full text of the docs in one file. That one isn't in the spec, and 8 of my
53 sites had it.
llms.txt vs robots.txt: robots.txt tells crawlers what they may fetch, and the major crawlers say they obey it. llms.txt only describes your content. It grants and blocks nothing. To control AI crawlers, you still need robots.txt or your CDN. I covered that in which AI bots to block on Cloudflare.
What Google says about llms.txt
Google published its guide to generative AI features in Search in May 2026 and clarified it in June. The current text is direct:
You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.
It then says keeping an llms.txt for other services is "completely fine", and that it "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them". The same guide lists "creating unnecessary AI text files (like llms.txt)" among the "AEO/GEO hacks" you can ignore for Google Search.
So anyone selling llms.txt as a way into Google's AI Overviews is selling something Google has ruled out in writing. What helps there is ordinary SEO: crawlable pages, clear answers and content nobody else has. See how to get cited by ChatGPT and AI Overviews.
Lighthouse checks it anyway: here's exactly how

Lighthouse 13.3.0 (May 2026) added an Agentic Browsing category, and one of its audits is called
llms.txt follows recommendations. The SEO blogs disagree on what it checks, so I installed
lighthouse@13.6.0 and read the audit and its gatherer:
- It requests
/llms.txton the root of the page's origin. A valid file on a subpath isn't found. - Any 4xx (404, 403, 410) marks the audit not applicable. No file costs you nothing.
- 5xx or 429 fails with "Failed with HTTP status".
- For a 200, there are three checks: a line starting with
#and a space (an H1), at least one Markdown link[text](url), and at least 50 characters. Miss any of them and it fails.
That's the whole check. It doesn't look at the content, the content type or whether the links work. A file with one heading and one link passes, the same as Stripe's 69 KB file.
The category shows a fraction (audits passed out of audits that apply), not a 0-100 score. I ran it in headless Chrome on three sites: this one passed every applicable audit; Moz, which has no file, scored 0.49 with llms.txt marked N/A; Reddit scored 0.33 with llms.txt failing.
I checked 53 sites: who actually has one

I requested /llms.txt from 53 domains (AI companies, developer platforms, website builders, SEO
tools, publishers and big tech) and applied Lighthouse's rules to each response. Results:
- 25 pass. Almost all are developer platforms and site builders: Cloudflare, Vercel, Next.js, React, Stripe, GitHub, Supabase, Netlify, Svelte, Shopify, WordPress.org, Wix, Squarespace, HubSpot and Zapier. Anthropic's docs domain redirects to a file of about 100 KB with over 900 links.
- 26 have no file at the root. That includes openai.com, anthropic.com, perplexity.ai, google.com and developer.chrome.com, the site that documents the Lighthouse audit. Publishers (New York Times, BBC, Guardian, Wikipedia) and Moz and Ahrefs don't have one either.
- 2 fail. Reddit answers
/llms.txtwith its normal HTML app and a 200 status. Notion's file starts with a blockquote and has no H1 line.
Two details stood out. The AI companies publish llms.txt for their documentation (OpenAI's and Perplexity's
developer docs both have one) but not for their main sites. And FastHTML, a project from Answer.AI, the company of the format's
author, keeps its file under /docs/. The spec allows that, but Lighthouse never requests it.
Does any AI actually read llms.txt?
As far as public documentation goes, no crawler says it does. OpenAI's, Anthropic's and Perplexity's crawler pages describe robots.txt controls and IP ranges, and none says the crawlers read llms.txt. Google says its search systems ignore the file.
Where it does get used is on demand. A developer can point a coding assistant at a docs site's llms.txt or llms-full.txt to load the docs in one go. That's why documentation sites dominate the pass list, and why the AI companies publish one for their docs. It's a format for tools that are asked to read it, not a signal crawlers go looking for.
I haven't measured whether any AI crawler requests this site's file, because I don't have server logs on this
machine. If you have logs, grep "GET /llms.txt" and look at the user agents. That's the only real answer
for your own site.
Is llms.txt worth it?
- Documentation, APIs, developer tools: yes. People feed docs to assistants, and an index plus Markdown pages makes that cleaner. Generate it from your docs source so it stays current.
- Business sites and blogs: optional. It won't move Google, and no crawler has said it will read it. If your framework can generate it automatically, the cost is close to zero, and a short, accurate description of what you do doesn't hurt. Don't pay anyone to write one for SEO.
- Any site with a catch-all route: check what
/llms.txtreturns today. A single-page app or a CMS that answers every unknown URL with a 200 HTML page fails the Lighthouse audit, like Reddit's. A plain 404 doesn't.
How this site generates its llms.txt (Next.js)
Here the file is a Next.js route handler that builds the Markdown from the same API the pages use, so new posts and tools appear without anyone editing a file. A trimmed version:
// app/llms.txt/route.ts
import { getPosts } from "@/lib/api";
export const revalidate = 86400; // rebuild once a day
export async function GET() {
const posts = await getPosts({ pageSize: 30 }).catch(() => ({ items: [] }));
const link = (name: string, url: string, note?: string) =>
`- [${name}](${url})${note ? `: ${note}` : ""}`;
const body = [
"# Example Studio",
"",
"> Web design studio building fast WordPress and Next.js sites.",
"",
"## Recent articles",
"",
...posts.items.map((p) =>
link(p.title, `https://example.com/blog/${p.slug}`, p.excerpt?.slice(0, 120))),
].join("\n");
return new Response(body, {
headers: { "Content-Type": "text/plain; charset=utf-8" },
});
}
Three things matter. The content type is text/plain, so browsers and tools don't treat it as a page.
The links are Markdown links, not bare URLs. And it is regenerated daily rather than written by hand, because a
hand-written file goes stale. Ours is about 22 KB and lists the services, every free tool and the 30 most recent
articles.
Whatever you use, check what's live:
# Status code and content type
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/llms.txt
# First lines: the first one should start with "# "
curl -s https://example.com/llms.txt | head -5
# Count Markdown links; 0 fails the Lighthouse audit
curl -s https://example.com/llms.txt | grep -c "\](http"
A 404 is fine. A 200 with text/html, no # line or zero links is the failing case.
Then run Lighthouse with the Agentic Browsing category to confirm. If you're tracking whether AI tools send you
visitors at all, set up an AI assistant channel in
GA4 first. It answers "is this worth my time" better than any file.





Join the conversation
No comments yet — be the first to share what you think.