Web Development

Does llms.txt Work? Google, Lighthouse and 53 Sites Checked

Google says Google Search ignores llms.txt, but Chrome's Lighthouse now audits it. What the audit checks, read from its source; which of 53 sites have a file; whether any AI says it reads one; and how to generate it in Next.js.

Kausar RazaBy 8 min read11 views
Share:
Does llms.txt work? Google ignores it, Lighthouse checks it, 25 of 53 sites have one
  • SEO
  • AI Search
  • ChatGPT
Available for projects

Need help with this?

I build and fix business websites - fast, mobile-first and built to bring enquiries.

  • Fixed quote, no surprises
  • Speed & SEO built in
  • Reply within a few hours
Get a free quoteChat on WhatsApp

llms.txt is a Markdown file at the root of a site that tells AI tools what the site is and which pages matter. The pitch is simple: hand language models a clean map instead of making them parse your menus and cookie banners. Plenty of plugins and agencies now sell it as AI SEO. The search box tells the other half of the story: "does llms.txt work", "is llms.txt actually used", "is llms.txt worth it".

I went to the sources instead of the opinions. I read Google's current guidance, then the source code of the Lighthouse audit that now checks the file. I requested /llms.txt from 53 well-known sites on 19 November 2026 to see who actually publishes one. This site has had one since September, so I can show how it's built too.

The short answer

  • For Google Search, no. Google's own guide says Google Search "ignores" llms.txt files. Adding one neither helps nor hurts rankings, AI Overviews or AI Mode.
  • Chrome's Lighthouse does check it, in the new Agentic Browsing category. It checks the format, not whether anything reads the file. A missing file is "not applicable". A broken one fails.
  • Adoption is real but narrow. 25 of the 53 sites I checked have a valid file. Most are developer platforms. OpenAI, Anthropic, Perplexity and Google don't have one on their main domains.
  • No AI company says its crawler uses it. I found no statement from OpenAI, Anthropic or Perplexity that their crawlers read llms.txt.
  • Worth it? For documentation and developer products, yes: it's cheap, and coding tools and people pasting your docs into an assistant can use it. For a typical business site, add it only if it's generated automatically, and never ship a broken one.

What llms.txt is

Jeremy Howard proposed the format on 3 September 2024 at llmstxt.org. It's plain Markdown with a fixed order:

  1. An H1 with the site or project name. This is the only required part.
  2. A blockquote summary in one or two sentences.
  3. Optional paragraphs or lists with more detail (no other headings).
  4. H2 sections that list links as [name](url): notes. A section called Optional marks links a tool can skip when it is short on context.
# Example Studio

> Web design studio in Leeds building fast WordPress and Next.js sites for
> independent shops. Projects start at four weeks.

We work with UK retailers only.

## Services

- [Services](https://example.com/services): what we build and how a project runs
- [Case studies](https://example.com/work): six shop rebuilds with before/after numbers
- [Contact](https://example.com/contact): email and booking form

## Guides

- [Shopify vs WooCommerce](https://example.com/blog/shopify-vs-woocommerce): cost and lock-in compared

## Optional

- [About](https://example.com/about): team and history

The proposal also suggests Markdown copies of your pages at the same URL plus .md. It allows the file on a subpath, such as /docs/llms.txt, which applies to everything under that path. Some sites also publish an llms-full.txt, with the full text of the docs in one file. That one isn't in the spec, and 8 of my 53 sites had it.

llms.txt vs robots.txt: robots.txt tells crawlers what they may fetch, and the major crawlers say they obey it. llms.txt only describes your content. It grants and blocks nothing. To control AI crawlers, you still need robots.txt or your CDN. I covered that in which AI bots to block on Cloudflare.

What Google says about llms.txt

Google published its guide to generative AI features in Search in May 2026 and clarified it in June. The current text is direct:

You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.

It then says keeping an llms.txt for other services is "completely fine", and that it "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them". The same guide lists "creating unnecessary AI text files (like llms.txt)" among the "AEO/GEO hacks" you can ignore for Google Search.

So anyone selling llms.txt as a way into Google's AI Overviews is selling something Google has ruled out in writing. What helps there is ordinary SEO: crawlable pages, clear answers and content nobody else has. See how to get cited by ChatGPT and AI Overviews.

Lighthouse checks it anyway: here's exactly how

Table of Lighthouse llms.txt audit results: 4xx is not applicable, 5xx or 429 fails, an HTML page fails, no H1 line fails, bare URLs fail, under 50 characters fails, an H1 plus a Markdown link passes
The rules, read from the audit's source in lighthouse 13.6.0.

Lighthouse 13.3.0 (May 2026) added an Agentic Browsing category, and one of its audits is called llms.txt follows recommendations. The SEO blogs disagree on what it checks, so I installed lighthouse@13.6.0 and read the audit and its gatherer:

  • It requests /llms.txt on the root of the page's origin. A valid file on a subpath isn't found.
  • Any 4xx (404, 403, 410) marks the audit not applicable. No file costs you nothing.
  • 5xx or 429 fails with "Failed with HTTP status".
  • For a 200, there are three checks: a line starting with # and a space (an H1), at least one Markdown link [text](url), and at least 50 characters. Miss any of them and it fails.

That's the whole check. It doesn't look at the content, the content type or whether the links work. A file with one heading and one link passes, the same as Stripe's 69 KB file.

The category shows a fraction (audits passed out of audits that apply), not a 0-100 score. I ran it in headless Chrome on three sites: this one passed every applicable audit; Moz, which has no file, scored 0.49 with llms.txt marked N/A; Reddit scored 0.33 with llms.txt failing.

I checked 53 sites: who actually has one

Bar chart of 53 sites checked for llms.txt on 19 November 2026: 25 pass Lighthouse's rules, 2 fail (reddit.com and notion.so), 26 have no file, including openai.com, anthropic.com, perplexity.ai and google.com
One request per domain on 19 November 2026, scored with Lighthouse's rules. Files on subpaths are not counted.

I requested /llms.txt from 53 domains (AI companies, developer platforms, website builders, SEO tools, publishers and big tech) and applied Lighthouse's rules to each response. Results:

  • 25 pass. Almost all are developer platforms and site builders: Cloudflare, Vercel, Next.js, React, Stripe, GitHub, Supabase, Netlify, Svelte, Shopify, WordPress.org, Wix, Squarespace, HubSpot and Zapier. Anthropic's docs domain redirects to a file of about 100 KB with over 900 links.
  • 26 have no file at the root. That includes openai.com, anthropic.com, perplexity.ai, google.com and developer.chrome.com, the site that documents the Lighthouse audit. Publishers (New York Times, BBC, Guardian, Wikipedia) and Moz and Ahrefs don't have one either.
  • 2 fail. Reddit answers /llms.txt with its normal HTML app and a 200 status. Notion's file starts with a blockquote and has no H1 line.

Two details stood out. The AI companies publish llms.txt for their documentation (OpenAI's and Perplexity's developer docs both have one) but not for their main sites. And FastHTML, a project from Answer.AI, the company of the format's author, keeps its file under /docs/. The spec allows that, but Lighthouse never requests it.

Does any AI actually read llms.txt?

As far as public documentation goes, no crawler says it does. OpenAI's, Anthropic's and Perplexity's crawler pages describe robots.txt controls and IP ranges, and none says the crawlers read llms.txt. Google says its search systems ignore the file.

Where it does get used is on demand. A developer can point a coding assistant at a docs site's llms.txt or llms-full.txt to load the docs in one go. That's why documentation sites dominate the pass list, and why the AI companies publish one for their docs. It's a format for tools that are asked to read it, not a signal crawlers go looking for.

I haven't measured whether any AI crawler requests this site's file, because I don't have server logs on this machine. If you have logs, grep "GET /llms.txt" and look at the user agents. That's the only real answer for your own site.

Is llms.txt worth it?

  • Documentation, APIs, developer tools: yes. People feed docs to assistants, and an index plus Markdown pages makes that cleaner. Generate it from your docs source so it stays current.
  • Business sites and blogs: optional. It won't move Google, and no crawler has said it will read it. If your framework can generate it automatically, the cost is close to zero, and a short, accurate description of what you do doesn't hurt. Don't pay anyone to write one for SEO.
  • Any site with a catch-all route: check what /llms.txt returns today. A single-page app or a CMS that answers every unknown URL with a 200 HTML page fails the Lighthouse audit, like Reddit's. A plain 404 doesn't.

How this site generates its llms.txt (Next.js)

Here the file is a Next.js route handler that builds the Markdown from the same API the pages use, so new posts and tools appear without anyone editing a file. A trimmed version:

// app/llms.txt/route.ts
import { getPosts } from "@/lib/api";

export const revalidate = 86400; // rebuild once a day

export async function GET() {
  const posts = await getPosts({ pageSize: 30 }).catch(() => ({ items: [] }));
  const link = (name: string, url: string, note?: string) =>
    `- [${name}](${url})${note ? `: ${note}` : ""}`;

  const body = [
    "# Example Studio",
    "",
    "> Web design studio building fast WordPress and Next.js sites.",
    "",
    "## Recent articles",
    "",
    ...posts.items.map((p) =>
      link(p.title, `https://example.com/blog/${p.slug}`, p.excerpt?.slice(0, 120))),
  ].join("\n");

  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

Three things matter. The content type is text/plain, so browsers and tools don't treat it as a page. The links are Markdown links, not bare URLs. And it is regenerated daily rather than written by hand, because a hand-written file goes stale. Ours is about 22 KB and lists the services, every free tool and the 30 most recent articles.

Whatever you use, check what's live:

# Status code and content type
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/llms.txt

# First lines: the first one should start with "# "
curl -s https://example.com/llms.txt | head -5

# Count Markdown links; 0 fails the Lighthouse audit
curl -s https://example.com/llms.txt | grep -c "\](http"

A 404 is fine. A 200 with text/html, no # line or zero links is the failing case. Then run Lighthouse with the Agentic Browsing category to confirm. If you're tracking whether AI tools send you visitors at all, set up an AI assistant channel in GA4 first. It answers "is this worth my time" better than any file.

Frequently asked questions

Does llms.txt work?
Not for Google Search: Google's guide says Google Search ignores llms.txt and that it neither helps nor harms rankings. No major AI crawler documents that it reads the file either. It works as an index for tools that are pointed at it, such as coding assistants loading a documentation site, which is why developer platforms are most of the sites that publish one.
Does Google use llms.txt?
Google Search does not. Its guide to generative AI features says you don't need machine-readable or AI text files to appear in Google Search, including AI Overviews and AI Mode, because Google Search doesn't use them. Chrome's Lighthouse does check the file's format in its Agentic Browsing category, which is a separate tool.
What does the Lighthouse llms.txt audit check?
Lighthouse requests /llms.txt at the root of the page's origin. A 4xx response makes the audit not applicable. A 5xx or 429 fails. A 200 passes only if the file has a line starting with '# ' (an H1), at least one Markdown link in [text](url) form, and at least 50 characters. The content itself is not judged.
Is a missing llms.txt bad for Lighthouse?
No. A 404 or any other 4xx marks the llms.txt audit as not applicable, so it doesn't count against the Agentic Browsing category. What fails is a broken file: a 5xx, or a 200 that returns an HTML page, has no H1 line or contains no Markdown links.
What is the difference between llms.txt and robots.txt?
robots.txt controls access: it tells crawlers which URLs they may fetch, and major crawlers say they obey it. llms.txt is only a description: a Markdown summary and list of important links. It cannot allow or block anything. To control AI crawlers, use robots.txt or your CDN's bot settings.
What is llms-full.txt?
llms-full.txt is a common companion file that puts the full text of a site's documentation into one Markdown file, so a tool can load everything in a single request. It is not part of the llmstxt.org proposal. In a check of 53 sites on 19 November 2026, 8 published one, mostly documentation platforms.
Should I add llms.txt to my WordPress or business site?
It's optional. It won't change Google rankings, and no crawler has said it reads the file. If a plugin or your framework generates it automatically and keeps it current, there's little downside. Don't pay for it as an SEO service, and make sure /llms.txt never returns an HTML page with a 200 status.

Join the conversation

No comments yet — be the first to share what you think.

Leave a comment

Never published.

Optional.

Keep reading

Related articles

View all