What robots.txt does - and does not do
robots.txt tells crawlers which URLs they may fetch. It is a request, not a lock: well-behaved crawlers follow it, and anything private still needs a password. It also does not remove pages from search - a blocked URL can still be indexed from links to it. To keep a page out of results, let it be crawled and use a noindex meta tag.
How the rules are matched
The format is standardised as RFC 9309. A crawler uses the group whose User-agent line matches it most specifically, falling back to User-agent: *. Within that group the longest matching rule wins, and if an Allow and a Disallow are equally long, Allow wins. * matches any characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDFs. The tester here applies exactly those rules.
Common mistakes it catches
Rules before any User-agent line (ignored), a leftover Disallow: / from a staging site (blocks everything), Noindex lines (Google stopped supporting them in 2019), Crawl-delay (Google ignores it), a missing Sitemap line, a file over 500 KiB (Google reads only that much), and a robots.txt that answers with a server error - which makes Google treat the whole site as off-limits until it recovers.
AI crawlers
You can test GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended, which controls whether Google may use your content for its AI models. Blocking Google-Extended does not affect Google Search. Then check your sitemap with the XML sitemap validator.
