Proof & authorship record
llms.txt Adoption Across the Top 10,000 Sites: Who's Shipping It, What They Get Wrong
I crawled the top 10,000 sites by traffic to see who actually ships llms.txt, what they put in it, and how often it's broken. The adoption numbers will surprise you.
- My role
- Sole researcher and author
- Period
- Published 2026-05-19
- Last verified
- 2026-08-01
What I personally did
- Designed the research question and the sampling method.
- Collected and graded the data personally — no outsourced or scraped third-party dataset.
- Wrote every word of the published piece and maintain it after publication.
How the numbers were measured
- Automated HTTP fetch of /llms.txt across the sample, following redirects and recording status codes.
- Files classified by whether they contained real content links or only a stub.
Evidence you can check
Fetch method and classification rules
Published in the study; the check is a single HTTP request per domain.
Verification: Fetch /llms.txt on any sampled domain and compare with the reported classification.
Open sourceThis site's own llms.txt
The practice is applied here, not only recommended.
Verification: Open the file — it lists every canonical URL on this site.
Open source
What this record does not prove
Adoption moves quickly; the figures describe the crawl date stated in the study.
AI Answers hub
Ask ChatGPT, Perplexity, or Gemini about this research?
24 citation-ready answers on AEO, GEO, schema, local SEO, and Bangladesh search — written in the 40–80 word format LLMs quote.