Should Digital Marketers Pay Attention to Llms.txt?
Although some of us thought the llms.txt conversation ended months ago, continued use and research on llms.txt have provided additional data points on whether llms.txt are helpful and necessary for AI agent optimization. So, let’s take a quick look back at the history of llms.txt and what recent research shows.
What is Llms.txt?
Llms.txt is a proposed Markdown file placed at the root of a website that was introduced back in September 2024. It is intended to give a summary of a website that is specifically curated to give AI tools key information in a way that the AI tools easily understand.
Sitemap.xml, Robots.txt, and Llms.txt?
Although llms.txt is not the same as robots.txt or sitemap.xml either in format or intent, there is overlap in terms of quick website summaries and how some crawlers use llms.txt.
Sitemap.xml provides a complete list of URLs on a site that web owners want to be crawled. Although it can provide some additional information aside from URLs (e.g., lastmod, changefreq, and priority), they’re secondary at best and completely ignored at worst.
Robots.txt provides very specific access rules. It is intended to act as an actual gatekeeper that blocks or allows crawlers to crawl specific areas of a site. Although it can be ignored, it is generally respected by standard crawlers.
Llms.txt is substantially more of a curation file than a discovery file (like sitemap) or access control file (like robots.txt). It does provide URLS, but is not intended to be a full comprehensive list. And along with the URL, it provides summaries and additional Markdown structure (e.g., groupings, headers, quotes).
What is Llms.txt Trying to Solve?
Some defenders of llms.txt point out that websites are frequently very large and have a lot of information (including unnecessary or duplicative content), so llms.txt can help improve efficiency in LLM ingestion and training. It’s like a cheat sheet or Cliff Notes. Critics of llms.txt say that llms.txt don’t actually solve a problem at all.
Current State of Llms.txt
The adoption for llms.txt has been weak. As of late 2025, some analyses showed that less than 1,000 websites were using llms.txt files.¹ Some of these llms.txt users are big name tech leaders such as Cloudflare and Anthropic. However, the general consensus within digital marketing, SEO, and IT conversations (even with pro AI thought leaders) indicates that llms.txt overall is unhelpful and uneeded for general purposes.
Some of the main sites that have an llms.txt (e.g., Anthropic) seem to use llms.txt for one specific, niche use case. It gives developers and coding assistants a structured entry point into API documentation. It’s developer documentation not anything related to GEO, SEO, or overall AI visibility.
So… why hasn’t llms.txt seen wider adoption and popularity?
Llms.txt are Self-Reported and Inherently Untrustworthy
John Mueller, Google’s Search Relations Team Lead, explained in an episode of Search Off the Record that LLMs should not look to a self-reported file to pick the best sources and best summary. An LLM should evaluate the full web content and make that summary and analysis itself. He’s also questioned the functionality of llms.txt in Reddit² conversations and overall called it “a stupid idea” on Bluesky.³
They’re Redundant
For broad GEO and AI visibility, llms.txt is generally just… redundant. We have URL discoverability and access info through sitemap.xml and robots.txt, we have better methods for summaries and machine-readable information through semantic structuring and schemas. Bluntly, llms.txt doesn’t really add much beyond existing measures we already know work for GEO and AI visibility.
If that’s not enough, let’s look at the biggest indictment. If we’re hoping to use llms.txt for AI/LLM visibility, then has it worked? Does having an llms.txt increase a site’s ingestion and visibility?
New Studies Confirm: LLMs Don’t Use Them
Aura, the agentic web research lab, ran a test recently and concluded that agents rarely used llms.txt.⁴ This confirms older research that showed the same conclusions. In this Otterly.ai piece, only 0.1% of AI bot traffic accessed an llms.txt (84 of 62,000+ visits).⁵ And a similar piece from June 2026 from Ahrefs concluded that 97% of llms.txt files never got read.⁶
Another recent analysis conducted by Common Crawl took a look at llms.txt by examining llms.txt files they could find on sample crawlable hosts.⁷ Essentially, this research asked “Since some llms.txt exist, what do they look like?” The findings show:
68% of the llms.txt originated from a plugin or template
22% of the llms.txt had no links at all
70% of the seeded links returned a 404
Only 20% returned healthy 200 site code
Of those 20% 200 site code URLs, only 46% had actual body text
Bluntly, the llms.txt files themselves were not healthy or accurate using even basic measures.
For now, llms.txts are still rare, poorly created, almost never accessed, and don’t seem to fill many broad use cases. As things continue to move fast in terms of AI tools and visibility, this could change. But, we’d hazard a guess that llms.txt have been around for a couple of years now and have made little to no traction.
We have better best practices to rely on for AI optimization even if those practices are not quite the cheat sheet many would like them to be.