In partnership with
Hello, hello! 👋🏻
I shocked a fellow agency owner last week.
They were excited about the success they'd seen adding llms.txt files to client websites.
I laughed and said, "The data shows no positive impact from adding it."
I respect this agency owner, they do solid work. But chasing the fad of the month will, at best, waste time and energy. At worst? It'll hurt you.
Let's review what the research actually says on llms.txt.
📊 The Data: 300,000 Domains Don't Lie
SE Ranking analyzed nearly 300,000 domains to measure llms.txt effectiveness. The findings were brutal:
Only 10.13% of websites have adopted llms.txt (9 out of 10 haven't bothered)
Zero correlation between having an llms.txt file and AI citation frequency
Removing llms.txt from their predictive model actually improved accuracy
Let that sink in. The file makes predictions worse.
The Hustle: Claude Hacks For Marketers
Some people use Claude to write emails. Others use it to basically run their entire business while they play Wordle.
This isn't just ChatGPT's cooler cousin. It's the AI that's quietly revolutionizing how smart people work – writing entire business plans, planning marketing campaigns, and basically becoming the intern you never have to pay.
The Hustle's new guide shows you exactly how the AI-literate are leaving everyone else behind. Subscribe for instant access.
🤖 AI Doesn't Care About Your llms.txt
Giphy
OpenAI: Documentation focuses on robots.txt controls. Makes no mention of llms.txt affecting ranking or citations.
Google: Google has been particularly direct in downplaying llms.txt. According to multiple reports, Google representatives have "likened llms.txt to an early, unused idea…comparable to the old meta keywords tag in terms of current impact".
Anthropic (creators of Claude): Even though they publish their own llms.txt file, they "haven't confirmed their crawlers rely on it." Ironic, right?
Perplexity: Cloudflare research found they're "repeatedly modifying their user agent and changing IPs to hide crawling activity"—actively working around robots.txt, let alone voluntary standards.
🔥 The Smoking Gun: Server Logs
Here's where it gets damning.
A Reddit analysis of 1,000 domains with llms.txt files checked server logs to see which bots actually requested the file.
The result? No GPTBot. No ClaudeBot. No PerplexityBot.
Instead, 95% of requests came from Google's traditional desktop crawler.
Multiple independent analyses confirm: "Many webmasters who implemented llms.txt reported that known AI bots never requested the file at all."
⚠️ Why ScaledOn Doesn't Use llms.txt
Beyond the lack of evidence, there are real downsides:
Version Drift: Unlike auto-generated sitemaps, llms.txt requires manual oversight. Fall out of sync? "LLMs may ingest outdated or misleading data, leading to hallucinated outputs."
No Standardization: There's no authoritative schema defining what makes a page "LLM-worthy" or how frequently files should update. Implementation remains "ambiguous and brittle."
Resource Drain: You'd need to maintain llms.txt, potentially llms-full.txt (which can exceed 200,000 words), sync all changes with primary content, and create markdown versions of everything. "This burden hits smaller teams hardest."
Canonical URL Confusion: If LLMs link to raw .md or .txt files instead of your primary domain, "users land on unstyled text instead of your polished website." That breaks navigation, user experience, and sales funnels.
🎯 The Bottom Line: Busted
A year after its September 2024 launch, no rigorous study has demonstrated measurable benefits from llms.txt.
The SEO community loves shiny new tactics. But this one? Skip it.
Focus your energy on these 20 items that actually move the needle on getting into ChatGPT.
Cheers,
Alec
P.S. If a vendor is pushing llms.txt as the next big thing, ask them for the data. I'll wait. 😏
