The llms.txt file

In July 2025, NerdyData ran an index of every domain with a published llms.txt file. The count: 951.

For context: there are approximately 1.9 billion websites on the internet. That means roughly 0.00005% of websites have taken this particular step toward AI visibility. If you publish a llms.txt file this afternoon, you will be in one of the most exclusive clubs in digital marketing — and the barrier to entry is about an hour of work.

The file was proposed in September 2024 by Jeremy Howard of Answer.AI. The concept is straightforward: AI language models have small context windows and can't efficiently read an entire website. llms.txt is a Markdown-formatted file that sits at your domain root and gives AI a curated, structured summary of your most important content — a briefing note rather than a full archive.

Is it a magic SEO ranking factor? No. As of early 2025, Semrush tested it on Search Engine Land's domain and found that GPTBot, PerplexityBot, and ClaudeBot didn't visit the file directly during their test period. The major AI platforms hadn't implemented native support yet.

So why do it anyway? Three reasons.

Reason 1: the early mover window is open

When robots.txt was introduced in 1994, early adopters had years of advantage before it became standard practice. sitemap.xml followed a similar trajectory. The standard takes time to establish, and the brands that establish clean, well-structured files during that formation period have better signal quality when adoption scales.

The llms.txt standard is actively being developed. Anthropic published their own llms.txt on their website — which is a meaningful signal about where the standard is heading, even if the timeline is uncertain.

At 951 domains out of 1.9 billion, the adoption window is wide open.

Reason 2: it already works for the right use cases

Custom AI implementations, IDE plugins, AI coding assistants, and AI tools built on top of foundation models actively use llms.txt. If your audience includes developers, researchers, or enterprise teams building AI workflows, your llms.txt is being read right now by the tools they use.

GitBook, which builds documentation platforms for thousands of tech companies, now automatically generates llms.txt for all sites. They describe it as essential for "making sure your product shows up accurately when users ask AI for product support."

Reason 3: it's the best investment per hour in technical GEO

An hour of work to create a file that signals AI-readiness, establishes a canonical description of your brand for AI systems, and positions you among the top 0.00005% of indexed websites for this signal — that's a favorable return on time, even if the direct citation impact is still emerging.

It also shows up positively in AI readiness audits, including AI Rank Score's AI Readiness module.

The format (with a real example)

# Your Company Name
> One honest sentence about what your site does and who it serves.
> Optional second sentence with a key differentiator.

## Core Pages
- [Product Page](https://yourdomain.com): What your product does.
- [Pricing](https://yourdomain.com/pricing): Pricing structure.
- [About](https://yourdomain.com/about): Company background.

## Blog
- [Your Best Article](https://yourdomain.com/blog/slug):
  One honest sentence about what this article covers.
- [Second Article](https://yourdomain.com/blog/slug-2): Same format.

## Key Facts
- Founded: [year]
- Primary use case: [what you do]
- Pricing: [from X/month, or free tier available]
- Coverage: [what your product tracks/covers/does]

## Crawling Policy
This site may be crawled and cited by AI language models.
All content is original and regularly updated.

The AI Rank Score llms.txt in practice:

# AI Rank Score

> AI Rank Score is a free GEO audit platform that measures how visible
> any website is to AI search engines like ChatGPT, Perplexity, Gemini,
> Claude, Grok, and DeepSeek.

## What We Do

We analyze any URL and return a GEO Score (0–100) across four modules:
Content Structure, Citability, Domain Authority, and AI Citation Testing.
The Citation Testing module tests your site live against auto-generated
industry prompts across 6 AI engines.

## Key Facts

- Free tier: 3 analyses per month, no signup required
- Paid plans: from €19/month
- AI engines tested: Perplexity, ChatGPT, Gemini, Claude, Grok, DeepSeek
- Analysis time: approximately 30 seconds

## Core Pages
- [GEO Audit](https://airankscore.com): Free AI visibility audit tool.
- [Pricing](https://airankscore.com/pricing): Plans and features.

## Blog
- [AI Engine Rankings 2026](https://airankscore.com/blog/...):
  Market share and traffic data for all major AI search engines.
- [GEO Tools Comparison](https://airankscore.com/blog/...):
  Profound, Semrush, Otterly and others compared.

## Crawling
This site may be freely crawled and cited.
Content is original and updated monthly.

Three technical details that matter

Location: Must be at yourdomain.com/llms.txt, not /static/llms.txt or any subdirectory.

Format: Plain text serving with Content-Type: text/plain. Verify with curl -I yourdomain.com/llms.txt — look for HTTP 200 and the correct content type.

For Next.js: Drop the file in /public/. It will be served at your root automatically. No configuration needed.

Don't block AI crawlers and add llms.txt. These work together. If PerplexityBot is blocked in your robots.txt, the llms.txt is irrelevant. Check your robots.txt first.

What to write (and what not to)

The most common mistake: writing it like a press release. Phrases like "the leading AI-powered solution for transformational business outcomes" tell an AI model essentially nothing useful.

Write it like a briefing you'd give a smart assistant:

• Factual, specific, unevaluative
• Describe what the site does, not how good it is
• Include specific numbers where they exist (price, coverage, scale)
• Keep descriptions to one sentence per page — that's enough

The goal is helping an AI model accurately answer questions about your brand. If it reads your llms.txt and forms an accurate impression, you've succeeded.

Sources: NerdyData domain index, July 2025 · llmstxt.org specification · Jeremy Howard / Answer.AI (2024) · Semrush llms.txt experiment, October 2025 · GitBook llms.txt documentation · Overdrive AI crawling report