What is llms.txt?
A machine-readable markdown file at your domain root that gives AI engines a structured index of your most important pages. Think of it as robots.txt's modern companion — specifically for LLMs.
The idea behind the standard
AI language models need to understand what a site contains and what matters. Traditionally they've done this by crawling pages and inferring structure. The llms.txt standard — proposed by Jeremy Howard and adopted by companies including Anthropic, Stripe, Vercel, and Cloudflare — gives models a more direct path.
A file at yoursite.com/llms.txt provides your brand name, a one-line description, and a grouped list of your most important pages. AI systems use it to build a cleaner picture of what you do and where to find relevant content — without having to crawl and parse every page.
The file format
llms.txt is plain markdown with a simple structure:
# Brand Name > One-line description: what you do and who you serve. ## Main sections (use headings to group related pages) ## Products - [Product Name](https://yoursite.com/product): Brief description - [Pricing](https://yoursite.com/pricing): Plans and pricing ## Resources - [Documentation](https://yoursite.com/docs): Getting started and API reference - [Blog](https://yoursite.com/blog): Articles and updates ## Company - [About](https://yoursite.com/about): Who we are - [Contact](https://yoursite.com/contact): How to reach us
Why AI engines benefit from it
AI engines have to infer site structure from URLs and page content. llms.txt makes the grouping explicit — this section is Products, this section is Docs, this section is Company. That clarity improves how models describe and cite your site.
A crawl may return hundreds of URLs. llms.txt says: these 50 are the important ones, in this order. Models are more likely to surface your key pages when they're explicitly flagged as important.
The header of llms.txt — brand name and one-line description — becomes a canonical signal for how AI systems describe your company. Consistent, accurate brand description here reduces the chance of AI misrepresenting what you do.
Instead of crawling and parsing dozens of pages to understand a site, a model can read the llms.txt in seconds and get a working map of what matters. For models with retrieval limits, this improves how much relevant information gets into the context.
Who's already publishing it?
Adoption is concentrated in the technical/developer ecosystem so far — companies building tools and infrastructure where AI developer usage is highest. Early adopters include:
Adoption is early. The opportunity for first-mover advantage is high — very few ecommerce or B2B SaaS sites have published one yet.
How to create and publish yours
- 1. Generate the file
Use our free llms.txt generator — it reads your public sitemap, selects the top 50 pages, groups them by section, and builds the file. Takes 30 seconds.
Generate your llms.txt free → - 2. Review and edit
Check the generated file for accuracy. Edit section names if needed, remove pages that shouldn't be indexed by AI (login pages, user-specific pages), and refine your description line to be specific and clear.
- 3. Upload to your domain root
The file must be accessible at yoursite.com/llms.txt — not in a subdirectory. Upload via FTP, your CMS media uploader, or add a server route that returns the file content.
- 4. Verify it's live
Navigate to yoursite.com/llms.txt in your browser. You should see the plain-text file content. If you get a 404, check your upload path or server routing.
- 5. Add to sitemap.xml (optional)
While not required, adding a reference to /llms.txt in your sitemap.xml helps AI crawlers that follow sitemaps to discover the file quickly.
Generate yours in 30 seconds
Free tool — reads your sitemap, no signup required.