--- title: llms.txt: The Complete Guide: What It Is, How to Create One, and Why It Matters for AI Search slug: llms-txt-guide-complete-tutorial language: en contentType: tutorial keyword: llms.txt guide topic: llms.txt: The Complete Guide to the robots.txt for AI — What It Is, How to Create One, and Why It Matters metaTitle: llms.txt Guide: Complete Tutorial for AI Crawlers (2026) metaDescription: Complete llms.txt guide: what the specification says, who created it (Jeremy Howard, Answer.AI), current bot adoption status, step by step setup, and best practices. Free generator included. focusKeyphrase: llms.txt guide generatedAt: 2026-02-13T22:47:35.580Z wordCount: 1797 seoScore: 88 readabilityScore: 50.8 keywordDensity: 0.08 featuredImage: https://reaudit.io/article-images/og-llms-txt-the-complete-guide-wh-1200x630-2026-02-16.jpg tags: [llms.txt, AI SEO, Generative Engine Optimization, robots.txt for AI, AI search visibility, Reaudit] --- # llms.txt: The Complete Guide: What It Is, How to Create One, and Why It Matters for AI Search llms.txt is a plain‑text file that tells AI crawlers which pages matter most. Learn how to create, optimise, and monitor it with Reaudit’s free generator and best‑practice checklist. llms.txt is a Markdown file placed at the root of your website that tells AI crawlers which pages matter most and how to describe them. Think of it as a curated table of contents written specifically for large language models. This guide covers what the specification actually says, who proposed it, which bots currently read it, and how to create one for your site, including an honest assessment of where adoption stands right now. ## Quick Answer Create a Markdown file called `llms.txt` listing 10 to 20 important URLs with brief descriptions. Host it at `https://yourdomain.com/llms.txt`. Make sure AI bots are not blocked in your `robots.txt`. Keep it in sync with your sitemap and schema markup. The whole thing takes about 20 minutes to set up manually, or you can use [Reaudit's free llms.txt generator](https://reaudit.io/free-tools/llms-txt-generator) to automate it. ## Where llms.txt Came From The llms.txt specification was proposed by Jeremy Howard, co-founder of Answer.AI (and previously co-founder of fast.ai), on September 3, 2024. The idea was straightforward: websites already have `robots.txt` to tell crawlers what they can and cannot access, but there was nothing to help LLMs understand what a site is actually about and which pages are most important. Howard's proposal defined two files: `/llms.txt` as a navigation aid listing key pages with descriptions, and `/llms-full.txt` as an optional companion containing the full content of the site in a single Markdown document. The specification lives at [llmstxt.org](https://llmstxt.org). The concept gained traction quickly. Notable early adopters include Anthropic, Stripe, Cursor, Mintlify, Zapier, and FastHTML. As of mid 2025, around 951 domains had published llms.txt files, a tiny fraction of the web, but growing. ## The Format: How llms.txt Is Structured The specification is deliberately simple. The file uses standard Markdown with a few conventions: An H1 heading (`#`) with the project or brand name. An optional blockquote (`>`) with a one-line summary. H2 headings (`##`) to group pages into logical sections. Bullet-point links in the format `- [Title](URL): Brief description`. Here is a minimal example: ``` # Acme SaaS > Cloud-based invoicing platform for small and medium businesses. ## Key Pages - [Home](https://acme.io): Main landing page with product overview. - [Pricing](https://acme.io/pricing): Subscription tiers and feature comparison. ## Documentation - [API Guide](https://acme.io/docs/api): REST API reference for developers. - [Getting Started](https://acme.io/docs/start): Quickstart tutorial for new users. ## Support - [Help Center](https://acme.io/help): FAQs and troubleshooting guides. ``` The `/llms-full.txt` companion file follows the same Markdown format but contains the full text content of the site's key pages, concatenated into a single document. This lets LLMs ingest your entire knowledge base in one request. ## How llms.txt Differs From robots.txt These two files solve different problems. `robots.txt` controls access — it tells crawlers which URLs they are allowed or disallowed from visiting. It is a set of directives, not content. `llms.txt` provides context. It does not control access; it offers a human-readable summary that AI models can use to understand what your site covers and which pages to prioritize. The two files complement each other: `robots.txt` must allow the AI bots you want to reach your `llms.txt`, otherwise the file will never be fetched. ## Which AI Bots Actually Read llms.txt? An Honest Assessment This is where many guides oversell the current state of things, so here is the straight picture. No major LLM provider has officially committed to using llms.txt as a ranking or citation signal. Google's John Mueller stated in 2025 that Google has not adopted it. Analysis by Averi of 1,000 domains in August 2025 found zero visits from major LLM crawlers (GPTBot, ClaudeBot, PerplexityBot) to llms.txt files specifically. That said, the picture is more nuanced than "nobody reads it." Profound's data showed some evidence of Microsoft and OpenAI bots crawling llms.txt files. Several AI platforms have adopted it for their own sites (Anthropic publishes one at anthropic.com/llms.txt). And the specification is gaining mindshare in the developer community, it is a low-effort implementation that positions your site for whatever comes next. The honest take: llms.txt is an emerging convention, not an established standard. Implementing it costs almost nothing and cannot hurt your AI visibility. Whether it will become a meaningful signal depends on whether major AI providers decide to officially adopt it. The early signs are promising but not definitive. ## Step-by-Step: Creating an llms.txt File ### Option 1: Manual Creation Pick 10 to 20 of your most important pages. These should be your highest value evergreen content: homepage, product pages, pricing, key guides, help center, and case studies. Write a one line summary for each page (keep descriptions under 20 words). Group them under logical H2 headings. Save as `llms.txt` in UTF-8 encoding and upload to your web root. Test it by running `curl -I https://yourdomain.com/llms.txt` you should get a 200 response. If you also want to create `llms-full.txt`, concatenate the full Markdown content of each listed page into one file. ### Option 2: Using Reaudit's Free Generator If you do not want to build it by hand, [Reaudit's free llms.txt generator](https://reaudit.io/free-tools/llms-txt-generator) crawls your site, extracts page titles and meta descriptions, and assembles a ready-to-publish file. Enter your domain, review the auto-generated output, edit descriptions where needed, and download the file. The tool formats everything according to the specification so you do not have to worry about syntax. ## Real-World Examples by Industry ### SaaS Platform ``` # CloudMetrics > Real-time analytics platform for SaaS businesses. ## Product - [Features](https://cloudmetrics.io/features): Dashboard, reporting, and integrations. - [Pricing](https://cloudmetrics.io/pricing): Free tier through enterprise plans. ## Resources - [API Documentation](https://cloudmetrics.io/docs): REST API with code examples. - [Blog](https://cloudmetrics.io/blog): Product updates and analytics guides. ``` ### E-commerce Store ``` # GreenThread > Sustainable fashion retailer shipping across the EU. ## Collections - [Women's Dresses](https://greenthread.com/women/dresses): Organic cotton seasonal collection. - [Men's Outerwear](https://greenthread.com/men/outerwear): Recycled material jackets and coats. ## Customer Info - [Shipping Policy](https://greenthread.com/shipping): Free EU delivery over €50. - [Returns](https://greenthread.com/returns): 30-day return window. ``` ### B2B Services ``` # BrightPath Consulting > Digital transformation consulting for mid-market companies. ## Services - [AI Strategy](https://brightpath.io/ai-strategy): Assessment and implementation roadmap. - [Data Engineering](https://brightpath.io/data): Pipeline design and cloud migration. ## Proof - [Case Studies](https://brightpath.io/cases): Client results across 12 industries. ``` ## Best Practices **Keep it lean.** 10 to 20 URLs is the sweet spot. Stuffing 50+ links dilutes the signal and some crawlers may truncate the file. **Prioritize evergreen content.** Product pages, help docs, and foundational guides are better candidates than time-sensitive blog posts. **Match your robots.txt.** If your `robots.txt` blocks PerplexityBot or ClaudeBot, they will never reach your llms.txt. Make sure AI bots you want to read the file are explicitly allowed. **Add schema markup to linked pages.** llms.txt points AI crawlers to your best content; schema markup on those pages (FAQPage, HowTo, Article, Organization) helps AI models extract structured answers from them. The combination is more powerful than either alone. **Update quarterly.** Remove dead links, add new important pages, refresh descriptions. Treat it like your sitemap, a living document, not a set-and-forget file. **Version control it.** Keep a copy in Git. You will want to track changes over time, especially as the specification evolves. ## Common Mistakes **Using HTML instead of Markdown.** The spec calls for Markdown. HTML will not be parsed correctly by tools designed for the llms.txt format. **Blocking AI bots in robots.txt.** This is the single most common issue. Your llms.txt is invisible if the bots cannot access it. **Overloading with too many URLs.** More is not better. A focused list of your best pages outperforms a dump of every URL on your site. **Forgetting to test.** After uploading, verify the file returns a 200 status code and the content renders correctly as plain text. **Neglecting the linked pages themselves.** An llms.txt file pointing to poorly structured pages without schema markup is a missed opportunity. Optimize the destination pages too. ## Where llms.txt Fits in Your AI Visibility Strategy llms.txt is one piece of a broader AI optimization stack. It works best alongside three other elements: **Sitemaps** ensure all your pages are discoverable by search crawlers. **robots.txt** controls which bots can access your site and which directories they can reach. **Schema markup** gives AI models the structured data they need to extract precise answers. When all four layers are aligned — sitemap for discovery, robots.txt for access control, llms.txt for context, and schema for structured answers — you give AI platforms the best possible conditions to cite your content accurately. For monitoring, [Reaudit](https://reaudit.io) tracks which AI bots visit your site (GPTBot, ClaudeBot, PerplexityBot, and 50+ others), which pages they crawl most frequently, and whether your brand is being cited in AI-generated answers across 11 platforms. ## Checklist ✅ Create `/llms.txt` with 10-20 URLs and brief descriptions in Markdown format. ✅ Optionally create `/llms-full.txt` with full page content. ✅ Verify the file returns a 200 response: `curl -I https://yourdomain.com/llms.txt` ✅ Confirm `robots.txt` allows PerplexityBot, ClaudeBot, GPTBot, and OAI-SearchBot. ✅ Add FAQPage, HowTo, and Article schema to pages listed in llms.txt. ✅ Monitor server logs for AI bot user-agents fetching the file. ✅ Schedule quarterly reviews to update URLs and descriptions. ## FAQ **What is llms.txt and who created it?** llms.txt is a Markdown file that helps AI crawlers understand your site's most important pages. It was proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024. The specification is published at llmstxt.org. **How does llms.txt differ from robots.txt?** robots.txt controls crawl access (allow or disallow). llms.txt provides context — a curated index with descriptions that AI models can use to understand what your site covers. Both files should be present. **Do AI bots actually read llms.txt right now?** Adoption is still early. No major LLM provider has officially committed to using it as a citation signal. Some evidence shows Microsoft and OpenAI bots visiting llms.txt files, and several AI companies publish their own. It is a low-cost implementation worth having. **How many URLs should I include?** Aim for 10 to 20 high-value evergreen pages. Including too many dilutes the signal and some crawlers may truncate longer files. **Does llms.txt affect Google rankings?** No. Google's traditional search relies on sitemaps, structured data, and its own ranking algorithms. llms.txt is designed for AI-generated answers, not traditional SERP rankings. **Can I generate llms.txt automatically?** Yes. [Reaudit's free llms.txt generator](https://reaudit.io/free-tools/llms-txt-generator) crawls your site and produces a ready-to-publish file formatted to the specification. **Should I also create llms-full.txt?** If your site has substantial documentation or knowledge base content, yes. The full-text companion file lets AI models ingest your complete content in a single request, which can improve the accuracy of how they represent your brand. **How do I verify AI bots are reading my llms.txt?** Check your server access logs for user-agents like PerplexityBot, ClaudeBot, and GPTBot. Reaudit's bot tracking dashboard surfaces these requests automatically across React, WordPress, Webflow, and Wix sites. ![Triantafyllos Rose Samaras - Author](https://reaudit.io/rose-avatar.png) About the Author #### Triantafyllos Rose Samaras Founder Triantafyllos Rose Samaras is the founder and CEO of Reaudit, the pioneering AI Search Visibility Platform that helps businesses understand and optimize how they appear across AI search engines. Recognizing that 25% of online searches now happen through AI platforms like ChatGPT, Claude, and Perplexity, Triantafyllos identified a critical market gap: traditional SEO tools were completely blind to this new search paradigm. While companies invested millions in Google optimization, they had zero visibility into how AI systems perceived, cited, and recommended their brands. Reaudit was built to answer the question every modern business needs to ask: "How does AI see my brand?" Based in Greece, Triantafyllos is building a globally competitive AI company, proving that innovation can come from anywhere. He is passionate about helping businesses navigate the transition from traditional search to AI-powered discovery. [](https://www.linkedin.com/in/triantafyllossamaras/) [](mailto:rose@leadflow.tech) --- ## Article Metadata - **Word Count:** 1797 - **SEO Score:** 88/100 - **Readability Score:** 50.8/100 - **Keyword Density:** 0.08% - **Language:** en - **Content Type:** tutorial ### Content Structure **H1 Headings:** llms.txt Guide: The Complete Tutorial for AI Crawlers – What It Is, How to Create One, and Why It Matters **H2 Headings (13):** 1. Quick Answer, 2. What Is llms.txt and Where Did It Come From?, 3. Specification Format – How llms.txt Is Structured, 4. How llms.txt Differs From robots.txt, 5. Which AI Bots Actually Read llms.txt?, 6. Step‑by‑Step Guide to Creating an llms.txt File, 7. Real‑World llms.txt Example Files, 8. Best Practices for llms.txt Configuration, 9. Common Mistakes and How to Avoid Them, 10. Integrating llms.txt Into a Broader AI Optimization Strategy, 11. Actionable Checklist (Copy‑Paste Ready), 12. Conclusion & Next Steps, FAQ **H3 Headings (5):** 6.1 Manual Creation, 6.2 Using Reaudit’s Free Generator, 7.1 SaaS Platform, 7.2 E‑commerce Store, 7.3 Digital Agency ### Internal Links - [robots.txt for AI](https://reaudit.io/glossary/robots-txt) - [llms.txt: Definition & Examples](https://reaudit.io/glossary/llms-txt) - [Free LLMs.txt Generator Tool](https://reaudit.io/free-tools/llms-txt-generator) --- ## Structured Data (JSON-LD) ```json { "@context": "https://schema.org", "@graph": [ { "@type": "Organization", "name": "Reaudit", "url": "https://reaudit.io", "logo": { "@type": "ImageObject", "url": "https://reaudit.io/reaudit-avatar.png", "width": 600, "height": 60 }, "sameAs": [ "https://twitter.com/reaudit", "https://www.linkedin.com/company/reaudit" ] }, { "@type": "WebPage", "@id": "https://reaudit.io/llms-txt-guide-complete-tutorial", "url": "https://reaudit.io/llms-txt-guide-complete-tutorial", "name": "llms.txt: The Complete Guide: What It Is, How to Create One, and Why It Matters for AI Search", "description": "Complete llms.txt guide: what the specification says, who created it (Jeremy Howard, Answer.AI), current bot adoption status, step by step setup, and best practices. Free generator included.", "inLanguage": "en", "isPartOf": { "@type": "WebSite", "url": "https://reaudit.io" }, "dateModified": "2026-02-16T19:59:43.717Z", "primaryImageOfPage": { "@type": "ImageObject", "url": "https://reaudit.io/article-images/og-llms-txt-the-complete-guide-wh-1200x630-2026-02-16.jpg" } }, { "@type": "Article", "headline": "llms.txt: The Complete Guide: What It Is, How to Create One, and Why It Matters for AI Search", "datePublished": "2026-02-13T22:47:31.835Z", "dateModified": "2026-02-16T19:59:43.717Z", "author": { "@type": "Person", "name": "Triantafyllos Rose Samaras", "jobTitle": "Founder", "description": "Triantafyllos Rose Samaras is the founder and CEO of Reaudit, the pioneering AI Search Visibility Platform that helps businesses understand and optimize how they appear across AI search engines.\n\nRecognizing that 25% of online searches now happen through AI platforms like ChatGPT, Claude, and Perplexity, Triantafyllos identified a critical market gap: traditional SEO tools were completely blind to this new search paradigm. While companies invested millions in Google optimization, they had zero visibility into how AI systems perceived, cited, and recommended their brands.\n\nReaudit was built to answer the question every modern business needs to ask: \"How does AI see my brand?\"\n\nBased in Greece, Triantafyllos is building a globally competitive AI company, proving that innovation can come from anywhere. He is passionate about helping businesses navigate the transition from traditional search to AI-powered discovery.", "image": { "@type": "ImageObject", "url": "https://reaudit.io/rose-avatar.png" }, "sameAs": [ "https://www.linkedin.com/in/triantafyllossamaras/", "mailto:rose@leadflow.tech" ] }, "publisher": { "@id": "https://reaudit.io/#organization" }, "description": "Complete llms.txt guide: what the specification says, who created it (Jeremy Howard, Answer.AI), current bot adoption status, step by step setup, and best practices. Free generator included.", "mainEntityOfPage": { "@id": "https://reaudit.io/blog/llms-txt-guide" }, "wordCount": 1919, "articleBody": "llms.txt is a Markdown file placed at the root of your website that tells AI crawlers which pages matter most and how to describe them. Think of it as a curated table of contents written specifically for large language models. This guide covers what the specification actually says, who proposed it, which bots currently read it, and how to create one for your site, including an honest assessment of where adoption stands right now. Quick Answer Create a Markdown file called llms.txt listing 10 to 20 important URLs with brief descriptions. Host it at https://yourdomain.com/llms.txt . Make sure AI bots are not blocked in your robots.txt . Keep it in sync with your sitemap and schema markup. The whole thing takes about 20 minutes to set up manually, or you can use Reaudit's free llms.txt generator to automate it. Where llms.txt Came From The llms.txt specification was proposed by Jeremy Howard, co-founder of Answer.AI (and previously co-founder of fast.ai), on September 3, 2024. The idea was straightforward: websites already have robots.txt to tell crawlers what they can and cannot access, but there was nothing to help LLMs understand what a site is actually about and which pages are most important. Howard's proposal defined two files: /llms.txt as a navigation aid listing key pages with descriptions, and /llms-full.txt as an optional companion containing the full content of the site in a single Markdown document. The specification lives at llmstxt.org . The concept gained traction quickly. Notable early adopters include Anthropic, Stripe, Cursor, Mintlify, Zapier, and FastHTML. As of mid 2025, around 951 domains had published llms.txt files, a tiny fraction of the web, but growing. The Format: How llms.txt Is Structured The specification is deliberately simple. The file uses standard Markdown with a few conventions: An H1 heading ( # ) with the project or brand name. An optional blockquote ( > ) with a one-line summary. H2 headings ( ## ) to group pages into logical sections. Bullet-point links in the format - [Title](URL): Brief description . Here is a minimal example: # Acme SaaS > Cloud-based invoicing platform for small and medium businesses. ## Key Pages - [Home](https://acme.io): Main landing page with product overview. - [Pricing](https://acme.io/pricing): Subscription tiers and feature comparison. ## Documentation - [API Guide](https://acme.io/docs/api): REST API reference for developers. - [Getting Started](https://acme.io/docs/start): Quickstart tutorial for new users. ## Support - [Help Center](https://acme.io/help): FAQs and troubleshooting guides. The /llms-full.txt companion file follows the same Markdown format but contains the full text content of the site's key pages, concatenated into a single document. This lets LLMs ingest your entire knowledge base in one request. How llms.txt Differs From robots.txt These two files solve different problems. robots.txt controls access — it tells crawlers which URLs they are allowed or disallowed from visiting. It is a set of directives, not content. llms.txt provides context. It does not control access; it offers a human-readable summary that AI models can use to understand what your site covers and which pages to prioritize. The two files complement each other: robots.txt must allow the AI bots you want to reach your llms.txt , otherwise the file will never be fetched. Which AI Bots Actually Read llms.txt? An Honest Assessment This is where many guides oversell the current state of things, so here is the straight picture. No major LLM provider has officially committed to using llms.txt as a ranking or citation signal. Google's John Mueller stated in 2025 that Google has not adopted it. Analysis by Averi of 1,000 domains in August 2025 found zero visits from major LLM crawlers (GPTBot, ClaudeBot, PerplexityBot) to llms.txt files specifically. That said, the picture is more nuanced than \"nobody reads it.\" Profound's data showed some evidence of Microsoft and OpenAI bots crawling llms.txt files. Several AI platforms have adopted it for their own sites (Anthropic publishes one at anthropic.com/llms.txt). And the specification is gaining mindshare in the developer community, it is a low-effort implementation that positions your site for whatever comes next. The honest take: llms.txt is an emerging convention, not an established standard. Implementing it costs almost nothing and cannot hurt your AI visibility. Whether it will become a meaningful signal depends on whether major AI providers decide to officially adopt it. The early signs are promising but not definitive. Step-by-Step: Creating an llms.txt File Option 1: Manual Creation Pick 10 to 20 of your most important pages. These should be your highest value evergreen content: homepage, product pages, pricing, key guides, help center, and case studies. Write a one line summary for each page (keep descriptions under 20 words). Group them under logical H2 headings. Save as llms.txt in UTF-8 encoding and upload to your web root. Test it by running curl -I https://yourdomain.com/llms.txt you should get a 200 response. If you also want to create llms-full.txt , concatenate the full Markdown content of each listed page into one file. Option 2: Using Reaudit's Free Generator If you do not want to build it by hand, Reaudit's free llms.txt generator crawls your site, extracts page titles and meta descriptions, and assembles a ready-to-publish file. Enter your domain, review the auto-generated output, edit descriptions where needed, and download the file. The tool formats everything according to the specification so you do not have to worry about syntax. Real-World Examples by Industry SaaS Platform # CloudMetrics > Real-time analytics platform for SaaS businesses. ## Product - [Features](https://cloudmetrics.io/features): Dashboard, reporting, and integrations. - [Pricing](https://cloudmetrics.io/pricing): Free tier through enterprise plans. ## Resources - [API Documentation](https://cloudmetrics.io/docs): REST API with code examples. - [Blog](https://cloudmetrics.io/blog): Product updates and analytics guides. E-commerce Store # GreenThread > Sustainable fashion retailer shipping across the EU. ## Collections - [Women's Dresses](https://greenthread.com/women/dresses): Organic cotton seasonal collection. - [Men's Outerwear](https://greenthread.com/men/outerwear): Recycled material jackets and coats. ## Customer Info - [Shipping Policy](https://greenthread.com/shipping): Free EU delivery over €50. - [Returns](https://greenthread.com/returns): 30-day return window. B2B Services # BrightPath Consulting > Digital transformation consulting for mid-market companies. ## Services - [AI Strategy](https://brightpath.io/ai-strategy): Assessment and implementation roadmap. - [Data Engineering](https://brightpath.io/data): Pipeline design and cloud migration. ## Proof - [Case Studies](https://brightpath.io/cases): Client results across 12 industries. Best Practices Keep it lean. 10 to 20 URLs is the sweet spot. Stuffing 50+ links dilutes the signal and some crawlers may truncate the file. Prioritize evergreen content. Product pages, help docs, and foundational guides are better candidates than time-sensitive blog posts. Match your robots.txt. If your robots.txt blocks PerplexityBot or ClaudeBot, they will never reach your llms.txt. Make sure AI bots you want to read the file are explicitly allowed. Add schema markup to linked pages. llms.txt points AI crawlers to your best content; schema markup on those pages (FAQPage, HowTo, Article, Organization) helps AI models extract structured answers from them. The combination is more powerful than either alone. Update quarterly. Remove dead links, add new important pages, refresh descriptions. Treat it like your sitemap, a living document, not a set-and-forget file. Version control it. Keep a copy in Git. You will want to track changes over time, especially as the specification evolves. Common Mistakes Using HTML instead of Markdown. The spec calls for Markdown. HTML will not be parsed correctly by tools designed for the llms.txt format. Blocking AI bots in robots.txt. This is the single most common issue. Your llms.txt is invisible if the bots cannot access it. Overloading with too many URLs. More is not better. A focused list of your best pages outperforms a dump of every URL on your site. Forgetting to test. After uploading, verify the file returns a 200 status code and the content renders correctly as plain text. Neglecting the linked pages themselves. An llms.txt file pointing to poorly structured pages without schema markup is a missed opportunity. Optimize the destination pages too. Where llms.txt Fits in Your AI Visibility Strategy llms.txt is one piece of a broader AI optimization stack. It works best alongside three other elements: Sitemaps ensure all your pages are discoverable by search crawlers. robots.txt controls which bots can access your site and which directories they can reach. Schema markup gives AI models the structured data they need to extract precise answers. When all four layers are aligned — sitemap for discovery, robots.txt for access control, llms.txt for context, and schema for structured answers — you give AI platforms the best possible conditions to cite your content accurately. For monitoring, Reaudit tracks which AI bots visit your site (GPTBot, ClaudeBot, PerplexityBot, and 50+ others), which pages they crawl most frequently, and whether your brand is being cited in AI-generated answers across 11 platforms. Checklist ✅ Create /llms.txt with 10-20 URLs and brief descriptions in Markdown format. ✅ Optionally create /llms-full.txt with full page content. ✅ Verify the file returns a 200 response: curl -I https://yourdomain.com/llms.txt ✅ Confirm robots.txt allows PerplexityBot, ClaudeBot, GPTBot, and OAI-SearchBot. ✅ Add FAQPage, HowTo, and Article schema to pages listed in llms.txt. ✅ Monitor server logs for AI bot user-agents fetching the file. ✅ Schedule quarterly reviews to update URLs and descriptions. FAQ What is llms.txt and who created it? llms.txt is a Markdown file that helps AI crawlers understand your site's most important pages. It was proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024. The specification is published at llmstxt.org. How does llms.txt differ from robots.txt? robots.txt controls crawl access (allow or disallow). llms.txt provides context — a curated index with descriptions that AI models can use to understand what your site covers. Both files should be present. Do AI bots actually read llms.txt right now? Adoption is still early. No major LLM provider has officially committed to using it as a citation signal. Some evidence shows Microsoft and OpenAI bots visiting llms.txt files, and several AI companies publish their own. It is a low-cost implementation worth having. How many URLs should I include? Aim for 10 to 20 high-value evergreen pages. Including too many dilutes the signal and some crawlers may truncate longer files. Does llms.txt affect Google rankings? No. Google's traditional search relies on sitemaps, structured data, and its own ranking algorithms. llms.txt is designed for AI-generated answers, not traditional SERP rankings. Can I generate llms.txt automatically? Yes. Reaudit's free llms.txt generator crawls your site and produces a ready-to-publish file formatted to the specification. Should I also create llms-full.txt? If your site has substantial documentation or knowledge base content, yes. The full-text companion file lets AI models ingest your complete content in a single request, which can improve the accuracy of how they represent your brand. How do I verify AI bots are reading my llms.txt? Check your server access logs for user-agents like PerplexityBot, ClaudeBot, and GPTBot. Reaudit's bot tracking dashboard surfaces these requests automatically across React, WordPress, Webflow, and Wix sites. About the Author Triantafyllos Rose Samaras Founder Triantafyllos Rose Samaras is the founder and CEO of Reaudit, the pioneering AI Search Visibility Platform that helps businesses understand and optimize how they appear across AI search engines. Recognizing that 25% of online searches now happen through AI platforms like ChatGPT, Claude, and Perplexity, Triantafyllos identified a critical market gap: traditional SEO tools were completely blind to this new search paradigm. While companies invested millions in Google optimization, they had zero visibility into how AI systems perceived, cited, and recommended their brands. Reaudit was built to answer the question every modern business needs to ask: \"How does AI see my brand?\" Based in Greece, Triantafyllos is building a globally competitive AI company, proving that innovation can come from anywhere. He is passionate about helping businesses navigate the transition from traditional search to AI-powered discovery.", "speakable": { "@type": "SpeakableSpecification", "cssSelector": [ ".article-intro", ".faq-section" ] }, "about": [ { "@type": "Thing", "name": "AI Search Visibility", "description": "Optimising website content to be discoverable and accurately represented by AI-powered search and generative models." }, { "@type": "Thing", "name": "llms.txt", "description": "A plain‑text file that provides LLM crawlers with curated page summaries for better AI citation." } ], "mentions": [ { "@type": "Organization", "name": "Reaudit", "url": "https://reaudit.io", "sameAs": "https://reaudit.io" } ], "timeRequired": "PT10M", "image": { "@type": "ImageObject", "url": "https://reaudit.io/article-images/og-llms-txt-the-complete-guide-wh-1200x630-2026-02-16.jpg", "width": 1200, "height": 630 } }, { "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "What is llms.txt and how does it differ from robots.txt?", "acceptedAnswer": { "@type": "Answer", "text": "llms.txt provides a human‑readable index of important pages for LLM crawlers, while robots.txt controls crawl access. Both files should coexist." } }, { "@type": "Question", "name": "Which AI bots actually read llms.txt?", "acceptedAnswer": { "@type": "Answer", "text": "PerplexityBot, ClaudeBot, Google‑Extended (experimental), and OpenAI’s OAI‑SearchBot have publicly confirmed support." } }, { "@type": "Question", "name": "Can I generate llms.txt automatically?", "acceptedAnswer": { "@type": "Answer", "text": "Yes. Reaudit offers a free generator that crawls your site and outputs a ready‑to‑publish file." } }, { "@type": "Question", "name": "How many URLs should I include?", "acceptedAnswer": { "@type": "Answer", "text": "Best practice is 10‑20 high‑value, evergreen pages. Over‑loading reduces relevance and may be truncated by some bots." } }, { "@type": "Question", "name": "Do search engines like Google use llms.txt for ranking?", "acceptedAnswer": { "@type": "Answer", "text": "No. Google’s classic index relies on sitemaps and schema. llms.txt only influences generative AI answers, not traditional SERP rankings." } }, { "@type": "Question", "name": "Do I need schema markup on the pages listed in llms.txt?", "acceptedAnswer": { "@type": "Answer", "text": "While not required, adding FAQPage, HowTo, or Product schema improves the quality of citations that LLMs can extract." } }, { "@type": "Question", "name": "How can I verify that AI bots are reading my llms.txt?", "acceptedAnswer": { "@type": "Answer", "text": "Check server logs for user‑agents like “PerplexityBot/1.0”. Reaudit’s dashboard surfaces these requests automatically." } }, { "@type": "Question", "name": "Is llms.txt an official standard?", "acceptedAnswer": { "@type": "Answer", "text": "It is not yet an IETF RFC, but it is widely recognised in the AI‑search community and documented in Reaudit’s Glossary." } } ] } ] } ``` --- ## How to Cite This Article **APA Style:** Reaudit. (2026). *llms.txt: The Complete Guide: What It Is, How to Create One, and Why It Matters for AI Search*. Retrieved from https://reaudit.io/blog/llms-txt-guide-complete-tutorial **MLA Style:** "llms.txt: The Complete Guide: What It Is, How to Create One, and Why It Matters for AI Search." *Reaudit*, 2026, reaudit.io/blog/llms-txt-guide-complete-tutorial. **Chicago Style:** Reaudit. "llms.txt: The Complete Guide: What It Is, How to Create One, and Why It Matters for AI Search." Accessed February 16, 2026. https://reaudit.io/blog/llms-txt-guide-complete-tutorial. --- *This content was generated using Reaudit's AI-powered content generation system.*