The web is changing fast due to the emergence of Artificial Intelligence, or AI, and more people now use AI chatbots and voice assistants to find answers instead of traditional searches.
However, sometimes AI struggles to fully understand and scrape complex websites, missing important content. Without clear guidance, AI might pull outdated or incorrect information, which can hurt your brand and user trust.
In this article, we will discuss llms.txt: a simple, new file that shows AI exactly what part of your website to read and use. Like a map, it points AI to your important pages in a clean and clear format to deliver better search answers.
We will also explain why llms.txt matters and how to start using it effectively for your website.
Key Takeaways
- llms.txt is a special Markdown file to help AI models find and prioritise important website content clearly.
- It was proposed in September 2024 by Jeremy Howard to solve AI confusion from cluttered, ad-filled, and complex website code.
- Unlike robots.txt, llms.txt guides AI on important site pages without blocking access.
- Currently, major AI providers like OpenAI, Google, and Perplexity do not fully support llms.txt, but some companies have already adopted it.
- The file's structure includes an H1 title, blockquote summary, main sections with links, and an optional section for secondary info.
- Creating an llms.txt file prepares your site for AI-driven search growth and gives greater control over AI content use.
What is llms.txt?
llms.txt is a proposed special text file that can be used on websites to help large language models (LLMs) such as ChatGPT and other AI systems find and understand the most important information on a website or selected content.
A screenshot of the proposed standard by Jeremy Howard. Source: llmstxt.org
It was proposed in September 2024 by Jeremy Howard, a co-founder of Answer.AI. He created it as a way to solve a common problem for AI systems: websites often have complex pages with numerous HTML elements, ads, scripts, and navigation, which makes it difficult for AIs to extract the useful content they need.
It was introduced to help control how AI models train on information. It aims to protect original content while still helping AI learn. The llms.txt file gives AI models a curated, easier-to-read version of the content, making the training or answering process more efficient.
You can check his full proposal at llmstxt.org.
This file is written in Markdown, a simple format that's easy to read and write. Markdown lets websites organise the instructions clearly with headers, links, and notes, which makes it easier for AI to understand what's allowed or disallowed.
For developers and business owners, llms.txt provides better control over how their site is picked apart and understood by machines. That said, anyone with a website can use it to guide AI, from teachers sharing classroom guides to shops explaining return policies. Tools and plugins now help generate llms.txt files, making it quick to set up and update.
How It Could Impact SEO and GEO
While this is a good proposal, currently, no major LLM provider supports llms.txt yet — not OpenAI or even Perplexity. So if you set up the file, today's top AI models are not using it to guide how they read your website. To be fair, not yet.
llms.txt could change SEO and GEO by helping AI models like ChatGPT and Google's Gemini find and use your best content. Although they have already been scraping websites for answers since then, this could be a game-changer through prioritisation of content as well.
Google AI Overviews (AIOs) are reshaping zero-click searches by pulling in clear, reliable content from across the web. If your content is relevant to the query, it has a greater chance of being featured inside an AIO — driving visibility, strengthening your SEO, and even opening up new opportunities in GEO.
Helping AI Find Your Best Content Easily
AI models often have to guess which parts of your website matter most. Websites are built for people, not AI, and pages include menus, ads, scripts, and other distractions that can confuse AI.
Because AI systems work with limited "attention spans" or tokens, having a clean, organised file like llms.txt could help them use their resources better. Your important FAQ, product descriptions, or blogs might get noticed by AI more easily, making your site more relevant in AI answers.
When AI models have this guidance, they can generate better, more accurate responses based on your content. This can improve the user experience and boost your chances of being featured in AI chat answers or voice queries, where many internet users are heading today.
Improving Visibility in AI-Driven Search Results
As AI-powered search grows, traditional Google-style search results are not the only way people find answers anymore. AI chatbots and assistants generate direct answers to questions, and these answers often pull from multiple web sources. If your content is hard for AI to find or interpret, it won't get included as often.
If fully accepted by LLM providers, llms.txt could help your website show up better in these AI-generated summaries by making your important content easy to spot and reliable. This can lead to a boost in brand visibility even if users don't click through to your site right away.
Giving Website Owners Control Over Content Use
One key benefit of llms.txt is that it could give website owners a stronger say in how AI systems use their content. Instead of AI scraping everything possible, you can highlight what content is okay to use and what should be avoided.
If AI uses outdated or incomplete information, it could harm your brand.
Although llms.txt doesn't guarantee that all AI models will follow your instructions, it could send a clear signal, as some major AI players like Anthropic are already starting to respect these files as part of their data selection process.
The Difference Between robots.txt and llms.txt
Both llms.txt and robots.txt are text files you place in your website's root folder, but they serve very different roles. Please don't confuse it as interchangeable or llms.txt being a replacement of robots.txt.
The robots.txt file mainly tells search bots which parts of your website to avoid crawling or indexing. On the other hand, llms.txt solely acts as a guide for AI language models. While they sound similar and live in the same spot, their purpose and audience are quite distinct.
Crawl Control vs Content Guidance
robots.txt and llms.txt serve different purposes for your website and online visibility.
robots.txt does:
- Control which pages search engine crawlers can visit.
- Use directives like "Disallow" or "Allow" to manage crawling.
- Help prevent sensitive or duplicate content from being indexed.
- Improve site crawl efficiency by focusing bots on key pages.
llms.txt does:
- Provide a clean, curated selection of key content for AI models.
- Use Markdown format to organise content clearly.
- Guide AI on what to prioritise for training and content generation.
- Help AI produce better, more accurate answers using your site.
File Formats and How They Speak to Machines
robots.txt is a plain text file with simple crawl rules. It uses straightforward commands written in a specific syntax that search engine bots know to follow. These commands tell bots what they can or cannot crawl or index, making it a binary yes/no interaction.
llms.txt, however, is written in Markdown. This format allows websites to organise content with headings, links, summaries, and notes. It's designed to provide LLMs with structured, readable information, not just crawl instructions.
How Each Affects Search Engines and AI Models
robots.txt is a key part of traditional SEO and a well-established way to manage how search engines crawl and index websites. By setting these rules, website owners can control what appears in search results and protect sensitive or duplicate content from being shown.
In contrast, llms.txt is designed specifically for LLMs and AI systems rather than traditional search engines. It doesn't block or allow access to your site like robots.txt does. While robots.txt controls crawling, llms.txt helps the AI use your content wisely.
Why Both Are Needed for Strong SEO and GEO Strategy
Robots.txt is still non-negotiable for your website and SEO, and remains essential for managing how search engines gather and display your website's content. It will continue supporting your site's presence on traditional search engines, whether you have llms.txt on your site or not.
At the same time, llms.txt is becoming just as important in the AI-driven age. It works alongside robots.txt by guiding AI models toward your best and most helpful content.
Combining both files poses no risk to your SEO and can actually improve both SEO and GEO performance. robots.txt controls access for traditional crawlers, while llms.txt enhances how AI interprets and presents your content. Their combination can be a game-changer for staying visible and relevant in both classic search results and new AI-powered answer engines.
How to Create llms.txt File
Creating an llms.txt file is very easy since it is a Markdown file. You just need to plan based on your website content and use the necessary tools to create one conveniently.
Now, let's look at the steps to make your own.
1. Gather Your Most Important Content
To start your llms.txt file, you need to decide which pages and sections of your website are the most important for AI models to read and understand. Usually, these key pages include:
- Your homepage, because it gives an overview of your site.
- Product or service pages, detailing what you offer.
- Blog or resource articles that provide useful information.
- Frequently Asked Questions (FAQ) pages, which clear up common queries.
- Case studies or testimonials that showcase success stories.
- About Us page, which explains who you are.
- Contact information pages for getting in touch.
Choosing these pages helps AI focus on meaningful content that best represents your website. Avoid including pages like login screens, private data, or duplicate content to keep things clear and useful.
2. Format Your Content in Markdown
After gathering your content, format it using Markdown. This is a simple way to add structure, like headings, lists, and links that are easy for both humans and AI to read.
To make this easier, consider using tools like Markdown editors or generators. These tools help you write clean Markdown quickly without needing to learn the whole syntax.
Examples include:
- Markdowner, an easy-to-use Markdown editor.
- Typora, which gives a live preview while editing.
- Dillinger, an online Markdown editor accessible from any browser.
When formatting, use headings (# for H1, ## for H2) to group content logically. Lists (- or *) work well for bullet points, and links go in square brackets with URLs in parentheses.
3. Write Your llms.txt with Key Sections
Your llms.txt file follows a specific Markdown format designed to clearly guide AI models. Here's the basic structure it should have:
# Title
> Optional description goes here
Optional details go here
## Section name
- [Link title](https://link_url): Optional link details
## Optional
- [Link title](https://link_url)The parts include:
- H1 Title: This is a required heading using one "#" symbol. It's usually your website or project name.
- Blockquote Summary: A short paragraph starting with">" that summarises your website's purpose or main content.
- Optional Details: Paragraphs or lists giving more context about your site that aren't headings.
- H2 Sections: These divide your site into main areas using "##". Each section groups related pages.
- Link Lists: Under each H2, use bullet points with Markdown links [title](URL). After a colon, you can add short notes describing the page.
- Optional Section: An H2 section to help you mark less critical content that AI can skip if needed.
Here's a simple example showcasing the hierarchy and formatting:
# Your Website Name
> A brief description of your website
## Products
- [Product A](https://example.com/product-a): Overview and specs
- [Product B](https://example.com/product-b): Features and pricing
## Resources
- [Blog](https://example.com/blog): Latest articles
- [FAQ](https://example.com/faq): Common questions
## Optional
- [About Us](https://example.com/about): Company backgroundIf you want to view a real example, check out Anthropic's llms.txt file.
4. Save and Upload to Your Website Root
After writing your llms.txt file, save it as a plain text file named exactly "llms.txt". Its location is important. Upload it to the root directory of your website so it can be found at yourwebsite.com/llms.txt.
Place the file where your main website files are stored, usually the public folder accessible via FTP or your hosting control panel. If you have a content management system (CMS), check how to upload files to the root or ask your web administrator.
Once uploaded, test the file by entering yourwebsite.com/llms.txt in a browser. It should show your Markdown content clearly.
For example, we already have our own llms.txt. You can check it by visiting www.adonis.media/llms.txt.
It should look like this:
# Adonis Media
> Adonis Media is a full-service B2B growth agency (Brisbane & Glasgow), serving global clients in tech, SaaS, services, and solar since 2004.
Adonis Media’s `llms.txt` provides a clear, curated guide for LLMs to understand and access key site content.
## Core Services & Resources
- [Lifecycle Marketing](https://www.adonis.media/lifecycle-marketing): Lifecycle marketing insights and services
- [Inbound Marketing](https://www.adonis.media/inbound-marketing): Inbound campaign strategies
- [Database Reactivation](https://www.adonis.media/database-reactivation): AI-driven CRM reactivation
- [Branding](https://www.adonis.media/branding): Brand identity and messaging
- [Web Design](https://www.adonis.media/web-design): Conversion-focused web design
- [Digital Transformation](https://www.adonis.media/digital-transformation): Business transformation services
## Tools & Templates
- [ROAS Calculator](https://www.adonis.media/tools/roas-calculator): Return-on-ad-spend calculator
- [CAC Calculator](https://www.adonis.media/tools/cac-calculator): Customer acquisition cost calculator
- [Inbound Marketing ROI Calculator](https://www.adonis.media/tools/inbound-marketing-roi-calculator)
- [UTM Code Generator](https://www.adonis.media/tools/utm-code-generator)
- [SMART Goals Template](https://www.adonis.media/templates/smart-goals-template)
- [Marketing Growth Plan Template](https://www.adonis.media/templates/marketing-growth-plan-template)
- [Buyer Persona Template](https://www.adonis.media/templates/buyer-persona-template)
## Insights & Thought Leadership
- [Insights Hub](https://www.adonis.media/insights): Full library of marketing, SEO, and GTM articles
- [Selected Highlights](https://www.adonis.media/insights/inbound-strategy-for-b2b-growth): Example insight—“Inbound Strategy for B2B Growth”
- [SEO Guides](https://www.adonis.media/insights/the-ultimate-guide-to-seo): Comprehensive SEO guides
## Other Sections
- [Podcast](https://www.adonis.media/podcast): Episodes exploring growth topics
- [Success Stories](https://www.adonis.media/success-stories): Case studies and client results
- [Website Reviews](https://www.adonis.media/website-reviews): Critiques of real-world sites
## Optional (Secondary Resources)
- [Careers](https://www.adonis.media/careers)
- [Learning Centre](https://www.adonis.media/learning-centre)
- [About](https://www.adonis.media/about)5. Test and Update Your llms.txt Regularly
Creating your llms.txt file is just the start. You need to keep it fresh and accurate to get the best results from AI models. Regularly test your file by visiting yourwebsite.com/llms.txt to make sure it's still accessible and displays correctly. Any errors or broken links can confuse AI.
Schedule updates whenever you add new pages, change URLs, or make big content changes. It's a good idea to review your llms.txt at least once every few months.
Keep track of changes as well with version control. Save backups and note what was updated and when.
So, Should You Use llms.txt Now?
llms.txt is still emerging, unlike robots.txt, which has already been a standard for decades. We know it is still gaining traction and not yet universally supported, but it offers a promising way to shape how AI interacts with your site's content.
If you want to be an early adopter and are willing to bet on llms.txt becoming important, then go ahead and use it. Setting up an llms.txt file now can give you a head start in guiding AI on how to read and use your content, which might pay off as AI usage and evolution grow exponentially.
However, if you prefer to wait and see how AI platforms fully support and accept llms.txt, that's also fine. It's not yet a must-have for every site, so you can hold off without hurting your current SEO.
In fact, some big names in SEO and AI have started using it, while others haven't jumped on board yet.
Some companies that are already using llms.txt:
And those companies that are not yet supporting llms.txt:
In our honest opinion, since llms.txt helps AI models better understand and use your content, it's a good idea to prepare your website even if it's not yet universally adopted. Being early can shape how AI tools see your site and protect your content from misuse.
We have fully implemented llms.txt on our website, as you've seen above, since we rely on AI for most of our processes, and AI significantly contributes to our brand awareness online.
In time, as AI platforms mature, the llms.txt file will likely become a standard part of managing how your web content works with AI. It's a way to be ready for the future of search and content discovery while showing you care about how AI uses your site.
Further Learning
Looking to learn more? I've collected some fantastic resources that go hand-in-hand with this topic if you want to dive deeper.
- llmstxt.org — The /llms.txt File Proposal and Official Website
- searchenginejournal.com — Google Says LLMs.Txt Comparable To Keywords Meta Tag
- ppc.land — llms.txt Adoption Stalls as Major AI Platforms Ignore Proposed Standard
- youtube.com — Cube Design Channel Video: "How to Add llm.txt to WordPress - ChatGPT, Perplexity Optimisation"
- reddit.com — Reddit r/SEO Thread: "Has anyone here implemented the llms.txt file on their website? Is it actually useful?"
- reddit.com — Reddit r/LLMDevs Thread: "/llms.txt directory – A growing list of sites adopting the emerging llms.txt standard!"
Preparing Your Site for Smarter AI Search
llms.txt is an emerging tool that helps AI understand your website better. While it's still optional or not yet critical for all sites, it can improve how AI-powered tools access and present your content. Preparing your site now means you're ready for potential changes in search and AI technology.
If you want to learn how to implement llms.txt tailored to your site, book a free consultation with us. We'll help make your website AI-friendly and set you up to succeed in the rapidly evolving digital landscape.
Got a question in mind? Check out the FAQs below for quick answers!
