Increased usage of AI is changing how people discover businesses and services online. While your website may be optimized for traditional search engines, large language models (LLMs) process your website’s information differently. Our new feature, llms.txt offers to bridge the gap. Yoast SEO generates a file that highlights the most important, up-to-date content on your website as an invitation for LLMs to get the right picture. It’s automatic, requires no technical setup, and is ready in one click.
Helping AI understand your website
Unlike search engines that regularly crawl and index websites, LLMs like ChatGPT and Google Gemini work differently. They don’t store website content for future use. Instead, they gather information in real time when responding to user queries.
This means LLMs often only access a small portion of a website while looking for answers. This is especially true for large websites such as news platforms or ecommerce stores. This can lead to incomplete or even inaccurate AI-generated responses. Not ideal if you’re aiming to improve your visibility in LLM-generated answers as part of your marketing strategy.
If you want to better understand what LLMs tend to look for when accessing websites, this guide on optimizing content for LLMs offers a helpful overview.
What is an llms.txt file?
The llms.txt file gives LLMs a suggested, pre-prepared slice of your website, highlighting your most important and up-to-date content.
Think of it like a helpful guide at the entrance of a large department store. Imagine you’re walking in looking for socks. Someone greets you and hands you a store map that highlights where the socks are, along with other key departments like shoes, checkout, and customer service. You don’t have to use the map, you can wander around on your own, but it makes it much easier to quickly find what you’re looking for.
In the same way, this file helps LLMs quickly identify the most relevant and useful parts of your website. While the models can still explore other areas, giving them clear guidance increases the chances that they’ll surface the right information in their responses.
How is it different from robots.txt?
robots.txt
- Tells bots what not to access
- Focuses on permission
- Used for search engine indexing and crawling
- Supported by traditional search engines
llms.txt
- Suggests what AI should read
- Focuses on guidance and clarity
- Helps AI answer user questions more accurately
- Designed for large language models like ChatGPT
How does Yoast SEO llms.txt work?
When you turn the feature on, it automatically generates an llms.txt file for your website, using a mix of relevant website data. It draws from:
- Your most recently updated content
- Technical SEO elements like your sitemap for context
- Descriptions you’ve added about your website
This offers large language models a website summary to understand what your website is about and what content is most important.
Managing your llms.txt file
The plugin automatically creates and maintains the llms.txt file for you, refreshing every week. You can preview the file to ensure it accurately reflects your brand and prioritizes the right content before implementation.
Want full control or prefer to manage it yourself? Learn how to manually add an llms.txt file to your website by visiting our developer documentation.
At Yoast, our mission is SEO for everyone
Setting up an llms.txt file manually may only be accessible to a technical few. By automating the process, we make it easier for all website owners to benefit from this new technology, without needing to dive into code.
At Yoast, we believe that everyone should have a say in how their content is seen and used. Especially as AI plays a bigger role in how people discover information online. That’s why we’ve introduced this feature as opt-in, so you can decide if and when it makes sense for your website. We’ve seen early signs that this is something more website owners are starting to think about.
Just as robots.txt tries to help search engines understand what to index, llms.txt suggests which parts of your website large language models should pay attention to.If you’d like to see what an llms.txt file looks like in practice, you can view the live version on yoast.com.
