AI is advancing leaps and bounds every day and continues to be a significant source of research and “knowledge” on the internet. By scraping uncountable numbers of websites and other online resources, AI search engines and agents deliver data to users that–while not always accurate–is often relied upon for decision-making and other important functions. So it’s worth making sure that the information AI Bots are gathering from your website is as correct as possible.
In this article we look at using an llms.txt file to help AI Bots learn properly from your content–and to protect content that you’re prefer not be used to train its agents.
What is an llms.txt file?
llms.txt is a simple text file that is published on a website’s root directory. Its function is to give instructions to Large Language Models (LLMs) like ChatGPT or Gemini on how to interpret and use the content on the site.
It is similar to robots.txt (which talks to Google’s search crawlers), but this one is made just for AI systems.
What are the benefits of using an llms.txt file?
This article from Bodhost offers some great explanations:
- Preserves Content Integrity: Provides clear instructions to AI systems about which content should not be used for training or generation purposes, helping preserve the value of copyrighted information and creative works.
- Full Authority on AI Training Permissions: Allows website owners to specify exactly which parts of their site can be used for AI model training, guaranteeing sensitive or premium content remains protected.
- Improved Content Privacy: Enables website owners to designate private or confidential sections that should be omitted from AI processing, maintaining better control over data privacy.
- Enhanced AI Understanding: Helps AI systems better understand the context and structure of your content, leading to more accurate and relevant AI-generated responses about your site.
- Professional Communication: Provides a standardized and clear way to communicate with AI companies about content usage policies and licensing requirements.
How to Use Yoast's llms.txt Feature
If you have Yoast installed on your WordPress website, then this super easy!
Within Yoast, you’ll find a function that allows you to manage llms.txt directly from your WordPress dashboard—no technical work required. It’s even included in the free version of the plugin!
Go to Yoast SEO / Settings / Site features / APIs and enable the llms.txt feature. Yoast also gives you the option to customize the file if needed.
How to Manually Create a llms.txt File
If you don’t use Yoast on your website (although we highly recommend that you do), then you still have a way to add the file to your site.
1. Create a llms.txt file
llms.txt files should be structured and formatted in Markdown in a plain-text editor (like Notepad).
Read this article from Semrush for top tips and examples.
2. Upload it
Upload the llms.txt file to the root folder of your website (same place as robots.txt) via FTP or your hosting file manager. The file should be available at https://yourwebsite.com/llms.txt.
Additional Ways to Help AI Bots Learn
Use Yoast to Improve SEO and Structure
Yoast SEO also helps AI understand your content by:
- Generating XML sitemaps to guide crawlers.
- Adding schema markup for articles, FAQs, and more.
- Making suggestions for how to improve meta titles and descriptions.
Keep Your Content Fresh
AI tools favor updated, authoritative information. Adding regular – and relevant – news articles and blog posts is a great way to do this.
Do you need help with SEO on your website? Contact us today and let us know how we can help you!