The proposal, published by Jeremy Howard in September 2024, suggests adding a /llms.txt Markdown file to websites to provide LLM-friendly content: a short description of the site and a list of its most useful pages, written so a language model can read them easily.
Unlike robots.txt, llms.txt doesn’t allow or block anything. It is a guide, not a rule, and nothing obliges a crawler to read it.
Google’s guidance on its AI search features is direct: Google Search doesn’t use such files, you don’t need them to appear in Google’s AI features, and maintaining one for other services will neither help nor harm your visibility in Google Search. Other AI companies have not, as far as we can find, said their search features rely on it either.
So treat it as cheap and optional. It takes minutes to write, it may help tools that do read it, and it costs nothing if they don’t. It is never a substitute for allowing OAI-SearchBot and publishing clear pages. Read the full guide: llms.txt: what it is and what the evidence says.