Core / Pillar 24 min read Published Updated
What Is llms.txt? (2026 Guide)
I still get the same question: what is llms.txt, and does it change whether ChatGPT or Perplexity cites you. Here is the spec, the 2026 adoption picture, and what my tests actually showed.
On this page
- 01
llms.txt is an optional Markdown convention for pointing agents at key pages, not a ranking control.
- 02
By mid-2026, published files were concentrated among developer tooling firms and a small slice of technology companies; major platforms had not made the format a first-class ranking input.
- 03
In my multi-domain tests, content quality moved citations; the file did not.
- 04
I treat the file as extra publisher metadata after canonicals and citable pages, and I only ship one when the map itself is cheap to keep honest.
What is llms.txt on a real domain
When someone asks what is llms.txt on a real domain, I show them a file rather than a definition. The ones I have pulled this year are mostly short Markdown documents: a title, a one-line summary, and a list of URLs with a sentence of description each. They are hand-curated enough to feel editorial, but flat enough to be parsed by a script. I keep a small folder of live examples, and the pattern is strikingly consistent across the domains that publish them. The file usually sits at /llms.txt, but the location is the least interesting part. What matters is the selection logic inside. When someone asks what is llms.txt, I point to that selection logic rather than the filename. For more, see AEO vs SEO.
A curated Markdown map of key URLs
Live examples I have archived tend to list five to forty URLs. The entries are not the homepage repeated; they are the pages a site's own team would send to a new engineer: API reference, docs quickstart, changelog, schema definitions, a few pillar guides. Each line usually carries a Markdown link followed by a colon and a short phrase. The descriptions run one clause, sometimes two. I rarely see persuasive adjectives or calls to action, and that is what makes them useful as a navigational map. They compress a sitemap into a mental model. One file I pulled recently covered a documentation product with sixteen entries, each description under ten words. The format stays legible because it does not try to summarize every page; it tries to mark the pages that matter for a specific task. That selectivity is the entire point, and it is also the reason the file ages quickly.
Not a crawl-permission file
The first distinction I make when someone asks about the file is that llms.txt does not grant or deny access. I have seen publishers assume it behaves like robots.txt, and that is not what the format does. robots.txt tells crawlers which paths they may fetch; llms.txt does not authenticate, whitelist, or block anything. A domain can publish a clean llms.txt and still keep sensitive paths disallowed in robots.txt. The two files can coexist, but they answer different questions. I treat robots.txt as an access boundary and llms.txt as an editorial pointer. The proposed convention does not include permission directives in the robots sense, and OpenAI's 2026 guidance frames it as an optional way to help agents understand structure, not as a control layer. If I need to manage crawling, I leave that to robots.txt and avoid expecting the Markdown file to do that job. That boundary is the clearest llms.txt explained around access: it is absent from the permission decision.
Why the question keeps coming
The question keeps coming because marketers want a lever for answer engines, and a text file is a very legible lever. I get asked about llms.txt in the same conversations where people ask more on what is answer engine optimization, because both feel like small changes that might shift how an AI answer cites a brand. The real reason the topic is sticky is that robots.txt and sitemaps are already familiar, so a third file looks like the missing piece. Yet the pressure is ahead of the evidence. I have seen teams publish llms.txt before they have fixed duplicate canonical tags or filled out their schema. The file becomes a symbol of AI-readiness, not necessarily a driver of it. That gap between marketer demand and what answer engines have said they consume is exactly where most of the confusion sits.
Video: Technical SEO for AI: Robots.txt, GPTBot & llms.txt Explained | 3.4. AEO Course by Ahrefs · Ahrefs
How the proposed spec is written
I read the proposed spec less as a standard and more as a set of conventions that emerged from AI-adjacent developer tools. When I trace the origin, I end up in GitHub discussions and library docs, not in a standards body with formal working groups. OpenAI's 2026 web publisher guidance describes llms.txt as an emerging, non-standard convention for exposing a concise Markdown map of a site's most authoritative resources. The phrase 'emerging, non-standard' matters because it tells me the format is still in the experimentation phase. I also keep returning to how the spec frames its audience: agents that need a quick map during a task, not providers that need a comprehensive crawl. That distinction shapes which of my clients I would ever ask to publish one. It also clarifies what is llms.txt as a task-time map rather than a training-time corpus.
A community proposal, not a formal standard
Reuters' 2026 coverage calls llms.txt a proposed web standard originating in the AI developer community, and that phrase is more precise than the shorthand I hear from marketers. A community proposal is not the same as a ratified spec. There is no single standards body that I can point to that has made it binding, and the format can still shift as implementations diverge. I treat it the way I treat early conventions like RSS in its first years: useful, widely discussed, but not something to build a compliance regime around. The existence of public discussion does not make it a formal standard. When a client asks whether they need to adopt it, I explain the difference between a documented convention people are trying and a requirement any major platform has adopted. As of mid-2026, I have not seen the second part happen. That is not a knock on the format; it is just where the record sits.
Aimed at inference-time retrieval
One of the cleaner ways I keep llms.txt from being misunderstood is to separate inference-time retrieval from training-time data collection. The file is intended for the moment an AI system is trying to answer a question or complete a task, not for the long process of assembling model weights. When a domain publishes a curated list of key pages, it is giving an agent a map to consult during a query, much like a librarian pointing to the reference shelf instead of handing over the whole archive. Reuters describes that distinction directly. The file's value, where it exists, is in reducing navigation friction for an agent that already has access to your site. It does not give you extra training presence or change how a model was built. I keep that separation in my own notes because it prevents me from promising that a Markdown file will alter a model's knowledge, as opposed to helping a system locate a page in the moment.
Optional metadata among several practices
OpenAI's guidance places llms.txt alongside clear canonical URLs, structured content, and API documentation as publisher-friendly metadata practices. That placement tells me the file is not a standalone ranking input; it is one of several ways to make a site easier for future agents to navigate. In practice, I treat it as a supporting artifact. A domain with messy canonical tags and no schema does not become easier to cite simply because it ships a tidy llms.txt. The file works best when it summarizes a site that is already technically coherent. I also see it as optional in the same way what is chatgpt shopping in 2026 is adjacent to, not essential to, a commerce site's AI strategy. The spectrum from optional metadata to required signal matters, and this file sits near the optional end. That positioning is llms.txt explained as a supporting artifact, not the main optimization.
llms.txt explained next to robots.txt
I am often asked to map llms.txt against robots.txt and XML sitemaps, because all three live at a domain root and all three sound like they talk to machines. That is llms.txt explained next to its two older neighbors. The simplest mental model I use is: robots.txt is about permission, a sitemap is about discovery, and llms.txt is about recommendation. One says what can be fetched, one lists what exists, and one highlights what matters. When I put the three files side by side in a technical audit, they rarely overlap in the way people fear. A page can appear in a sitemap but not in llms.txt, or be highlighted in llms.txt while still being disallowed in robots.txt. Understanding those differences keeps teams from replacing one file with another. The mistake I see most often is treating llms.txt as a new sitemap, when its job is narrower and more editorial.
robots.txt still controls access
robots.txt remains the file I check when a crawler needs to know whether it may fetch a path. It works on allow and disallow directives, and it is the closest thing a publisher has to an access boundary for well-behaved bots. llms.txt does not participate in that decision. If a team removes robots.txt and assumes llms.txt will keep sensitive pages out of an agent's context, that is a misreading of both files. In my audits, the permission layer stays separate from the editorial map. A clean llms.txt sitting next to an overly permissive robots.txt changes nothing about what a crawler can retrieve. I keep pointing teams back to robots.txt whenever the question is about crawling, indexing, or exposure. The Markdown file might tell an agent where to look, but it never tells an agent where it is allowed to go. That job belongs to robots.txt, and I have not seen a 2026 proposal that changes that.
Sitemaps still list URLs for discovery
An XML sitemap is an exhaustive inventory. In most crawls I run, it lists the full set of URLs a site wants discovered, including pages that would never earn a line in an llms.txt. That is exactly the difference: a sitemap optimizes for completeness, while llms.txt optimizes for priority. I have pulled sitemaps with tens of thousands of URLs and llms.txt files from the same domain with only a dozen entries. They serve different consumers. A search crawler uses the sitemap to find new or updated pages; an agent uses llms.txt to get a fast summary of where the domain's authority lives. I would not ask a sitemap to be concise, and I would not ask llms.txt to be comprehensive. The two can complement each other, but mapping one to the other rarely makes sense; the selection criteria are just not aligned. If I had to answer what is llms.txt here, it is the priority layer, not the inventory.
Why a third file showed up
A third file appeared because agents need a faster navigation surface than a raw crawl or an XML list can provide. When I look at Perplexity-style answer engines and agentic tools, the common pattern is a system arriving with a task and a short time budget. It does not want to parse a sitemap with thousands of entries; it wants a domain's owner to point at the ten pages that answer the likely questions. That is the same logic behind more on what is perplexity ai as a question-answering surface, the system benefits from curated context, not just more URLs. llms.txt attempts to provide that shorter map. The reason it has not replaced sitemaps is that discovery still requires completeness. The reason it has not replaced robots.txt is that navigation still requires permission. I see the third file as a response to a specific agent-driven need, not a replacement for the other two, and I keep my own expectations scoped to that need.
Who shipped llms.txt by mid-2026
When I hear “what is llms.txt” from a marketer, I usually check whether they mean the file format or the footprint it has actually built. By mid-2026, the footprint was narrow. I sampled domains in ecommerce, editorial SEO, SaaS, and developer tooling; most consumer and editorial categories had no llms.txt, while a slice of infrastructure and AI-adjacent companies had shipped one. That concentration tells you who the spec reached first. It did not reach the broader web.
Developer tooling and a thin tech slice
The files I was finding by mid-2026 followed a tight pattern: developer tooling firms, documentation-heavy products, and a small set of technology companies were the ones publishing llms.txt. That matches a June 2026 Reuters technology report describing adoption as concentrated among developer tooling and a small subset of technology companies. On ecommerce category pages and editorial sites, I rarely found one even when robots.txt and sitemap.xml were both present.
The early adopters were teams already building for AI agents, not teams treating this as a citation shortcut. In my monitoring, the file mostly appeared next to documentation hubs and changelogs, where someone had decided it was cheap to maintain. It still felt like a convention among people who read spec proposals, not a mainstream marketing file. That context is how I answer what is llms.txt for a non-technical team.
What platforms have acknowledged
Major AI platforms have acknowledged llms.txt, but the acknowledgment is careful. The same Reuters report notes that the format has been recognized by major platforms without being declared a first-class input to ranking or citation systems. From what I could verify on official pages in 2026, no major answer engine documentation listed llms.txt as a signal it promises to use for retrieval.
Acknowledgment usually took the form of a support article or forum post saying the file may be read in some agent contexts, not a statement that publishing it affects visibility. That distinction matters when a marketer asks what is llms.txt for SEO. The answer I kept coming back to was: an optional file some platforms know about, not an input they publish as ranking-relevant. Documentation treated it the same way, which is llms.txt explained as acknowledged but not activated.
Experimental and non-binding, as vendors put it
The language I saw most often from search and AI vendors was experimental and non-binding. In its coverage, Reuters reported that several vendors regard llms.txt this way and warned that the file’s presence or absence does not currently correlate with measurable improvements in AI search visibility. I found the same tone in official publisher guidance: support for the concept, no commitment on outcomes.
That leaves llms.txt as a community experiment that vendors are watching, not a lever with documented citation effects. For me, that is the honest wording to pass on to a founder or content lead. I can publish the file in ten minutes, but I cannot promise it changes how often ChatGPT cites a page.
Whether the file changed my citation counts
I did not rely on speculation. Across several domains I publish, I ran paired checks on whether adding llms.txt changed how often answer engines cited my pages. I measured the same questions before and after publishing, then followed OpenAI’s 2026 publisher guidance to understand what retrieval actually uses. The short version: the file did not move my citation counts. The content and pages that were already citable stayed citable; the file itself was not the reason. That experiment gave me a clearer answer to what is llms.txt as a citation variable: it was not one.
Files on multiple domains
I published llms.txt on multiple domains: a documentation-heavy project, a content site, and a small marketing site. On each one, I kept every other variable steady for a few weeks and tracked direct answer citations. The documentation domain already got cited in agentic answers before the file existed; it continued to get cited after. The marketing site’s thin pages did not start getting cited because I added a file.
The pattern repeated across domains: pages with clear claims, specific numbers, and a direct explanation showed up in answers, while pages that were hard to parse did not. My read from those tests is that llms.txt is not a substitute for a citable page. It may describe the page, but the model still has to retrieve and trust the underlying content.
Core retrieval still runs on the open web
OpenAI’s 2026 web publisher guidance states that core answer-engine behavior, including retrieval for ChatGPT-style systems, is driven by general web crawling, embeddings, and other proprietary signals rather than llms.txt. I read that after my tests and it matched what I saw: my pages appeared in answers because they were crawlable and relevant, not because the file pointed to them.
The guidance frames llms.txt as an optional way to help agents understand structure in contexts like tools-based browsing, which is a narrower use case than most marketers assume. When I asked what is llms.txt for ChatGPT-style retrieval, the honest field answer is that it is not the thing doing the lifting. The open web remains the primary corpus.
No guaranteed citation lift
OpenAI’s guidance is explicit that publishing llms.txt is not a guaranteed method to improve citation frequency or ranking in OpenAI products. I repeat that because it is easy to invert the logic: if a file exists, it must be used. My checks do not support that. I saw no reliable increase in citations after publishing, and I also saw pages get cited on domains where I had not published a file at all.
The honest conclusion is that llms.txt is a compatibility affordance, not a growth lever. For company leaders asking whether it matters, I’d answer: it may matter in future agentic tools, but in my 2026 tests it did not change the outcomes I was tracking.
GEO tests I ran that included the file
I did not test llms.txt in isolation forever. Across the last year, I logged more than ten strategy experiments around answer-engine visibility: listicle rewrites, structured summaries, AI-optimized video landing pages, and the llms.txt file. The point was to see what moved citations and what only added maintenance. I kept the same measurement method across all of them so one test did not pollute another. When I explain what is llms.txt inside that program, I call it one metadata treatment among many.
Ten-plus strategy experiments
I ran the experiments as one program on the same set of pages, changing one variable at a time. Some tests targeted format: I rewrote listicles from stacked introductions into direct answer blocks, then checked whether answer engines pulled the new structure. Others targeted metadata, including llms.txt, clean canonical URLs, and schema. A third group looked at AI-optimized video: I added timed summaries and answer-first transcripts beneath short explainer videos.
Each experiment had a fixed observation window and a repeatable set of question prompts. The llms.txt test was not the headline test; it was one metadata treatment among several. By the end, I had a clearer view of which changes were active and which were just site hygiene. The file landed in the hygiene bucket.
What moved the needle
What changed citations in my runs were citable, self-contained passages: a direct answer in the first 80 words, a named stat with context, a table that survived being read as plain text, and pages that answered one question fully instead of chasing eleven keywords. Rewrites that made claims explicit and removed vague category talk showed visible increases in cited answers.
The llms.txt file did not produce a change I could isolate. Canonical cleanup helped indirectly by stopping duplicate URLs from splitting authority. AI-optimized video helped when the page under the video supplied the answer, not when the file pointed to the page. I have no invented percentage to attach to those changes. I counted cited answers across repeated prompts and compared periods, and the winning side was always content structure over file metadata. That result is llms.txt explained next to the other variables I ran.
How I measured
I did not rely on one tool or one AI platform. I kept a fixed prompt set for each domain: question prompts that could plausibly cite the page, plus brand-adjacent and category prompts. I ran them in fresh sessions across ChatGPT, Perplexity, and Google AI Overviews where available, then logged whether a domain appeared in the answer, whether the page was linked, and whether the answer paraphrased a specific claim I could trace back.
I repeated the same prompts across multiple days, noted the spread, and looked for movement before and after each change. For llms.txt specifically, I added file-only changes and kept visible page content identical. That method is not a scientific panel, but it is inspectable. Anyone on my team could rerun the prompt set and see the same absence of lift.
What I put in the file when I ship one
When I do publish a file, I treat it like a hand-edited index, not a dump. The tests in the previous section taught me that the file is unlikely to change how often a model cites the site, so the only reason worth the effort is keeping it accurate enough that it still functions as a map. I write for a future agent that has already entered the site through another door and needs quick orientation, not for a crawler that needs everything. That orientation focus is my practical answer to what is llms.txt on a content team.
Which URLs earn a line
I don't start from the sitemap. I start from the handful of pages I would hand to a new team member if they only had ten minutes to understand what the site does. On a SaaS domain that usually means the home page, the main use-case pages, the docs index, one or two reference implementations, and a changelog if it is published consistently. On a publishing site it means the topic hub pages, the editorial policy, and the pages that define terminology other pages depend on. The test is whether a page answers a structural question rather than a single article question. If a page only matters because it contains one embedded code sample or one seasonal campaign, it usually doesn't earn a line. I would rather list seven URLs I can keep accurate than forty that drift within a quarter.
Short descriptions, not sales copy
When I write the Markdown blurbs, the rule is descriptive, not persuasive. The docs index line says "Platform API reference and authentication scope," not "Everything you need to launch faster." The home page line says "Product overview and current positioning," not a slogan. I keep descriptions under roughly a dozen words because longer filler adds no navigational signal. If a page has an official title that already describes the content, I reuse it almost verbatim rather than inventing a new phrase. I also avoid words like "leading," "best," or "trusted" in the file for the same reason I would avoid them in a database description: they don't help an agent decide whether the page matches the task. The file is instructions for navigation, not a conversion page.
Keeping the file from rotting
I budget one short review per quarter. I tie it to the same content audit where I check canonicals and remove outdated reference pages, so the llms.txt update doesn't become a separate chore. The review takes less than thirty minutes on most sites. I open the file, click each URL, and confirm the page still exists and still describes the same function. When a page moves, I update the line immediately and let the other references follow in the regular audit. I don't treat the file as a permanent record; I treat it as a pointer that has to keep pointing at the right thing. The failure mode I have seen on other domains is a file whose listed URLs redirect to the homepage or return a 404, which is worse than not having a file because the map now misleads. If I can't commit to the quarterly check, I don't publish the file at all.
My 2026 call on whether it matters
My 2026 call splits into three parts. The format is cheap to publish and technically coherent. Adoption is still thin outside a narrow developer slice. And the measurable impact on ChatGPT-style citations in my tests was indistinguishable from not having the file. None of that makes llms.txt pointless; it makes it a small, optional piece of metadata with a specific future use case rather than a ranking lever. That is my shorter answer when someone asks what is llms.txt in 2026.
Useful for future agents, maybe
The strongest case I can make for llms.txt is not current retrieval. It is the agent that browses a site under a task constraint, like finding every integration page or locating the right API endpoint. In that scenario, a curated map saves steps because the agent is reading fewer pages, not being asked to trust a summary of the site. Current ChatGPT-style answer engines retrieve from the open web and then compose an answer from extracted passages; the file is not the retrieval index and it is not the judgment layer. So I don't expect llms.txt to matter much for answer engines as they looked in mid-2026. I do expect a clean, accurate map to be a small advantage if tool-using agents start traversing sites directly, because the alternative is watching an agent infer structure from a nav menu. That use case is plausible, but it hasn't reached the point where a missing file costs a brand a meaningful number of citations.
What I prioritize instead
Before I spend time on llms.txt, I check the things that actually showed up in my citation logs. Canonicals come first, because a wrong canonical tells every parser that the page's authority belongs to a different URL. Structured content, especially FAQ and HowTo where the page genuinely answers a question, gave me more retrievable passages than a map of the site ever did. I also prioritize pages written so that a one-paragraph claim can be lifted without losing its meaning, because answer engines quote passages more often than they summarize whole domains. Internal links that label the destination plainly do more for an agent's navigation than a separate file in the root, and they are maintained by the same editorial process that maintains the content itself. llms.txt is the last piece I add, not the first, because it amplifies a map that already exists; it doesn't create one.
When I still publish one
I still publish a file when three conditions overlap. The first is that the domain has a small number of pages that genuinely define the brand, so the file is a real map rather than a padded list. The second is that I can update the file at least once a quarter without a separate maintenance process, because a stale file is more trouble than the missing file it was meant to replace. The third is that the team already has clean canonicals, clear structured content, and well-labeled internal links, so the file is not being asked to compensate for weaker foundations. When all three hold, publishing llms.txt costs almost nothing and gives future agents one more signal to work with. When any condition fails, I skip it and spend the same hour on the page content itself. My expectation on publish day is exactly zero citation lift; I publish it the way I would update a public API changelog, not the way I would launch an SEO campaign.
Frequently asked
I don't think most sites need one in 2026. It's a Markdown map, not a permission file, a proposed convention that gives language models a concise map of your most authoritative pages with brief descriptions. OpenAI calls it emerging and non-standard; Reuters notes adoption is still concentrated among developer tooling and a small subset of technology companies.
OpenAI's 2026 guidance says its ChatGPT-style retrieval is driven by general crawling, embeddings, and proprietary signals rather than llms.txt, and publishing the file is optional. I have not seen equivalent official Gemini documentation, so I would not assume Gemini treats it as a ranking input.
I explain it as a sitemap-shaped idea reframed for models. A sitemap lists URLs for crawlers; llms.txt is a Markdown file with a curated set of key pages and short descriptions meant to guide inference-time retrieval. Reuters describes it as mirroring robots.txt and sitemaps but for AI question-answering. In the shortest form, that is llms.txt explained for the sitemap comparison.
No, not by itself. OpenAI's 2026 guidance says publishing llms.txt is not a guaranteed method to improve citation frequency or ranking in its products. Reuters reports vendors view it as experimental and non-binding, with no current correlation to measurable AI search visibility. I treat it as infrastructure, not a citation switch.
I would include a concise Markdown list of your most authoritative URLs, each with a short description, and keep the file focused on resources an agent should actually read. OpenAI's guidance frames llms.txt alongside canonical URLs, structured content, and API documentation as publisher-friendly metadata, not as a ranking signal.
No. OpenAI calls it an emerging, non-standard convention. Reuters describes it as a proposed, experimental format with adoption concentrated among developer tooling firms and a small subset of tech companies, while major platforms acknowledge it without declaring it a formal standard. I treat it as an early convention, not an established spec.
