What helps every system, and what is still vendor-specific
robots.txt is the front door — and it is often closed by accident
Every web crawler, including those run by AI assistants, checks robots.txt before fetching anything else. The file lives at the root of the domain and uses a simple User-agent / Allow / Disallow grammar. The bots most commonly relevant in 2026 are GPTBot (OpenAI's training and search crawler), OAI-SearchBot and ChatGPT-User (OpenAI's search-time and on-demand fetchers), ClaudeBot and ClaudeBot-User (Anthropic), CCBot (Common Crawl, the dataset behind many open-source LLMs), Google-Extended (Google's AI training opt-out signal), and Applebot-Extended. Many sites still ship a default robots.txt that blocks one or more of these — often without anyone noticing. A deliberate stance is fine; a silent block by inheritance is not.
sitemap.xml with lastmod tells agents what changed and when
A sitemap is an XML file listing every public URL on the site, ideally with a lastmod date for each. Crawlers use it as a fast index of what exists; without it they have to follow links and may miss pages with shallow internal linking. The lastmod field is what AI agents actually read for freshness — a sitemap with last-modified dates from 2018 looks abandoned even if the underlying pages are current. Generating a sitemap is built into most static-site generators and content-management systems; if a site has no sitemap.xml, that is the single fastest discoverability win available.
llms.txt: a proposed convention that Google has declined to support
Proposed in 2024 by Jeremy Howard, llms.txt is a plain-text or markdown file at the site root that gives AI models a curated index of the site's important content. Some assistant teams and documentation sites have adopted it. Vercel's agent-readability framework counts it as a positive site-level signal. A companion file, llms-full.txt, flattens key page content into one fetch. Google's May 2026 guidance lists llms.txt under the practices its systems do not look for, alongside other novel AI markup files; the company has stated that no new machine-readable formats are needed to appear in its generative AI features. So the call depends on which assistants matter for the site. Publishing llms.txt is cheap and may help with the smaller subset of assistants that read it, but Google has said it is not a Google-search lever, and it does not replace the standard work of clean sitemaps, accurate schema, and good crawlability.
AGENTS.md and markdown mirrors close the gap with coding and search agents
AGENTS.md, served at the site root or under /.well-known/, is a markdown file that gives coding agents (Cursor, Copilot, Claude Code, Codex — see the Cursor, Windsurf, and Claude Code comparison for the current options) the install, configuration, and usage information they need to generate correct code that uses your product. It is not relevant to every site, but for any product with an API or developer-facing surface it pays off well. Separately, a growing convention is to publish a markdown mirror — page.md alongside page.html — and add a link rel='alternate' type='text/markdown' on the HTML page. Servers that support content negotiation (Accept: text/markdown) return the markdown directly when an agent requests it. Both patterns spare the agent the work of parsing and stripping HTML chrome, which leads to cleaner citations.
Things to consider — privacy, consent and AI rules
robots.txt is a consent signal at the indexing level: it tells crawlers what they may fetch. It does not, on its own, address downstream use of personal data. Sites that publish testimonials, reviews, or any text written by identifiable people should remember that privacy laws such as CCPA in California, and equivalent regimes in other countries, still apply once AI agents read and quote the content. AI-specific rules in several jurisdictions add transparency and documentation obligations for systems that interact with consumers, and indirectly for the sites those systems read. The practical effect is to keep an honest data-handling notice, a working cookie banner, and a record of what content is published with what consent. A robots.txt that explicitly allows AI bots while the privacy notice says nothing about AI is a small but increasingly visible inconsistency.
Open your site's robots.txt in a browser. Does it allow GPTBot, ClaudeBot, CCBot, and Google-Extended? Most sites silently block at least one — usually because someone copied a defensive default years ago.