AIRANKS — The Authoritative Rankings for AI Web Content

AIRANKS measures AI visibility: we ask AI models real product and service questions, capture the complete answers as immutable observations, and publish what they contain — which brands were mentioned, which domains were cited, and which exact pages were linked. Every domain gets an AIR score from 1–10 (a decile of visibility in the active dataset; 0 means insufficient data), with the methodology in the open.

Skip to main content

AIRVER. FIGHTING — 2026

Artificial Intelligence Rankings
Does ChatGPT cite YOUR site?Get the Toolbar →

the files AI reads before it reads you

LLM Web Indexing Files

Seven surfaces decide how AI systems discover, read, cite, and train on your site. This page breaks down each one: what it is, what it provably does, how to implement it, and — the part most guides skip — which benefits are VERIFIED and which are still THEORY. AIR's own domain reports audit every file below on any site, free.

How to read the badges

VERIFIED a primary source confirms it: the spec, the vendor's own documentation, or a study with methodology.

COMMUNITY-REPORTED practitioners observe it; no vendor confirmation.

THEORY plausible, widely repeated, unproven. Not a reason to skip it — a reason to know what you're buying.

1. llms.txt

A curated markdown index at /llms.txt, proposed by Jeremy Howard (Answer.AI) in September 2024. Where robots.txt tells crawlers what they may not read, llms.txt tells AI readers what they should read: an H1 title, a blockquote summary, then H2 sections of annotated links, with an ## Optional section for the skippable tail.

  • VERIFIED: coding agents and IDE tools (Cursor, Claude Code, MCP-based agents) fetch it at inference time; OpenAI, Anthropic, and Google all publish one for their own API docs; adoption sits around 6–9% of major sites (Thunderbit/Rankability crawls, mid-2026).
  • VERIFIED (negative): Google Search does not use it — John Mueller said so directly (June 2025). It is not a ranking signal.
  • THEORY: that chat assistants (ChatGPT, Claude, Perplexity consumer apps) consult it when answering. No vendor has confirmed this; publish it for the agent ecosystem that provably does.
  • Implement: hand-write it — it's a curation, not an export. Keep it under a screenful, describe each link's payoff, put your API/docs first. A larger llms-full.txt can carry the whole story (~1% adoption; cheap to add).
  • Tooling: Mintlify auto-generates for docs sites; Yoast and AIOSEO ship WordPress plugins; validators exist at llmstxt.org. For most sites a text editor is the right tool.

2. robots.txt — the AI agent roster

The oldest file is now the most consequential, because the major vendors run separate crawlers for separate purposes — and blocking the wrong one costs you visibility without protecting anything.

  • VERIFIED: OpenAI runs three agents with independent gates: GPTBot (training), OAI-SearchBot (the search index behind ChatGPT search), ChatGPT-User (live fetches when a user asks about your page). Blocking GPTBot opts out of training without removing you from search results — and vice versa. Published IP ranges let you verify impostors.
  • VERIFIED: Anthropic split similarly (Feb 2026): ClaudeBot (training), Claude-SearchBot (search), Claude-User (live). Google-Extended is a robots-only token gating AI training — it is not a crawler and blocking it does not touch Googlebot. Applebot-Extended, CCBot (Common Crawl), and Meta-ExternalAgent round out the training gates.
  • COMMUNITY-REPORTED: ByteDance's Bytespider is repeatedly observed ignoring robots.txt despite vendor claims; Cloudflare documented Perplexity fetching through undeclared headless browsers (2024). If opt-out matters to you, robots.txt alone is a request, not a wall — WAF rules are the enforcement.
  • Implement: decide training and search-visibility separately. A measurement-era default: allow the search and live-fetch agents, decide training by business model, and never block what you haven't named (a bare Disallow: / for * takes you out of every AI answer at once).

3. ai.txt

Spawning's machine-readable usage policy — now an IETF draft (draft-car-ai-txt-wellknown-00, June 2026) placing it at /.well-known/ai.txt. Where robots.txt is binary access control, ai.txt expresses nuance: "crawl me, cite me, don't train on me," per-agent rules, license terms, attribution requirements.

  • VERIFIED: the format and its IETF draft status; the EU AI Act requires machine-readable opt-outs be honored, which is the regulatory tailwind behind it.
  • THEORY (the honest part): no major vendor has announced reading it. Compliance today is approximately zero. Publishing one is cheap future-proofing and a legal breadcrumb, not an active control.
  • Implement: a short plain-text policy at both /ai.txt and /.well-known/ai.txt. State the license in one line humans can also read — ours permits training and reproduction with attribution.

4. ai-plugin.json

The OpenAI plugin manifest at /.well-known/ai-plugin.json — a JSON card pointing at your OpenAPI spec so agents can discover and call your API.

  • VERIFIED: the plugin program it served was deprecated in April 2024, replaced by GPT Actions and, increasingly, MCP (Model Context Protocol). Its primary consumer no longer exists.
  • COMMUNITY-REPORTED: some 2026 agent runtimes still probe it as a fallback discovery path.
  • Implement: only worth publishing if you have a real public API — and then it's ten minutes: name, description written for a model, and a pointer to a live OpenAPI URL. Treat it as a signpost to the spec, which is the artifact that matters. If you're building for agents in 2026, an MCP server is the higher-value investment.

5. JSON-LD structured data

Schema.org markup in <script type="application/ld+json"> blocks: Organization, WebSite, Article, FAQPage, Product. The most theory-encrusted item on this list.

  • VERIFIED (and sobering): the best measured study (Ahrefs, difference-in-differences over 1,885 pages that added JSON-LD vs 4,000 controls, Aug 2025–Mar 2026) found no citation lift: −4.6% in Google AI Overviews, +2.2–2.4% noise elsewhere. No AI vendor documents JSON-LD as a citation signal.
  • VERIFIED: 53% of AI-cited pages carry JSON-LD — but that's correlation traveling with better technical quality generally. And Google's July 2026 policy change gates self-serving review/rating markup: publishing stars about yourself now risks eligibility rather than earning it.
  • VERIFIED (the trap): JSON-LD injected by JavaScript is invisible to AI crawlers — tests across ChatGPT, Claude, Perplexity, and Gemini fetches found none of them extracted JS-injected markup. If your schema comes from a tag manager, AI systems have never seen it. Server-render it or don't bother.
  • Implement: emit Organization + WebSite site-wide and Article/FAQPage where true, from the server. Skip anything Google's policy now treats as self-serving. Tooling: spatie/schema-org (PHP), schema-dts (TypeScript), and validator.schema.org.

6. sitemap.xml

  • VERIFIED: AI search crawlers discover through sitemaps referenced in robots.txt; the classic limits hold (50k URLs / 50MB per file, index files above that). Cloudflare's crawl-economics data is the eye-opener: ClaudeBot crawled ~38,000 pages per referred visitor, OAI-SearchBot ~1,100:1, versus Googlebot's ~5:1 — AI crawlers read far more than they send back.
  • THEORY: that an honest lastmod speeds AI citation refresh. Plausible, unmeasured. Keep lastmod truthful anyway — lying to crawlers is how you train them to ignore you.
  • Implement: generate from your router or CMS, reference it in robots.txt, keep it fresh mechanically (ours is a Laravel route that reads the live dataset).

7. The one that outranks the other six: render your content server-side

Every file above is advice about your content. None of it matters if AI systems can't read the content itself.

  • VERIFIED: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Meta's crawler fetch HTML but do not execute JavaScript (Vercel/MERJ crawler telemetry; only Gemini's Google infrastructure and Applebot render JS). A client-rendered SPA reads as an empty shell to most of the AI ecosystem. <noscript> does not save you — AI crawlers ignore it.
  • We ate this one ourselves: our own summarizer judged airanks.net "essentially no substantive content" — because this site is a client-rendered app. The fix (real server-rendered content in the initial HTML, removed for JS-capable browsers) shipped the same day, and the follow-up summary is on our own domain report. The audit tool must pass its own audit.
  • Implement: SSR/SSG for content that must be cited; otherwise a server-rendered floor in the initial HTML. Verify like a crawler would: curl your page and read what comes back with JavaScript off.

Audit any site in one click

The AIR domain report checks every file on this page for any domain — presence, robots.txt AI-agent verdicts, server-rendered JSON-LD types, and live file contents — alongside the domain's AIR score and the Machine's own summary of the site. The build log documents how we optimized our own files, before and after.