Making Your Website Crawlable and Citable by LLMs Using llms.txt

Why LLMs Struggle with Modern Websites

Google's crawler is patient and sophisticated. It renders JavaScript, follows redirects, resolves dynamic imports, and scores pages on Core Web Vitals. LLM crawlers are not like that. They prefer fast, clean HTTP responses with real text content. A Next.js or Nuxt app that client-renders its main content — or that buries page text inside deeply nested component trees — is going to get a shallow or empty parse.

Even with SSR, the problem isn't fully solved. AI crawlers are looking for structured, readable context: what is this site about, what are its main sections, what does each page say. An HTML page with global nav, cookie banner, footer, and then three paragraphs of actual content is noisy. A markdown file with the same three paragraphs is perfect.

The other problem is discoverability. LLMs building their training data or doing live web search need a way to discover what's on your site without crawling every URL. That's exactly what llms.txt is for.

The Three Artifacts You Need

The llms.txt spec calls for three deliverables. Each solves a different part of the discoverability problem:

1. /llms.txt — The Site Index for AI Crawlers

Think of this as robots.txt for LLMs — except instead of blocking crawlers, it's actively helping them. The file groups your pages by content type and links to the .md version of each page. A typical section looks like this:

The format is plain markdown. Each link points to the .md route of the page, not the HTML version. The idea is that an AI agent that wants to understand your site can read this file in seconds and then follow links to get clean content for any page it cares about.

2. /llms-full.txt — The Bulk Corpus File

Some AI agents — particularly those doing bulk ingestion or building RAG indexes — prefer to download a single document containing your full site content. llms-full.txt is that document. It's intentionally curated, not a dump of everything:

• Homepage content — who you are, what you do
• Core service pages — the full description of each offering
• Selected blog posts — your most useful evergreen content
• Portfolio or case study highlights

I don't include location pages (too repetitive across a city portfolio), thin pages, or anything that's more template than content. The goal is to give an LLM a dense, accurate, high-signal picture of the site in one pass.

3. Page-Level .md Routes

This is the part that makes the whole system work as a network rather than just a static document. Every canonical page on your site gets a .md equivalent at the same path:

• /voegwerk → /voegwerk.md
• /nieuws/some-post-slug → /nieuws/some-post-slug.md
• /locaties/amsterdam → /locaties/amsterdam.md

Each markdown file starts with metadata: canonical HTML URL, page type, last-modified date. Then it's clean prose content — no nav, no footer, no cookie consent noise. Internal links inside the markdown point to the .md version of linked pages, not the HTML. This means an AI agent can follow a link from one page's markdown into another's, crawling your site's content graph without touching HTML at all.

The Next.js Approach: Build-Time Generator

For Next.js sites on Cloudflare Pages (free plan), there's no runtime — no Cloudflare Worker, no API routes that run server-side in production. Everything has to be static. That turns out to be a feature, not a constraint: generating all LLM artifacts at build time and writing them to public/ means they're served as static files from the CDN edge globally, with zero cost and zero latency.

Architecture: Route-First, Manifest-Driven

The key insight that makes the generator maintainable is building around a canonical route manifest. Instead of scanning the file system or parsing built HTML, you maintain one source of truth that every other output consumes:

Both the sitemap generator and the LLM generator consume this same manifest. They can never drift apart — if a page exists in the sitemap, it has an LLM artifact, and vice versa. Adding a new page means one entry in the manifest, and both outputs are updated on the next build.

Folder Structure

The generator lives in two places: core logic in lib/llms/, and the CLI runner in scripts/llms/:

Content Extractors: Pull from Data, Not HTML

Each extractor knows how to pull content for its content type from your existing structured data files — not from scanning built HTML. If your service pages are driven by a data/services.ts file, the services extractor imports that directly and renders clean markdown. This means the generated content is always consistent with what the site actually says, and there's no dependency on a running dev server or built output:

Stale File Cleanup

One problem that bites you quickly: if you remove a route from the manifest, the old .md file still sits in public/ and gets deployed. You need a cleanup mechanism that deletes only files this generator created, without touching other assets in public/.

The solution is a small tracking file: public/.llms-generated.json. On each run, the generator reads the list of previously-generated file paths, deletes those files, then writes fresh ones and updates the tracking file. Unrelated assets like images and fonts are never touched.

Build Integration

The generator runs before next build. On Cloudflare Pages, prebuild hooks do not run — Cloudflare calls your build command directly. So the generation step has to be baked into the build script itself, not a pre-hook:

The verify script runs post-generation and checks that the number of .md files matches the number of manifest entries, that all internal links resolve within the manifest, and that llms.txt and llms-full.txt were both written. Running it in CI catches issues before they reach production.

What to Include and What to Exclude

The manifest should only contain canonical, indexable pages. A few hard rules I apply:

• Include: all pages you'd include in a sitemap. Service pages, blog posts, location pages, core pages (about, contact, homepage).
• Exclude: noindex pages, legacy redirect routes, admin or API routes, paginated duplicates (page 2, page 3 of a category).
• City + service combos: if your site has /locaties/amsterdam/voegwerk as a real HTML page, include it. If it doesn't exist as a real page, don't invent a .md file for it. LLMs trust your URLs — don't create a discrepancy between what the markdown implies exists and what actually loads in a browser.

Verified Numbers from Production

I've shipped this generator on two client sites. Here's what the output looks like at scale:

• GevelPRO (gevelpro.nl): 138 canonical routes in the manifest, 140 generated files (138 .md pages + llms.txt + llms-full.txt, minus the tracking file)
• Overbeek (voegersbedrijfoverbeek.nl): 92 canonical routes, 94 generated artifacts

Build time overhead is under 4 seconds for either site. The generated files are committed to public/ or generated fresh on each Cloudflare Pages build — either approach works. I prefer generating fresh each build so the last-modified dates stay accurate.

The Nuxt Approach: nuxt-ai-ready Module

For Nuxt projects, there's a much simpler path. The nuxt-ai-ready module (version ^0.10.10) handles the full llms.txt pipeline automatically — including markdown route generation, IndexNow submission, and an MCP server for AI agents with full-text search over your content.

Installation

Configuration

The module requires a Cloudflare D1 database for persistent page indexing and FTS5 full-text search. Wire it up in your nuxt.config.ts and wrangler.toml:

The D1 database is what enables persistent indexing and search. Without it, the module can still generate static files, but the MCP server and FTS5 search won't work. For most client sites on Cloudflare, creating a D1 database is a one-minute operation in the Cloudflare dashboard.

One Gotcha: Prerender Errors for .md Routes

When Nitro prerenders your Nuxt site, it will attempt to crawl the .md routes that the module generates — and may throw prerender errors if those aren't explicitly handled. Add them to your Nitro ignore patterns if you hit this:

The module is still relatively new and the prerender integration may improve in future versions. Check the module's GitHub repo for the latest guidance.

Why This Is Worth Doing Now

I want to be honest about where the llms.txt standard is right now: it's gaining traction, it's not universal. Major AI systems don't all formally support it yet — there's no RFC, no W3C endorsement, no guarantee that Perplexity's crawler specifically looks for /llms.txt the way Googlebot looks for /robots.txt. The spec is maintained independently by the people who proposed it, and adoption is organic.

That said, the underlying logic is sound regardless of standard adoption. Clean markdown files served as static assets are trivially cheap to generate and maintain. An LLM that hits your /voegwerk.md URL — whether following an llms.txt link or just discovering the URL through other means — is going to get a dramatically better read of your page than it would from the full HTML. The format is self-reinforcing: even if llms.txt itself is ignored, the .md routes are independently useful.

The AI citation channel is real. I've seen Perplexity cite client sites with specific quotes pulled from service pages. Those quotes are accurate when the markdown is clean and inaccurate when it's being parsed from noisy HTML. The investment in clean content surfaces pays off.

Sites that give LLMs clean, structured, markdown-formatted content get cited more accurately — and eventually, more often.

— The working hypothesis behind this whole system

How to Start

If you're on Nuxt: install nuxt-ai-ready, create a D1 database, wire up the config. You can be done in an hour.

If you're on Next.js with Cloudflare Pages: start with the manifest. Sit down and list every canonical URL on your site with its content type, title, and last-modified date. That manifest is the hard part — once it exists, the generator can be built around it incrementally. Start with just llms.txt and a handful of core .md pages, then expand from there.

Either way: run your verify script, deploy once, and check that https://yourdomain.com/llms.txt and https://yourdomain.com/llms-full.txt are publicly accessible and return sensible content. That's the whole thing. It's a two-hour project for most sites, and the static files cost nothing to serve.

The window where this is differentiating is now, while most sites haven't done it. In two years, having llms.txt will be table stakes — the same way having a sitemap is today. Might as well be early.