Docs

Content Optimization

llms.txt

How llms.txt Studio generates and optimizes your machine-readable site description.

What is llms.txt

llms.txt is a proposed standard for providing AI-friendly content to large language models. Similar to how robots.txt instructs search engine crawlers, llms.txt provides a structured, markdown-formatted description of your site that AI agents can parse quickly and accurately.

The file lives at the root of your domain (/llms.txt) and gives agents a direct path to understanding your business, products, and capabilities without crawling and interpreting your entire site.

llms.txt vs llms-full.txt

Two files serve different purposes:

  • /llms.txt — A concise summary of your site. Contains your business identity, core capabilities, key product information, and links to deeper resources. Optimized for fast agent consumption.
  • /llms-full.txt — A comprehensive, detailed version containing expanded product descriptions, full pricing tables, complete FAQ content, and detailed feature documentation. Used by agents that need thorough context for complex queries.

Most agent platforms request llms.txt first. If they need more detail to answer a user query, they follow up with llms-full.txt.

What goes in llms.txt

A well-structured llms.txt file includes:

  • Site name and URL — Your canonical business identity.
  • Description — A clear, factual summary of what your business does.
  • Key capabilities — Primary products or services, listed with brief descriptions.
  • Product information — Names, categories, and summary-level details.
  • Pricing — Current pricing tiers or pricing model description.
  • FAQs — Frequently asked questions relevant to purchase or evaluation decisions.
  • Contact — How to reach your team, support channels, and relevant links.

Format example

# Acme Cloud Platform

> Cloud infrastructure platform for deploying, scaling, and monitoring web applications.

## Products

- [Acme Compute](https://acme.dev/products/compute): Serverless compute instances with auto-scaling. Starting at $0.01/vCPU-hour.
- [Acme Storage](https://acme.dev/products/storage): Object storage with 99.999% durability. Starting at $0.023/GB-month.
- [Acme Monitor](https://acme.dev/products/monitor): Real-time application monitoring with alerting. Free tier available.

## Key Facts

- 99.99% uptime SLA
- SOC 2 Type II certified
- 12 global regions
- Free tier available for all products

## Links

- [Documentation](https://docs.acme.dev)
- [Pricing](https://acme.dev/pricing)
- [Status Page](https://status.acme.dev)
- [Contact Sales](https://acme.dev/contact)

llms.txt Studio

llms.txt Studio is the dashboard tool within Inception Agents that manages your llms.txt and llms-full.txt files. It provides a visual editor, version history, and performance analytics in a single interface.

Auto-generation

After you complete your integration (server-side middleware or DNS proxy), Inception Agents crawls your site and automatically generates both llms.txt and llms-full.txt. The auto-generation process:

  1. Crawls all accessible pages on your domain.
  2. Extracts product information, pricing, company details, and FAQ content.
  3. Structures the content into the standard llms.txt markdown format.
  4. Publishes both files and begins serving them to agent traffic.

No manual setup is required. The initial version is ready within minutes of integration.

Manual editing

You can edit and customize your generated files directly in the Studio:

  • Reorder sections to prioritize specific content.
  • Add custom sections not detected during the crawl.
  • Edit descriptions, adjust wording, or add context.
  • Pin specific content blocks to prevent automatic changes.

All edits are versioned. You can revert to any previous version at any time.

Optimization

The learning engine continuously tests different versions of your llms.txt content:

  • Headline and description variations are tested against agent traffic.
  • Section ordering is adjusted based on which layouts correlate with higher recommendation rates.
  • Content density is tuned per-platform — some agents prefer concise summaries, others perform better with more detail.
  • Performance data feeds back into the next optimization cycle automatically.

How agents use llms.txt

When an AI agent (ChatGPT, Perplexity, Claude, Gemini, and others) needs to understand your site, it requests /llms.txt as a first step. This gives the agent:

  • A structured overview without parsing HTML.
  • Accurate, up-to-date product and pricing information.
  • Direct links to relevant pages for deeper context.
  • A format specifically designed for LLM comprehension.

Agents that support llms.txt include ChatGPT (via GPTBot), Perplexity (via PerplexityBot), and a growing number of agent platforms that recognize the standard.

Verification

Verify your llms.txt is being served correctly:

curl -s https://yourdomain.com/llms.txt

You should see your markdown-formatted site description. To verify the full version:

curl -s https://yourdomain.com/llms-full.txt

Both files should return 200 OK with Content-Type: text/markdown or text/plain headers. If you receive a 404, check that your integration is active in the dashboard.