Agentic Engine Optimisation (AEO): How to Make Your Site Readable by AI Agents

What is Agentic Engine Optimisation?

Humans browse. Search engines crawl. AI agents consume. The difference matters more than it sounds.

When an AI coding agent reads your documentation, it makes a single HTTP request, strips the HTML, counts the tokens, and decides in milliseconds whether the content is useful. It doesn't scroll. It doesn't navigate. It doesn't run JavaScript. If your content is locked behind a client-side render, buried in 50,000 tokens of boilerplate, or missing a structured index that tells agents where everything is — your documentation might as well not exist from the agent's perspective.

AEO is the practice of fixing those gaps. It's about making your site discoverable, readable, and usable by AI systems while keeping it just as good for humans.

The agentic-seo tool by Addy Osmani

Addy Osmani — engineering lead at Google Chrome and author of Essential JavaScript Design Patterns — created agentic-seo: a Node.js CLI auditing tool that evaluates documentation sites for AEO readiness. It runs ten checks across five categories, scores your site out of 100 points, and produces a grade from A to F with specific fixes for every failure.

You run it with a single command from any directory:

npx agentic-seo              # Audit current directory
npx agentic-seo ./docs       # Audit specific directory
npx agentic-seo --url https://docs.example.com  # Audit live URL

It auto-detects your framework — Next.js, Docusaurus, Astro, Hugo, VitePress, and more — locates the build output, and runs all ten checks against your actual content.

The five categories and ten checks

The audit is structured around five questions an AI agent implicitly asks when it encounters your site:

1. Discovery (25 pts) — Can agents find your documentation?

Three checks here: robots.txt (are AI crawlers explicitly allowed?), llms.txt (does a structured navigation index exist?), and AGENTS.md (is there a project context file with structure and conventions?).

llms.txt is the most important new primitive. It's a plain text file at your domain root — like a sitemap, but designed for LLMs rather than search crawlers. A well-structured llms.txt lists your key pages with descriptions and token counts, letting an agent quickly decide which pages are worth consuming before making any further requests. The spec is at llmstxt.org.

AGENTS.md is the AI equivalent of a README for coding agents. If you work with Claude Code, you probably already have a CLAUDE.md — AGENTS.md follows the same idea but for a wider audience of agents. It contains project structure, development commands, coding conventions, and anything an agent needs to orient itself before touching code.

2. Content Structure (25 pts) — Is content machine-readable?

Two checks: content structure (heading hierarchy, semantic HTML, code examples with language tags, data tables) and markdown availability (can agents access your raw Markdown source rather than parsing HTML?).

The markdown check is subtle but important. If your site only serves HTML, an agent has to parse through navigation, headers, footers, and scripts to get to the actual content. If you serve Markdown at a predictable URL pattern — or link to it in llms.txt — the agent can consume raw text directly with near-zero noise.

3. Token Economics (25 pts) — Does content fit agent context windows?

Two checks: token budget (how many tokens is each page?) and meta tags (do AI-friendly meta tags provide accurate summaries?).

Context windows are not infinite. Even large models have practical limits on how much they can process in one go, and oversized pages waste tokens that could be used for actual task context. The tool flags pages above 30,000 tokens (and anything above 50,000 as critical) and recommends splitting them.

4. Capability Signalling (15 pts) — Do agents know what your APIs do?

Two checks: skill.md (does a capability description file exist?) and agent-permissions.json (are agent access rules and rate limits defined?).

skill.md is a YAML frontmatter + Markdown file that describes what your API or service can do — inputs, outputs, constraints, examples. Think of it as a machine-readable capability card. agent-permissions.json defines which endpoints agents can access, at what rate, and with what authentication.

5. UX Bridge (10 pts) — Does UX support human + agent workflows?

One check: copy-for-AI buttons. Does your documentation include a visible way to copy page content in a clean format? Tools like Context7 and Cursor already expect this — raw view links, copy-to-clipboard buttons, or dedicated "Copy for AI" actions that strip navigation and serve clean Markdown.

Grading scale

Running the CLI

Installation

No install required — run it directly with npx. If you audit frequently, a global install is faster:

# No install — always pulls latest version
npx agentic-seo

# Global install (run as just: agentic-seo)
npm install -g agentic-seo

Requires Node.js 18 or later. No API key or account needed — all checks are structural and run entirely locally or against public URLs.

Auditing a local directory

The most common use case. Point the CLI at your project root or build output and it auto-detects the framework, locates the built HTML files, and runs all ten checks:

# Audit the current directory (auto-detects framework)
npx agentic-seo

# Audit a specific project directory
npx agentic-seo ./my-docs-site

# Explicitly specify the build output directory
npx agentic-seo --output-dir ./my-project/out

Auto-detection covers eleven frameworks out of the box: Next.js, Docusaurus, Eleventy, Astro, Hugo, Jekyll, Gatsby, VitePress, MkDocs, Sphinx, and Vite. If your framework isn't on the list, use --output-dir to point directly at the folder containing your built HTML.

Auditing a live URL

For sites you don't have locally — a production URL, a competitor's docs, or any public domain — use the --url flag. The tool fetches pages over HTTP and runs the checks that work against live content:

npx agentic-seo --url https://docs.example.com

URL mode is more accurate than the online AEO Checker because the CLI evaluates the fully rendered page, not just the raw HTML response. Copy-for-AI buttons added by JavaScript and content that only appears after the page loads will both be detected correctly.

Auditing with a local server

If you want the accuracy of URL-based checks against a local build — without deploying — use --serve. The CLI spins up a static file server on a random port, runs the full audit over HTTP, then shuts it down:

npx agentic-seo --serve ./build

Running specific checks

If you only want to audit one or two signals — useful when iterating on a specific fix — pass a comma-separated list of check IDs:

# Run only the discovery checks
npx agentic-seo --checks robots-txt,llms-txt,agents-md

# Check token budget across all pages
npx agentic-seo --checks token-budget

# Valid check IDs:
# robots-txt, llms-txt, agents-md,
# content-structure, markdown-availability,
# token-budget, meta-tags,
# skill-md, agent-permissions,
# copy-for-ai

Output modes

By default the CLI prints a coloured human-readable report. Two flags change the output format:

# JSON output — pipe into other tools or save to file
npx agentic-seo --json
npx agentic-seo --json > aeo-report.json

# Score only — compact one-liner for quick checks
npx agentic-seo score

# Verbose — show all findings including info-level messages
npx agentic-seo --verbose

All CLI flags

Scaffolding missing files with init

Running npx agentic-seo init creates the AEO files you are missing — llms.txt, AGENTS.md, skill.md, and agent-permissions.json — with sensible templates pre-populated from your detected project structure. Fill them in, then re-run the audit.

npx agentic-seo init

This is the fastest path from zero to a baseline score: init scaffolds the files, you add real content, the next audit shows the delta.

CI/CD integration

The --threshold flag exits with code 1 when the score falls below the minimum, making it a drop-in CI gate. Combine with --json to store the full report as a build artefact:

# Fail the build if AEO score drops below 60%
npx agentic-seo --threshold 60

# Store the full report as a JSON artefact
npx agentic-seo --json > aeo-report.json

# Score-only check — no noise in CI logs
npx agentic-seo score

Persist your configuration in .aeorc.json at the project root so the threshold and custom check settings apply on every run without repeating flags:

// .aeorc.json
{
  "outputDir": "_site",
  "threshold": 70,
  "checks": {
    "token-budget": { "maxTokensPerPage": 20000 },
    "robots-txt": { "requiredAgents": ["ClaudeBot", "GPTBot"] }
  },
  "ignore": ["**/node_modules/**"]
}

You can also put the same config under an "aeo" key in your existing package.json instead of a separate file.

Programmatic API

For more advanced integrations, agentic-seo exposes a clean programmatic API:

import { audit, auditWithServer } from 'agentic-seo';

// Audit a built directory
const report = await audit('./docs/build');
console.log(report.grade);       // 'B'
console.log(report.percentage);  // 78
console.log(report.findings.errors); // Array of errors with fixes

// Audit a live URL
const liveReport = await auditWithServer('./docs/build', { port: 3001 });

Each check returns a structured result with id, name, category, score, maxScore, status, and findings — making it easy to build custom dashboards or integrate into larger tooling.

Quick start: getting from F to B

Most documentation sites fail on the same three things. Here's the fastest path to a meaningful score improvement:

1. Create llms.txt. Add a file at your domain root that lists your key pages with one-line descriptions. Even a minimal version scores you 15+ points. See llmstxt.org for the format.
2. Create AGENTS.md. Write a 200–400 word overview of your project: what it is, directory structure, how to build and run it, and coding conventions. This is immediately useful to any AI coding agent that touches your repo.
3. Fix your robots.txt. Add explicit User-agent blocks for GPTBot, ClaudeBot, and PerplexityBot. Explicit rules — even if they just say Allow: / — score higher than relying on the wildcard rule.

Those three steps alone typically move a site from an F to a C or B. The remaining checks — token budget, skill.md, copy-for-AI UX — are refinements for sites that are already baseline-ready.

Online AEO checker

If you want to check a public URL without running the CLI, use the AEO Checker tool on this site. It runs seven checks against any public URL: llms.txt, AGENTS.md / CLAUDE.md, robots.txt AI rules, skill.md, AI meta tags, agent-permissions.json, and Copy for AI UX.

For the full ten-check audit — including token budget analysis, content structure, skill.md, and copy-for-AI UX — run the CLI locally with npx agentic-seo.

The broader picture

AEO is not a replacement for SEO. It's an extension of the same underlying principle: make your content legible to the systems that distribute it. For the past twenty years, that meant search crawlers. Today it also means AI agents.

The sites that invest in AEO now will have a compounding advantage as AI-driven workflows become the default for developers. If an AI agent can't understand your documentation, it won't recommend your library, your API, or your product. That's not a search ranking problem — it's a content visibility problem.

Addy Osmani's agentic-seo tool is the clearest framework available right now for diagnosing and fixing that problem. It's open source, actively maintained, and takes under two minutes to run. There's no good reason not to try it.

Full credit to Addy Osmani for building and open-sourcing this tool. The agentic-seo repository is the canonical reference for AEO tooling.