Agent-native
A first-class CLI and MCP server. The same AI agent that built your page can crawl it, read the issues, and ship the fixes — no human in the loop.
#4 on Hacker News · the most-upvoted open-source SEO crawler
Technical SEO that
Technical SEO that
works with your agents.
Free, open-source crawler for SEO and AI-search readiness. 46 checks, plain-English fixes, a CLI and an MCP server. Runs locally. Costs nothing.
$
npm i -g crawlie Rust · MIT · macOS / Linux / Windows · CLI + MCP · 46 checks
The problem
You can vibe-code a landing page in an afternoon. But is it built to be found?
AI made shipping a marketing site trivial — gorgeous, live, fast. The catch: most of what gets generated was never built for search. Missing metadata, broken links, no structured data, nothing for AI search to read. You don’t find out until the traffic never shows up — and the tools that catch it cost real money and can’t talk to your agents. crawlie fixes that.
Why crawlie
Everything you need to ship findable sites
Built for the way sites get made now — fast, AI-assisted, and audited by agents.
GEO / AI-search readiness
A dedicated GEO score for being cited by ChatGPT, Perplexity, and Google AI Overviews: structured data, semantic HTML, answer-ready content, authorship.
Plain-English fixes
Every finding tells you why it matters, how to fix it, and what happens if you ignore it — not just a cryptic error code.
Local-first & fast
A tiny async Rust binary that runs on your machine. No accounts, no cloud, no per-crawl billing. Your data never leaves your laptop.
46 checks & counting
Broken links, redirects, 4xx/5xx, titles & meta, canonicals, robots, hreflang, structured data, mixed content, thin content, and more.
Free & open source
MIT licensed. Read it, fork it, extend it. SEO shouldn’t be locked behind a $259/year license.
Agent-native
Hand the audit to your agent
crawlie ships an MCP server, so any MCP-compatible agent — Claude, Cursor, your own — can crawl a site, read structured issues, and propose exact fixes. Wire it up once:
$
claude mcp add crawlie crawlie-mcp Or install the one-step Claude Code plugin — it bundles the MCP server and ready-made audit skills.
You > Audit crawlie.dev and give me the top fixes for AI search. crawlie · crawl_site → 28 pages, Health 86, GEO 62 crawlie · explain_issue → geo-no-structured-data Top fixes (GEO): 1 Add Article + FAQ schema to /blog/* 12 pages 2 Add an llms.txt at the root site-wide 3 Author bylines on guides (E-E-A-T) 7 pages 4 Fix 2 redirect chains on /docs 2 pages
Two scores, one crawl
Health for search engines. GEO for AI.
86Health
Health score
Classic technical SEO: links, status codes, metadata, canonicals, robots, performance, security, mobile & international signals.
62GEO
GEO score
How citable you are by AI search: structured data, semantic HTML, answer-ready blocks, dated content, authorship and E-E-A-T.
How it compares
Screaming Frog energy, none of the price tag
| crawlie | Screaming Frog | Sitebulb | |
|---|---|---|---|
| Price | Free & open-source | £259/yr to unlock | from £13.50/mo |
| Engine | Rust, async, tiny binary | Java (JVM) | .NET |
| CLI with JSON output | ✓ | partial | — |
| MCP server (agent-native) | ✓ | — | — |
| GEO / AI-search audit | ✓ | — | — |
| “Why it matters” built in | every issue | — | partial |
| Shareable HTML report | ✓ | paid | ✓ |
| Source you can read & extend | ✓ | — | — |
Audit your site in 30 seconds — free.
One command. No account. No cloud. No catch.
$
npx -y crawlie crawl https://your-site.com