GEO / AI Search

Common Crawl

Definition

What is Common Crawl?

Open-source web archive that captures billions of pages monthly and ships them as a public corpus. Used to bootstrap most foundation model training datasets. The CCBot user-agent identifies its crawler.

Related terms

Cross-linked concepts

GPTBot

OpenAI's web crawler used to populate ChatGPT's training data and live citations. Identified by the User-Agent string GPTBot/1.0. Sites that block it via robots.txt are excluded from ChatGPT-generated answers.

ClaudeBot

Anthropic's web crawler for Claude. Identified by User-Agent ClaudeBot/1.0 and anthropic-ai. Allowing it in robots.txt is required to be cited in Claude-generated answers.

Same category

More GEO / AI Search terms

Generative Engine OptimizationGEO
Answer Engine OptimizationAEO
Answer Capsule
Atomic Fact
Entity Authority
llms.txt

Browse all 121 glossary terms →

Want to put Common Crawl to work?

We run GEO / AI Search-led engagements every week. A 20-minute call gets you a read on whether common crawl is the right next move for your stack.

Book a strategy call →

Spinning up