Spinning up
Live
+312AI citations this month
Press⌘Kto search
Spinning up
GEO / AI Search
Definition
OpenAI's web crawler used to populate ChatGPT's training data and live citations. Identified by the User-Agent string GPTBot/1.0. Sites that block it via robots.txt are excluded from ChatGPT-generated answers.
Related terms
ClaudeBot
Anthropic's web crawler for Claude. Identified by User-Agent ClaudeBot/1.0 and anthropic-ai. Allowing it in robots.txt is required to be cited in Claude-generated answers.
PerplexityBot
Perplexity AI's web crawler. Two variants: PerplexityBot (training) and Perplexity-User (live citation lookup). Citation rates correlate strongly with named-author bylines, visible publish dates, and high domain authority.
robots.txt
A plain-text file at the domain root that tells crawlers which paths they can access. Can target specific User-Agent strings (Googlebot, GPTBot, ClaudeBot). Blocking AI crawlers here removes the site from AI-generated answers.
Same category
We run GEO / AI Search-led engagements every week. A 20-minute call gets you a read on whether gptbot is the right next move for your stack.