Spinning up
Live
+312AI citations this month
Press⌘Kto search
Spinning up
SEO
Definition
A plain-text file at the domain root that tells crawlers which paths they can access. Can target specific User-Agent strings (Googlebot, GPTBot, ClaudeBot). Blocking AI crawlers here removes the site from AI-generated answers.
We run SEO-led engagements every week. A 20-minute call gets you a read on whether robots.txt is the right next move for your stack.