SEO
robots.txt
Definition
What is robots.txt?
A plain-text file at the domain root that tells crawlers which paths they can access. Can target specific User-Agent strings (Googlebot, GPTBot, ClaudeBot). Blocking AI crawlers here removes the site from AI-generated answers.
Same category
More SEO terms
Want to put robots.txt to work?
We run SEO-led engagements every week. A 20-minute call gets you a read on whether robots.txt is the right next move for your stack.