SEO
Crawl Budget
Definition
What is Crawl Budget?
The number of URLs a search engine will crawl on a site in a given window. Constrained by server capacity and the engine's allocation. Large sites optimize by ensuring high-priority URLs are easy to discover and not blocked by thin or duplicate pages.
Related terms
Cross-linked concepts
- IndexingThe process by which a search engine adds a page to its searchable database after crawling. A page is only available to rank once indexed. Google Search Console reports indexing status under Coverage.
- robots.txtA plain-text file at the domain root that tells crawlers which paths they can access. Can target specific User-Agent strings (Googlebot, GPTBot, ClaudeBot). Blocking AI crawlers here removes the site from AI-generated answers.
Same category
More SEO terms
Want to put Crawl Budget to work?
We run SEO-led engagements every week. A 20-minute call gets you a read on whether crawl budget is the right next move for your stack.