Spinning up
Live
+312AI citations this month
Press⌘Kto search
Spinning up
SEO
Definition
The number of URLs a search engine will crawl on a site in a given window. Constrained by server capacity and the engine's allocation. Large sites optimize by ensuring high-priority URLs are easy to discover and not blocked by thin or duplicate pages.
Related terms
Indexing
The process by which a search engine adds a page to its searchable database after crawling. A page is only available to rank once indexed. Google Search Console reports indexing status under Coverage.
robots.txt
A plain-text file at the domain root that tells crawlers which paths they can access. Can target specific User-Agent strings (Googlebot, GPTBot, ClaudeBot). Blocking AI crawlers here removes the site from AI-generated answers.
We run SEO-led engagements every week. A 20-minute call gets you a read on whether crawl budget is the right next move for your stack.