Definition
Crawl budget is the number of URLs a search engine wants to and can fetch on your site within a given period. Google derives it from two parts: crawl capacity (how many requests your server can handle without slowing down) and crawl demand (how interesting and how frequently changed your pages are). Together they determine how many pages Googlebot visits per day and which ones.
For small sites, crawl budget is rarely a problem: a site with a thousand pages gets crawled completely on its own. It becomes relevant from several tens of thousands of URLs, or earlier if your site generates many URLs that add nothing: filter combinations in a webshop, internal search results, calendar pages, session IDs and sorting parameters. Every minute Googlebot spends on those does not go to the pages that actually matter.
The symptoms of a shortage show up in Search Console. Under 'Pages' you then see many URLs as 'Discovered - currently not indexed' or 'Crawled - currently not indexed'; new pages take weeks to enter the index; and the Crawl stats report shows a large share of 404s, redirects or parameter URLs. Server logs show it even more sharply: which URLs Googlebot actually requests and how often.
You improve crawl budget by removing the noise and making the important pages findable. Block worthless URL patterns in robots.txt, give facets and sort orders a canonical or noindex, remove or redirect broken links, ensure a fast server response (TTFB) and keep the XML sitemap clean with only indexable URLs. A good internal link structure then makes sure the budget reaches the pages that matter.
AI crawlers such as GPTBot, ClaudeBot and PerplexityBot have their own budget and are often less patient than Googlebot: they fetch fewer pages and do not render JavaScript. Traze tracks which published pages have been fetched by which crawlers and submits new pages for indexing immediately, so they do not linger in the queue.