What is crawl budget?

Crawl budget describes how much a search engine can and wants to crawl your site. It matters most when many URLs compete for crawler attention.

Definition

Crawl budget is the amount of crawling a search engine allocates to your site. For Google, it depends on crawl capacity, including how much traffic your server can handle, and crawl demand, including how often content needs revisiting. It is not a fixed allowance of pages you can submit each day.

When it matters

Large sites, rapidly changing inventories and sites generating many duplicate URLs are more likely to need crawl management. For a small site, missing pages are often better investigated through discovery, content quality, noindex or canonical choices before assuming the budget is exhausted.

Reduce unnecessary URL variations, fix persistent server errors and keep internal links useful. Use an XML sitemap to identify important canonical URLs. Blocking crawling with robots.txt does not by itself remove an indexed URL from search.

Where to investigate

Search Console's Crawl Stats report shows Google's crawl activity and host health. Server logs can reveal which URLs crawlers request. URL Inspection gives last-crawl information for individual pages, while Page indexing reports help distinguish discovery and indexing issues.

In Bing Webmaster Tools, review crawl errors and affected URLs. A page that was crawled but not indexed has passed a different stage from one the engine has only discovered. Neither status alone proves a crawl-budget problem.

Ask mowteng for evidence

mowteng can inspect Google's index status and last crawl for URLs, and retrieve Bing crawl issues when Bing is connected. These checks support investigation; they do not measure your total crawl budget. See the technical SEO prompts for related questions.

Ask your assistant

Inspect these important URLs for Google index status and last crawl, then check Bing crawl issues. Separate observed crawling errors from indexing exclusions and explain what still needs investigation.

Connect Search Console, Bing and Yandex to your AI assistant with one URL. 7-day free trial.

Connect mowteng