Crawl budget

← Back to glossary · Category: Technical

Crawl budget is a loose way of describing how much attention a search engine gives to crawling a site in a given period. Crawling is how search engines discover pages: automated programs visit pages, follow links, and record what they find. No search engine can crawl every page on the web constantly, so each site receives a share of crawling attention, shaped by what the engine can handle and what it wants to cover. That share is what people mean by crawl budget.

It is important to say at the outset that crawl budget is rarely the first problem for a small blog. A site with a modest number of pages, where new articles are crawled promptly after publication, does not need advanced crawl management. The concept matters most for very large or fast-changing sites, and for any site whose structure wastes crawling attention on pages that do not matter. Understanding it helps a small business avoid creating that waste, even if it never needs to optimise for it directly.

What crawl budget is, and what it is not

Google defines a site's crawl budget as the set of URLs it can and wants to crawl. Those two halves matter. "Can" is about capacity: how much crawling a site's servers can support without being overloaded. "Wants" is about demand: how much the engine's systems want to crawl the site, based on its size, update frequency, page quality, and relevance. A site can receive less crawling even when its servers have spare capacity, if demand for its content is low.

Crawl budget is not a fixed allowance handed to every site, and it is not a number a site owner can look up and spend. It shifts with server health, with how often content changes, and with the quality of what is found. It is also defined per hostname, so a main domain and a subdomain are treated as separate sites with separate budgets. That detail matters when different sections of a site seem to receive very different treatment.

It is not the same as indexing, either. Crawling is the visit; indexing is the decision to store and consider a page for search results. A page can be crawled without being indexed, and a page can be known about without being crawled recently. Crawl budget concerns the visiting stage, and its practical effect is about discovery: whether new and updated pages get visited promptly, or whether crawler attention is spent elsewhere.

Finally, crawl budget is not a ranking factor to chase. A site does not rank better because it "spent" its budget well; it benefits because its important pages are discovered and kept fresh. The goal is efficient discovery, not a higher score.

How crawl budget works

Crawling starts from known pages and follows links, including links listed in a sitemap. A sitemap is a file listing the URLs a site cares about, and search engines read it regularly. Keeping it up to date, and including updated pages with their last-modified information, is the simplest way to point crawling attention at the pages that matter most.

Capacity moves with server health. If a site responds consistently and quickly, the crawling limit can rise over time; if it slows down, returns server errors, or signals rate limiting, the limit falls and less gets crawled. This is a practical reason to care about site speed and reliability beyond user experience: a struggling server quietly reduces how much of a site gets visited.

Demand moves with inventory, popularity, and staleness. Without guidance, a crawler tries to cover most of the URLs it knows about on a site. If many of those URLs are duplicates, parameter variations of the same page, or removed pages, a large share of crawling attention goes to URLs that add nothing. That is the waste crawl-budget advice is really about: not a shortage of attention in the abstract, but attention spent on the wrong URLs. Consolidating duplicate content, using a canonical URL to name the preferred version of a page, and returning proper not-found codes for permanently removed pages all reduce that waste.

Redirect chains also cost attention, because each hop in a chain is another request before the real page is reached. Long chains, and chains that loop or end nowhere, are a common and fixable drain. The redirect checker can help trace what a URL actually leads to, and the broken link checker finds links that lead nowhere at all.

Why crawl budget matters for a blog post and for search

For a blog post, crawl budget matters at the moment of publication and update. A new article needs to be discovered before it can be found by readers through search, and discovery happens through crawling: through links from other pages, and through the sitemap. A blog whose sitemap is stale, whose internal links are broken, or whose structure buries new posts several clicks from the homepage makes discovery slower than it needs to be, even on a small site.

For search, the honest scope is narrow for most small businesses. Google's own guidance says that if a site's pages are crawled the same day they are published, keeping the sitemap up to date and checking the indexing report regularly is adequate, and the advanced guide is written mainly for very large or rapidly changing sites. A small blog should therefore treat crawl budget as hygiene, not as a project: clean structure, current sitemap, working links, one address per page. That hygiene also supports everything else, because a site that is easy to crawl is easy to understand.

There is a related benefit for readers. The same problems that waste crawling attention, duplicates, dead ends, and tangled redirects, also confuse human visitors. Fixing them improves the site for both audiences at once, which is why crawl hygiene is worth doing even when crawl budget itself is not a constraint.

Common mistakes

The most common mistake is worrying about crawl budget before the basics are in place. A small site with broken internal links and a stale sitemap does not have a crawl-budget problem; it has a maintenance problem. Fix the maintenance first, and the crawling usually follows.

The second mistake is letting duplicate URLs multiply. Parameter variations, print versions, tag pages, and session addresses can create many URLs for the same content. Each one is another address a crawler may visit, and another version a reader may land on. Consolidating duplicates and naming one canonical version is the fix, and it helps readers as much as crawlers.

The third mistake is blocking important pages by accident. A miswritten robots rule can hide a whole section from crawlers, including pages the business most wants found. Robots rules deserve the same care as any other site change, and a check after editing them is worth the minute it takes.

The fourth mistake is ignoring server health. Slow responses and frequent errors do not just frustrate readers; they reduce how much crawling a site receives. Speed and reliability work is therefore crawl work too, even if it is never labelled that way.

The fifth mistake is building long redirect chains during site changes. Each redesign, migration, or restructuring adds hops if old redirects are never cleaned up. A periodic redirect audit keeps the path from any old address to its current page short and direct.

How to apply it step by step to a small-business blog

Start by checking whether you have a problem at all. If new posts are crawled promptly after publication, you almost certainly do not need advanced crawl management. Keep the sitemap current, and check the indexing report in Search Console from time to time. That is the whole routine for most small blogs.

Next, clean the structure. Make sure every important page is reachable through internal linking from the homepage or a main section, and that new posts are linked from the homepage or an index page when published. Remove or fix broken links, and shorten redirect chains. The guide on internal linking strategy for blogs shows how a clear linking structure serves both readers and crawlers.

Then, consolidate duplicates. Where the same content appears under several addresses, choose one preferred address, redirect the others to it where possible, and name the preferred version as canonical. Keep the sitemap limited to pages you actually want crawled, rather than every address the site has ever generated.

After that, check server health basics: pages loading reliably, no frequent server errors, and no accidental blocks on important sections. The guide on measuring blog results in Search Console shows where to look for crawling and indexing information, and the SEO blog writing checklist for beginners includes the publishing steps that keep new posts discoverable.

Finally, revisit the structure only when something changes: a redesign, a migration, or a big growth in pages. Crawl hygiene is maintenance, not a campaign. A short check after each structural change catches most problems before they grow.

How Rankenza handles it

Rankenza keeps article pages clean: one canonical URL per article, internal links from the customer's sitemap, and no duplicate auto-generated pages in the article system. Each article starts from research of the top ten search results for its keyword, filtered for search intent, and a human reviews the draft before delivery. The first article is free, then $3.60 per article or $89 for 50. That cleanliness is deliberate: a new article should be easy to discover, easy to link to, and easy to keep at one address.

What the research says

According to Google's crawl budget guidance, titled "Optimize your crawl budget", a site's crawl budget is the set of URLs Google can and wants to crawl, determined by crawl capacity limit and crawl demand. The guide is explicit about who it is for: large sites with very many pages changing moderately often, medium or larger sites with very rapidly changing content, and sites with a large share of URLs classified as discovered but not currently indexed. It also states plainly that if a site's pages are crawled the same day they are published, an up-to-date sitemap and regular checks of the Page Indexing report are adequate, and the advanced guide need not be read at all.

The same guidance lists the practices that maximise crawling efficiency: managing the URL inventory, consolidating duplicate content, blocking unimportant URLs, returning proper not-found codes for removed pages, keeping sitemaps up to date, avoiding long redirect chains, and making pages efficient to load. The chart below turns that list into a qualitative diagram of where crawling attention goes on a tidy site versus a messy one. It shows no figures, because the source describes proportions qualitatively, not as amounts.

Chart source note: qualitative diagram based on Google's "Optimize your crawl budget" guidance and its best-practice list (see Sources). No figures are shown because the source gives none for this allocation.

How this looks on Rankenza

A typical article system might generate tag pages, parameter versions, and duplicate addresses for every post, quietly multiplying the URLs a crawler must sort through. Rankenza's own pages take the tidy side of the chart: one canonical address per article, as on measuring blog results in Search Console, internal links drawn from the customer's sitemap, and no duplicate auto-generated pages. Articles start from research of the top ten search results, filtered for search intent, with human review before delivery. The first article is free, then $3.60 per article or $89 for 50. No traffic or ranking figures are claimed here: the comparison is method against method.

Frequently asked questions

Does my small blog have a crawl budget problem? Probably not. If new posts are crawled promptly after publication, the standard routine, a current sitemap and occasional indexing checks, is adequate. Advanced crawl work is mainly for very large or rapidly changing sites.

Can I increase my crawl budget directly? Not by requesting a number. Capacity rises with server health and demand rises with content quality, freshness, and popularity. The levers available to a site owner are indirect: faster, more reliable pages and better, more current content.

Do subdomains share a crawl budget with the main domain? No. Google treats each unique hostname as a separate site for crawl budget purposes, so a subdomain has its own budget, separate from the main domain.

What is the single most useful crawl habit for a small site? Keeping the sitemap up to date and the internal linking clean. Those two habits point crawling attention at the pages that matter, and they help human readers navigate the site too.

Sources

• Google, Optimize your crawl budget (Crawl Budget Management) — capacity limit and crawl demand, who the guide is for, best-practice list.
• Google Search Central, SEO Starter Guide: The Basics — duplicate content and canonical versions, sitemaps, site organisation.