Crawl Budget

Crawl Budget

What Is Crawl Budget?

Crawl Budget is the number of pages Googlebot crawls and indexes on a website within a given timeframe.

Why Is Crawl Budget Important for SEO?

In short: if Google doesn’t index a page, it’s not going to rank for anything.

So if your number of pages exceed your site’s crawl budget, you’re going to have pages on your site that aren’t indexed.

Very high page total might affect indexing

That said, the vast majority of sites out there don’t need to worry about crawl budget. Google is REALLY good at finding and indexing pages.

That said, there are a few cases where you do want to pay attention to crawl budget:

  • You run a big site: If you have a website (like an ecommerce site) with 10k+ pages, Google can have trouble finding them all.
  • You just added a bunch of pages: If you recently added a new section to your site with hundreds of pages, you want to make sure that you have the crawl budget to get them all indexed quickly.
  • Lots of redirects: Lots of redirects and redirect chains eat up your crawl budget.

With that, here are some simple ways to maximize your site’s crawl budget.

Best Practices

Improve Site Speed

Improving your site’s page speed can lead to Googlebot crawling more of your site’s URLs.

In fact, Google states that:

“Making a site faster improves the users’ experience while also increasing crawl rate.”

In other words:

Slow loading pages eat up valuable Googlebot time.

Slow loading pages are hard to crawl

But if your pages load quickly, Googlebot has time to visit and index more of your pages.

Quick load means more time to index other pages

Use Internal Links

Googlebot prioritizes pages that have lots of external and internal links pointing to them.

Yes, ideally you’d get backlinks pointing to every single page on your site. But that’s not realistic in most cases.

That’s why internal linking is so key.

Your internal links send Googlebot to all of the different pages on your site that you want indexed.

Internal links send Googlebot to all pages you want indexed

Flat Website Architecture

According to Google:

“URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.”

And in the world of Google, popular=link authority.

That’s why you want to use a flat website architecture on your site.

A flat architecture sets things up so that all of your site’s pages have some link authority flowing to them.

Flat architecture lets link authority flow to all pages

Avoid “Orphan Pages”

Orphan pages are pages that have no internal or external links pointing to them.

Avoid unlinked orphan pages

Google has a really hard time finding orphan pages. So if you want to get the most out of your crawl budget, make sure that there’s at least one internal or external link pointing to every page on your site.

Limit Duplicate Content

Limiting duplicate content is smart for a lot of reasons.

As it turns out, duplicate content can hurt your crawl budget.

Factors affecting crawl budget

That’s because Google doesn’t want to waste resources by indexing multiple pages with the same content.

So make sure that 100% of your site’s pages are made up of unique, quality content.

This isn’t easy for a page with 10k+ pages. But it’s a must if you want to get the most from your crawl budget.

Learn More

Optimize your crawling & indexing: A helpful guide to how Google finds, crawls and indexes pages.

Complete Guide to Crawl Budget Optimization: Super in-depth video on optimizing your crawl budget (includes real life examples).

Crawl Stats report (websites): A post from Google on how to read and interpret the Crawl report in the Google Search Console.