Google Says Most Sites Needn't Worry About Crawl Budget

Google has recently been publishing more information about the crawl budget.

Last month, the search giant dedicated an episode of its SEO Mythbusting YouTube series to the topic. Before that, Google’s John Mueller answered a query about the benchmark for the crawl budget.

Crawl budget refers to the number of pages that the Googlebot crawls and indexes when on a website within a specific period. It usually depends on the site’s speed and user demand.

In the latest episode of the Search Off the Record podcast, Google’s Gary Illyes discussed the topic in length.

According to the developer advocate at Google, a “substantial segment” does have to care about the number of times that the search engine bot crawls a page. However, the majority of sites needn’t worry about it.

Illyes explained:

“And I stand my ground, and I still say that most people don’t have to care about it [crawl budget]. We do think that there is a substantial segment of the ecosystem that has to care about it.”

So, how do you know whether or not to care?

When Should you Worry About Crawl Budget

It’s tempting to believe that there’s a hard number when it comes to how Google crawls a page

For example, you might think that a site must have a specific number of pages before the crawl budget becomes an issue. However, Illyes pointed out that it doesn’t work that way.

He noted:

“… well, it’s not quite like that. It’s like you can do stupid stuff on your site, and then Googlebot with a start crawling like crazy. Or you can do other kinds of stupid stuff, and then Googlebot will just stop crawling altogether. “

Illyes further stated that roughly a million URLs are the baseline before a site owner needs to care about the crawl budget. That means websites with fewer than a million URLs don’t need to worry.