Marketing 2 min read

Google Says Most Sites Needn't Worry About Crawl Budget

PureSolution /

PureSolution /

Google has recently been publishing more information about the crawl budget.

Last month, the search giant dedicated an episode of its SEO Mythbusting YouTube series to the topic. Before that, Google’s John Mueller answered a query about the benchmark for the crawl budget.

Crawl budget refers to the number of pages that the Googlebot crawls and indexes when on a website within a specific period. It usually depends on the site’s speed and user demand.

In the latest episode of the Search Off the Record podcast, Google’s Gary Illyes discussed the topic in length.

According to the developer advocate at Google, a “substantial segment” does have to care about the number of times that the search engine bot crawls a page. However, the majority of sites needn’t worry about it.

Illyes explained:

“And I stand my ground, and I still say that most people don’t have to care about it [crawl budget]. We do think that there is a substantial segment of the ecosystem that has to care about it.”

So, how do you know whether or not to care?

When Should you Worry About Crawl Budget

It’s tempting to believe that there’s a hard number when it comes to how Google crawls a page

For example, you might think that a site must have a specific number of pages before the crawl budget becomes an issue. However, Illyes pointed out that it doesn’t work that way.

He noted:

“… well, it’s not quite like that. It’s like you can do stupid stuff on your site, and then Googlebot with a start crawling like crazy. Or you can do other kinds of stupid stuff, and then Googlebot will just stop crawling altogether. “

Illyes further stated that roughly a million URLs are the baseline before a site owner needs to care about the crawl budget. That means websites with fewer than a million URLs don’t need to worry.

Read More: Mueller Explains Why Sites Lose Their Google Rankings

First AI Web Content Optimization Platform Just for Writers

Found this article interesting?

Let Edgy Universe know how much you appreciate this article by clicking the heart icon and by sharing this article on social media.

Profile Image

Edgy Universe

EDGY is an SEO incubator, forecaster, and support center for deep learning, technological advancement, and enterprise-level end-to-end search programs.

Comments (0)
Most Recent most recent
share Scroll to top

Link Copied Successfully

Sign in

Sign in to access your personalized homepage, follow authors and topics you love, and clap for stories that matter to you.

Sign in with Google Sign in with Facebook

By using our site you agree to our privacy policy.