Speeding up Google’s Crawl Rate

So Google now lets you control the max crawl rate through a nifty slider.

While most sites start at a max-allowed rate of 0.5 pages/sec, I’ve been able to get a few sites to 10 pages/sec.

I had some large sites that had pretty low max rates (eg 1 page/sec), and it took me some figuring out how to get Google to go faster quicker – sitemap. By submitting a sitemap with a sizeable # of pages, we were able to boost from 1 page/sec to 2.5 pages/sec in roughly 96 hours.

Across all my sites Googlebot hits me at least a million times every day.

Return to top