It would certainly seem that way as crawl rates seem to have plummeted dramatically in recent months. I came across the problem when working on a scrap car website for one of my clients and noticed that the changes I had made were taking a ridiculously long time to index. This prompted me to look at some of my other sites and low and behold it was happening on some of them too.

The graph below is from webmaster tools and shows crawl rates from month to month. As you can see crawl rates have slowed down for some reason.

So why are crawl rates so slow recently? Google has not given any official line of the recent trend but after doing some research on the topic some people are suggesting that Google’s bots have become more efficient and there is no need for them to crawl as frequently. I am not sure if I buy into that but will be interesting to see what happens over the coming months.

  • Simon Wharton

    I can’t see how the bots could be considered more efficient if they crawl less. How would they pick up on relevant change? Doesn’t make sense to me

  • Mark

    they were implying that although the bot is crawling less it is crawling more pages. I would not agree with the view either.

  • Roy

    Maybe even Google bots need a summer break?

  • Simon Wharton

    Ah, we think we’ve cracked this. Looks like a duplicate content issue. One of the clients other domains which is has been redirected to the main domain but using a server based DNS re-direct and returns a 200 OK message. We’ve asked for it to be changed and identified a few other domains where a similar technique was used.

  • Tim

    I don’t see why a slightly dodgy redirect would reduce the crawling google does on your site. Did you notice a change once the problem was fixed? It looks like a random blip to me.

    Did you guys achieve the current results for Metro Salvage? Looks like you did your job pretty well. 😉

  • Mark

    Hi Tim,

    Thanks for the comment. The crawl rate returned to normal after a while but its hard to say whether that was down to the redirect or not.

    Thanks for the kind words on the Metro Salvage project. We worked pretty hard to get them the rankings they achieve. It also converts extremely well.