Is Googlebot unnecessarily crawling a large number of distinct URLs on your site?

Have you seen this message? “Googlebot encountered an extremely high number of links on your site.” I have and more than once and more than twice. The main problem with organic internet marketing is understanding that the core, the foundation, of your success is technology and having Google to go through unnecessarily high numbers of URLs can be caused by a variety of problems for your online marketing efforts. One of them being that Google just gives up and throws your web out.

If your URLS look like this:

http://www.example.com/search/noheaders?click=6EE2BF1AF6A3D705D5561B7C3564D9C2&clickPage=OPD+Product+Page&cat=79

Start by getting a Google Webmaster account and activate it, implement it on your site. Then look at your data as it is collected, you might actually get an email from Google Support saying something like:

“Googlebot encountered an extremely high number of links on your site. This may indicate a problem with your site’s URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.”

If this is the case ACT on it, this is Google giving you heads up before the real problem starts.

Remember that the best solution is to choose a good search engine friendly web CMS when you start your online presence or you are reinventing your company online.

Be Sociable, Share!
  1. How do I get this Google Webmaster account?

  2. Glad to see your post, and I have received the warning recently. Does it mean I cannot use the complicated URL format any more? As you know the web pages about search have such kind of URLs normally. Or we just tell the robot not to crawl these part of content. Any suggestions plz?

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Switch to our mobile site