GoogleBot Using huge amounts of bandwidth

Hey Guys

My new site www.tubefetch.com is only 3 weeks old, and i have noticed that the site is being crawled huge amounts...especially by google bot.

In the first 3 weeks we had 26 different robots crawl the site, and google bot ALONE used over 12GB of bandwidth. Today (first day of the month) we have had 6 different crawlers including google bot, adsense, MSN and Yahoo and Google Bot has used 1.1GB of bandwidth (today!).

The re-visit meta tag is set to 20 days, but the google, msn and yahoo bot visited both today and yesterday.

Does anybody know why this is? It just doesn't seem to make any sense to me and im not sure what consequences it could have.

Thanks
Josh

dam, I never realized... thanks for sharing

Try using a robots.txt file.

Could you explain that a bit more please joe...thanks

Originally Posted by Josh.UK Could you explain that a bit more please joe...thanks

http://www.robotstxt.org/ would do a much better job explaining anything you may want to know about robots.txt files than someone here, you may want to have a look.

Thanks amross

This used to happen on my older forums. On one hand its great that bots are filing your forum, however if you have a small hosting plan bots can be dangerous to your site's uptime lol. Back then I had a huge hosting plan of 2000 gigs of bandwidth, 200 gigs of storage, for $70 per year, so bots were not much of a problem.

Same for me Erik, I'm on dreamhost and their limits are well....unlimited so to say.

I would be actually happy if Google crawled my site every day (of course it also matters the fact if you can afford that much of bandwidth), as Amross advised you on robotstxt.org you could find your solution. But i am not sure if just BLOCKING the bot would actually be a Solution.

You could register on Google for free, from there, you can chose the speed and how many times the Googlebot should crawl your site. Take some times (if you have) and test that feature (personally i haven't tested it).

Cheers.

i wish i could be so lucky to have that much traffic from bots. but if you really think this is bad, try google sitemaps. it allows you to see how much google is crawling and for you to slow it down.