Comments on: 4 unusual tricks to not give SE-robots a chance to waste all your traffic http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/ web, technologies, seo, web2.0, web3.0 (why not?), etc. Fri, 12 Mar 2010 05:02:17 +0000 http://wordpress.org/?v=2.7.1 hourly 1 By: L. Stetz http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/comment-page-1/#comment-28460 L. Stetz Sun, 02 Nov 2008 14:12:46 +0000 http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/#comment-28460 I've been hit by a bot attack from Yahoo. Over 30 hours this week and last week too. they've been getting more aggressive for weeks. My question: Don't I limit Yahoo crawler?(crawl.yahoo.net) I mean, what's with this "Slurp"? Also MSN is doing it too with their inktomisearch.com search bot. They all just send a different bot every time. I've got lots of images and want them on the searches. of course, as you say, Google doesn't update and alot of the images they 've plucked from my pages don't match search queries. I just wonder about the NAMES to limit the bots. Can you give any suggestions? I’ve been hit by a bot attack from Yahoo. Over 30 hours this week and last week too. they’ve been getting more aggressive for weeks.
My question: Don’t I limit Yahoo crawler?(crawl.yahoo.net) I mean, what’s with this “Slurp”?
Also MSN is doing it too with their inktomisearch.com search bot.
They all just send a different bot every time.

I’ve got lots of images and want them on the searches. of course, as you say, Google doesn’t update and alot of the images they ‘ve plucked from my pages don’t match search queries.

I just wonder about the NAMES to limit the bots. Can you give any suggestions?

]]> By: John H. Gohde http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/comment-page-1/#comment-25308 John H. Gohde Mon, 09 Jun 2008 03:12:13 +0000 http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/#comment-25308 I tend to agree that excessive hits from search engines is a waste of bandwidth. If half of your hits are coming from search engines then your site definitely has a problem. I think think that most sites really don't want to have their images indexed. It is an issue of copyright violation. But, really have made no effort to prevent it. In the past, I have experienced excessive indexing from Yahoo. With Google, it will index the same web pages over and over again while totally ignoring new content. I tend to agree that excessive hits from search engines is a waste of bandwidth. If half of your hits are coming from search engines then your site definitely has a problem.

I think think that most sites really don’t want to have their images indexed. It is an issue of copyright violation. But, really have made no effort to prevent it.

In the past, I have experienced excessive indexing from Yahoo. With Google, it will index the same web pages over and over again while totally ignoring new content.

]]> By: I made update of tricks with saving traffic and SE-robots http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/comment-page-1/#comment-9887 I made update of tricks with saving traffic and SE-robots Fri, 20 Apr 2007 11:59:40 +0000 http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/#comment-9887 [...] April 20, 2007 at 7:59 am · Tags: update   tutorial For long time I was going to update my tutorial ‘3 unusual tricks to not give SE-robots a chance to waste all your traffic’, finally done with this. Now it’s ‘4 unusual tricks to not give SE-robots a chance to waste all your traffic‘. [...] [...] April 20, 2007 at 7:59 am · Tags: update   tutorial For long time I was going to update my tutorial ‘3 unusual tricks to not give SE-robots a chance to waste all your traffic’, finally done with this. Now it’s ‘4 unusual tricks to not give SE-robots a chance to waste all your traffic‘. [...]

]]> By: samlowry http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/comment-page-1/#comment-200 samlowry Tue, 05 Sep 2006 19:17:16 +0000 http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/#comment-200 >if they come back often is beacuse they think your site is worth spidering.deeper. You are absolutely wrong. I mean huge number of requests recieved in short time. Yahoo or MSN crazy robots can make several K of hits perd day to a small well indexed site. I experienced the real DDOS attack to one of my server from Yahoo Slurp before I made Crawl-delay:10 for all sites on that server. >get a real host if your serious about the web I have several ;) And I am very serious. >if they come back often is beacuse they think your site is worth spidering.deeper.
You are absolutely wrong. I mean huge number of requests recieved in short time. Yahoo or MSN crazy robots can make several K of hits perd day to a small well indexed site. I experienced the real DDOS attack to one of my server from Yahoo Slurp before I made Crawl-delay:10 for all sites on that server.

>get a real host if your serious about the web
I have several ;) And I am very serious.

]]> By: Maurice http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/comment-page-1/#comment-197 Maurice Tue, 05 Sep 2006 16:04:50 +0000 http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/#comment-197 Yeh Right I want to stop search engines spiding my site - if they come back often is beacuse they think your site is worth spidering.deeper. get a real host if your serious about the web or leave it to the pros Yeh

Right I want to stop search engines spiding my site - if they come back often is beacuse they think your site is worth spidering.deeper.

get a real host if your serious about the web or leave it to the pros

]]> By: Easy Webbers » Blog Archive » Speedlinking http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/comment-page-1/#comment-158 Easy Webbers » Blog Archive » Speedlinking Mon, 04 Sep 2006 19:03:43 +0000 http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/#comment-158 [...] - An ajax enhancement for wordpress blogs. - 3 unusual tricks to not give se-robots a chance to waste all your traffic. - Generate a screenshot of your website in about 5 seconds, for free. - 10 things businesses should know before building a website. - Pageviews are obsolete? - If you must have a shoutbox, this is the one to use. - And finally some usability guidelines. [...] [...] - An ajax enhancement for wordpress blogs. - 3 unusual tricks to not give se-robots a chance to waste all your traffic. - Generate a screenshot of your website in about 5 seconds, for free. - 10 things businesses should know before building a website. - Pageviews are obsolete? - If you must have a shoutbox, this is the one to use. - And finally some usability guidelines. [...]

]]> By: Jonathan http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/comment-page-1/#comment-110 Jonathan Sun, 03 Sep 2006 12:16:35 +0000 http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/#comment-110 Some good advice. However, I find people often visit my site through an image search, so I wouldn't want to disable that. I suggest people check their logs first to see where visitors come from. The image crawlers don't visit too often: Google image search is always quite out of date with lots of broken links (which isn't good from the user's point of view!) Some good advice. However, I find people often visit my site through an image search, so I wouldn’t want to disable that. I suggest people check their logs first to see where visitors come from. The image crawlers don’t visit too often: Google image search is always quite out of date with lots of broken links (which isn’t good from the user’s point of view!)

]]> By: The Web Design Blog http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/comment-page-1/#comment-99 The Web Design Blog Sun, 03 Sep 2006 01:28:53 +0000 http://web3.0log.org/2006/09/02/3-unusual-tricks-to-not-give-se-robots-a-chance-to-waste-all-your-traffic/#comment-99 [...] If bandwidth is an issue for you or your clients, this technique will stop search engine spiders from over-crawling your website, and more importantly, stop the spiders from crawling and indexing your images, because images tend to use more bandwidth, and lets face it, no-one really views your site when using image search. Big waste of bandwidth. « Turning visitors into users   [...] [...] If bandwidth is an issue for you or your clients, this technique will stop search engine spiders from over-crawling your website, and more importantly, stop the spiders from crawling and indexing your images, because images tend to use more bandwidth, and lets face it, no-one really views your site when using image search. Big waste of bandwidth. « Turning visitors into users   [...]

]]>