Comments on: Yahoo Slurp Makes a Mess

By: Tobias Schwarz

Tobias Schwarz — Thu, 28 Jun 2007 23:49:02 +0000

I’m interestet in those robots.txt violations. Did you verify that those IP’s belong to Google / Yahoo by doing a reversedns lookup and pinging the host resolved? On bigger projects there are always some users running aroung with Googlebot or Slurp-Agent.

By: Dr. Pete

Dr. Pete — Thu, 28 Jun 2007 20:41:07 +0000

I’ve had a lot of trouble with Yahoo with a client recently. Their overall visitors and unique visitors have increased by almost 50% over the past 3-4 weeks, and it looks to be almost entirely visits from Slurp, across a wide range of IPs. It seems to be obeying robots.txt to some extent, but not robots META tags, and I’m also getting massive duplicate visits.

By: Nathan Holley

Nathan Holley — Thu, 28 Jun 2007 18:35:50 +0000

this was discussed on the LED Digest http://www.led-digest.com/content/view/1703/190/ and also here by Barry http://www.seroundtable.com/archives/007215.html .

By: michael

michael — Thu, 28 Jun 2007 17:41:54 +0000

I think i found my answer! si?

Google in Action and Other Graphs -

By: michael

michael — Thu, 28 Jun 2007 17:40:23 +0000

Great post - great charts.

Are the above charts reported from your analytics tools that you use, or can they be found elsewhere?

Thank you,
mb

By: matt

matt — Thu, 28 Jun 2007 12:52:30 +0000

When’s Urbanspoon coming to Providence?

By: SISTRIX Suchmaschinen Blog

SISTRIX Suchmaschinen Blog — Thu, 28 Jun 2007 09:59:58 +0000

Googlebot vs. Yahoo Slurp…

Adam Doppelt hat in seinem Blog einige interessante Daten und Vergleiche zum Crawlverhalten der Bots von Google und Yahoo zusammengetragen. Analysiert wurden dazu die Logfiles von urbanspoon.com, einem Restaurantführer für die USA. Während der Googl…