ProjectX Blog

<rant>

I’ve just got a response from Yahoo about a question I asked them yesterday and I got a “canned” response.

When will companies, learn that canned responses wastes your customers time!

Here’s my email to yahoo.

Hi

Yahoo Slurp is ignoring robots.txt for my site.

I had the rule

User-agent: Slurp
Crawl-delay: 200

and it is being ignored. (Its worked for months like this)

I have now updated our robots.txt about 12 hours ago to

User-agent: Slurp*
User-agent: Yahoo*
Crawl-delay: 200

And its still ignoring the robots.

What’s going on ?

Here’s there response.

Thank you for writing to Yahoo! Search.

We apologize for the excess traffic to your site. In a continuing effort
to improve our index freshness and comprehensiveness we are bringing
online additional crawler machines to prepare them for production.

To limit the amount of traffic your site receives, you can place a
crawl-delay in your robots.txt, specifying the number of seconds each
crawler will wait between each request. Please check out the following
pages for detailed information about this:

Thanks Yahoo for answering my question. NOT
</rant>

This entry was posted on Friday, June 27th, 2008 at 7:50 am and is filed under Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.