ProjectX Blog

Here’s my two cents about Yahoo search.

We get a lot of traffic from search engines. So we spend a lot of time optimising our SEO and our site to make sure we have as much content in the search engine index.

When it comes down to the hard numbers of referrers, 98% of our search engine referrers come from Google and only 2% comes from Yahoo. (We have about the same number of pages in both indexes) Even though Yahoo is now powering the search for Xtra, the return for us is still so low. We’d rather spend our time and energy optimising SEO for Google we get far more return.

I’m sorry Yahoo, you don’t deliver the traffic to make it worth our while keeping our index fresh. As a result of poor click return, we restrict the rate that Yahoo crawler can access our site. (Not that they pay attention to robots.txt !)  That way Google has more of our CPU cycles and bandwidth to crawl our site.

<rant>

I’ve just got a response from Yahoo about a question I asked them yesterday and I got a “canned” response.

When will companies, learn that canned responses wastes your customers time!

Here’s my email to yahoo.

Hi

Yahoo Slurp is ignoring robots.txt for my site.

I had the rule

User-agent: Slurp
Crawl-delay: 200

and it is being ignored. (Its worked for months like this)

I have now updated our robots.txt about 12 hours ago to

User-agent: Slurp*
User-agent: Yahoo*
Crawl-delay: 200

And its still ignoring the robots.

What’s going on ?

Here’s there response.

Thank you for writing to Yahoo! Search.

We apologize for the excess traffic to your site. In a continuing effort
to improve our index freshness and comprehensiveness we are bringing
online additional crawler machines to prepare them for production.

To limit the amount of traffic your site receives, you can place a
crawl-delay in your robots.txt, specifying the number of seconds each
crawler will wait between each request. Please check out the following
pages for detailed information about this:

Thanks Yahoo for answering my question. NOT
</rant>

The Velocity conference on web performance and operations is in full swing there are a wealth of information in the slides.

Here is a selection of the most interesting presentations:

• AOL PageTest – AOL have created an opensource tool call PageTest to test performance on IE 7/8 and they have created a online page test to show the performance of your website from East coast USA (Slides)
• High performance Ajax application – Yahoo. Another awesome slide deck from the Yahoo performance team. (Slides)
• Improving Netflix performance – Mesuring and testing performance at Netflix (Slides)
• Jiffy open source performance measuring tool – a new tools to measure the performance of your javascript. (Zip of Slides)
• Profiling Dynamic Web Applications in Firebug – (Slides)

Enjoy!

Xlinks is a selection of interesting links as discovered by the ProjectX staff:

Study proves Bad boys really do get the girls
Added on 06/23/2008 at 03:15PM
The Pixar Touch – The Making of a company – NY Times
Added on 06/23/2008 at 03:11PM

Happiness as your business model
Added on 06/23/2008 at 03:09PM

Hacking TCP/IP to support location awareness
Added on 06/23/2008 at 12:44PM

Bootstrappers beware
Added on 06/21/2008 at 01:01PM

Ajax Avenue no exit
Added on 06/20/2008 at 10:51PM

Subversion 1.5.0 released – simpler branching and merging
Added on 06/20/2008 at 04:36PM

17 mistakes that startups make
Added on 06/20/2008 at 08:48AM

The power of remarkable
Added on 06/20/2008 at 07:02AM

Web browsers and Memory Fragmentation
Added on 06/20/2008 at 07:01AM

Visualisation of the Code swarm for Python: AWESOME
Added on 06/19/2008 at 05:35PM

The Google User experience
Added on 06/19/2008 at 03:13PM

Adaptive path behind Myspace re-launch
Added on 06/19/2008 at 09:42AM

Open Cell Id database
Added on 06/19/2008 at 09:21AM

Value of a College Education for Ajax Developers
Added on 06/18/2008 at 11:24AM

Cooking for engineers
Added on 06/18/2008 at 10:15AM

Web design trends 2008
Added on 06/17/2008 at 11:28AM

VC’s Angels or bootstrapping
Added on 06/17/2008 at 10:19AM

Create more value than you capture.

– Tim OReilly
Added on 06/17/2008 at 09:23AM

Modules.js – a standalone dynamic javascript loader
Added on 06/17/2008 at 08:58AM

Its official, Google are now giving priority to faster sites. (As mentioned a month ago) Slow sites risk their “Quality” score being reduced.

So its important to Google….

Larry and Sergey, articulated this philosophy in Ten things Google has found to be true. One of these principles is “Fast is better than slow.” We’ve found this rule to be especially applicable to the landing pages of AdWords ads. When a user clicks an ad, a landing page that loads quickly provides a better user experience than a landing page that loads slowly.

So here’s the rub.

Starting today, this load time factor will be incorporated into your keywords’ Quality Scores. Keywords with landing pages that load slowly may get lower Quality Scores (and thus higher minimum bids).

Now that your Google ads monies at risk, there’s never better a time to speed up your pages.

We’re lifting the lid on some functionality that we’ve been working for a while to make it easier to browse content on ZoomIn. For the first release (currently in “Garage Mode”) we’re focusing on browsing photos. So now you can look at all of the photos around any location, as you zoom and pan around the photos will constantly update.

To get the browse photos explorer either click on  explore photos on the home page or search for any address or place and expand the results box to reveal the Explore photos link.

We have two modes of browsing Photo bar and Thumbnails.

Photo Bar style

Browse the photos via the photo bar.

Thumbnail style

View all of the photos on the map.

Explore Photos is in Garage mode, which means we’ll be tinkering and improving over the next few weeks.

We appreciate your suggestions and feedback, so let us know what you think.

Firefox 3 has been officially released and looking under the hood it has been tweaked for faster page loading.

Here are the key connection values in Firefox 2 and 3.

As we can see the maximum number of connections and number of connections per server has been increased. So that’s good news for more parallel downloading. And they’ve also increased the number of persistance connections so that future requests will be a bit faster. Another interesting thing is that http pipelining is turned off by default.

So far Firefox 3 has been snappier and more memory efficient! Download Firefox 3 today and squeeze a bit more performance from your bandwidth!

Xlinks is a collection of interesting links as discovered by the ProjectX Team

Steve Yegge on Server side Javascript
Added on 06/17/2008 at 07:21AM

The passive agressive programmer or why nothing gets done
Added on 06/16/2008 at 02:44PM




Added on 06/16/2008 at 02:40PM

Now you can upload PDFs into Google Docs
Added on 06/16/2008 at 02:40PM

Calculus in 20 minutes
Added on 06/16/2008 at 02:39PM

Facebook no longer the second largest social network
Added on 06/16/2008 at 02:39PM

How to create keypress navigation in jQuery
Added on 06/16/2008 at 02:37PM

37 signals on You have to treat employees like customers
Added on 06/16/2008 at 02:36PM

Algorithmic ink in Javascript
Added on 06/16/2008 at 02:34PM

Zen Planning: 4 Simple Rules for Balancing Short Term Necessity with Long Term Direction
Added on 06/16/2008 at 02:33PM

NZ Broadband is trailing the rest of the world
Added on 06/16/2008 at 02:03PM

Runtime page optimiser
Added on 06/16/2008 at 12:19PM

Javascript framework usage at top sites
Added on 06/12/2008 at 08:39AM

37 signals link to BMW’s Gina
Added on 06/11/2008 at 10:13PM

Yahoo Design templates -wireframes
Added on 06/11/2008 at 05:58PM

Yahoo Design Patterns
Added on 06/11/2008 at 05:57PM

Free Rails 2.1 PDF
Added on 06/10/2008 at 04:53PM

How not to be viral – software adoption patterns
Added on 06/10/2008 at 03:49PM

iPhone NZ bound July 11
Added on 06/10/2008 at 12:54PM

Passenger 2.0 rc1 and ruby enterprise released
Added on 06/10/2008 at 09:43AM

How cool is this ????

Here’s a look at some ZoomIn traffic from last Sunday. The video was based on the realtime visualisation of ZoomIn using the glTrail tool by fudgie.org. (They also make the awesome gltail tool)

Who knew ZoomIn would build its own deathstar !

The team from ActionThis have created a new product called Runtime Page Optimizer that will dynamically optimise your webpages at runtime. It looks like a fantastic product to solve a lot of the problems highlight by the NZ Homepage hall of shame.

Runtime Page Optimiser features:

• Combines all javascript and css on the fly
• Generates CSS sprites on the fly
• No code modifications required!
• Runs on ASP.net 2 and and IIS 6/7
• Compatible with IE6/7 , Firefox 2/3, Safari 3

The product is currently in Beta and from the demos it already looking really great.

Well done guys, I look forward to seeing this product hit 1.0!