ProjectX Technology

I’ve been having performance issues on a 512Mb VM where I run 3 wordpress sites for projectx and lushai. The problem was that the memory footprint default apache configuration would explode when traffic arrived. Lots of traffic would cause the Apache prefork module to spawn more processes until the machine started swapping and then cause the VM to spiral into a forced reboot.

Part one of my solution was to switch to Nginx webserver. I still wanted to use Apache, because I’m using the mod_pagespeed library for some extra performance juice, but I didn’t want to blow out memory usage. The solution involves using Nginx as a reverse proxy to apache. This allowed me to reduce the number of Apache servers down to 10. Nginx as a reverse proxy uses about 5Mb of memory It worked a treat . But it was a little slow See the AB test below.

projectx@projectx:/px/sites/httparchive$ ab -n 1000 -c 100 http://www.projectx.co.nz/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.projectx.co.nz (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: nginx
Server Hostname: www.projectx.co.nz
Server Port: 80

Document Path: /
Document Length: 85698 bytes

Concurrency Level: 100
Time taken for tests: 112.181 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 85937000 bytes
HTML transferred: 85698000 bytes
Requests per second: 8.91 [#/sec] (mean)
Time per request: 11218.133 [ms] (mean)
Time per request: 112.181 [ms] (mean, across all concurrent requests)
Transfer rate: 748.10 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 3.2 0 13
Processing: 474 10689 2000.6 11051 12967
Waiting: 272 10458 1996.3 10821 12692
Total: 487 10690 1997.7 11051 12967

Percentage of the requests served within a certain time (ms)
50% 11051
66% 11220
75% 11381
80% 11525
90% 11927
95% 12409
98% 12687
99% 12799
100% 12967 (longest request)

I wasn’t happy about the performance, so I decided to throw Varnish as a proxy cache into the mix.

Using WordPress Varnish and varnish wordpress tips from the varnish site, I was able to quickly configure Varnish on the server.

Here’s the result in the performance, from the AB test.

projectx@projectx:~$ ab -n 1000 -c 100 http://www.projectx.co.nz/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.projectx.co.nz (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software: nginx
Server Hostname: www.projectx.co.nz
Server Port: 80

Document Path: /
Document Length: 85689 bytes

Concurrency Level: 100
Time taken for tests: 0.366 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 86009000 bytes
HTML transferred: 85689000 bytes
Requests per second: 2733.35 [#/sec] (mean)
Time per request: 36.585 [ms] (mean)
Time per request: 0.366 [ms] (mean, across all concurrent requests)
Transfer rate: 229582.36 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 4 12 4.6 11 23
Processing: 16 25 4.9 24 36
Waiting: 2 10 5.3 10 23
Total: 29 36 4.6 37 42

Percentage of the requests served within a certain time (ms)
50% 37
66% 41
75% 41
80% 41
90% 42
95% 42
98% 42
99% 42
100% 42 (longest request)

Its a massive improvement from 8 reqs/sec to 2700 reqs/sec. Now I can remove Nginx from the mix, and now to tune mod_pagespeed to optimise my site

I’ve been inspired by Pat Meenan’s excellent posts of the State of optimization summary and the state of web performance optimization 2010 , so I thought I’d do another “Homepage hall of shame“. – which is a web performance review of the top 100 NZ based websites. (Top 100 sites is based on Alexa traffic).

I’ll post the full report soon, here are some of the findings…

Summary Findings

• The homepages for 100 of the top NZ based websites was measured using  webpagetest.org.nz on 1.5Mb DSL connection Thanks to aptimize.com.
• The average page size of homepages was 554.6K Up from (359.6K in May 2009 & 305.1K in 2008)
• Fastest home pages load < 2 sec. Top site loads in 1.46 secs ! The average download time was 9 secs using broadband connection. And average 4.6 seconds for a cached reload.
• 62% of websites use  WEB COMPRESSION
• The largest pages were over 2000K. The largest site was 4.6M and had images that were more than 1Mb.
• A lot Advertising javascript still is not being compressed
• Some sites have dozens of Javascript and CSS files, worst offender had 34 javascript files.
• The fastest sites have less total requests. There seems to be a strong co-relation to lower total requests to speed.
• There are two main things that websites should do speed up their sites. 1) Properly Set cache expiry 2) Javascript + CSS bundling.

Percent of sites that got a passing grade for the basic optimization checks:

**Note: I’ve removed the CDN measurement as very few NZ sites use a CDN.

For more web optimisation comparison check out the Home Page drag race.

It been a long long while since I’ve written a blog post. This year has been a bit of a blur with lots of things happening – lots of consulting to companies, selling the services side of ProjectX, going on a holiday for two months and more consulting.

This blog will change as the focus of “ProjectX” the company has changed to represent my personal consulting business and interests. I’ve got a number of blog post queued up that I’ll be pushing out over the next few weeks and months.

Web performance: Its been a while since I posted an update to the “NZ Homepage hall of Shame” and I’ve created some interested videos on how “fast” a few of the key websites actually are to load compared to their market alternatives.

Blood rules: I’ve got some suggestions, advice, observations and learnings from the world of running an IT Start-up. These rules have been earned in “blood” ie. learnt the hard way via pain by those concerned. Topics that I will cover include founders, communication, seeking advice, partnerships and execution.

Geo-Location: I’ve been following with interest the rise of Foursquare and a demise of Yellowpages. I’ve got some interesting thoughts on where the industry is heading and how New Zealand will be lagging behind.

[The picture is of me posing next to a massive bath tube at Vatican Museum in Rome]

Geo-visualisation and visualisation is becoming more main-stream with media publications creating more and more great visualisations. The New York times have been releasing a constant stream of great geo visualisations, they even have their own visualisation lab on the NYT site. (I’ve also notice that the Guardian has their own http://www.guardian.co.uk/news/datablog also.) Last week the NY Times released an interactive mashup featuring the popularity of Netflix rental across several cities in US.

Using the mashup, you change look change the titles and see the rental popularity of the movie across different zipcodes. Its a great mashup to bring interactivity to data.

I discovered that one of the authors Matthew Bloch has a blog page with all list of the visualisation he’s created for NY Times. Wow, what a great selection of visualisations!

Here’s a sample of my favourite visualisations.


2008 Election results – Obama vs McCain


US Movie Revenue popularity


US Inflation breakdown

Hopefully, as data is becoming more open, we’ll see more Geo-visualisations in New Zealand.

Recently, we released the cycling and walking journeyplanner for our clients the Greater Wellington Regional Council. The journeyplanner website is an advanced map mash-up show advanced directions from different parts of the  region. What makes this site special is that we have combined a lot more data from the councils to include walkways, cycleways, tracks and extra information from all over the regional.

There are a number of unique features:

• Custom routing and directions of walking tracks over the entire region eg.  walking around the Botanical gardens

Routing through the Botanical Gardens

• Suggested journeys from all over the region to highlight some of more scenic walks around the region.

• Draggable routing over all streets, tracks and walkways around the region. We build our own custom routing engine with draggable routing front-end over google maps.

Routes are fully drag-able

Simply select and hold the line, drag to where you would like

Then let go. The route and description will automatically update.

• Elevation graph with live feedback on the map. See the little icon travel across the map as move across the elevation graph.

• Points of interest showing locations of variety of points for both walkers and cyclists such as landmarks, pedestrian crossings, traffic lights, bike shops, bike racks, street lighting and toilets.

• Calorie counter to figure out how much energy you could burn.

This has been one of the bigger projects for ProjectX and we’ve got a number of people and partners to thank.

Jill, Simon and Ian from GW. Thanks for your belief in the project and the patience to see it through to end, from wireframes to through all eight iterations until the final release. Thanks to Michael for being our physical location guru who helped connect the physical world to the data files.

Big thanks to the ProjectX team for the work on their hard work on project – Thong, Tim, Gaetan, Stephen, Cameron, Boris, Felix, Raja and Nahum. Dr Geof Leyland from Incremental for the providing our routing algorithm and advice on making it super-fast.

Our design team via Lushai user experience team- Lulu for her interaction design, and wireframes Amiria for the lead graphics, Brent and Chika for supplementary graphics.