Foo Camp – Summary of "How to run big f**king websites"
I thought I’d do a write up on one of the technical sessions at Foo Camp, Artur Bergman’s of Sixapart fame on “How to run big f**king websites”. Being a system admin / network architect in a previous life, I found it very very interesting. As you know from my previous articles, I’m obsessed with website optimisation and make webpages faster.
Background
Sixapart run a number of large scale websites, typepad, movabletype, vox and livejournal. Artur provided some stats on livejournal to give us an idea of how big these site are. eg. There are 2,000,000 Typepad users and at any given instant there are about 500,000 people reading LiveJournal, with 100,00 to 200,000 live http connections!
How do they do it ?
One of the key concepts that Artur talked about was using software to manage application and operating redundancy. (Chris Di Bona also confirmed that they use software to manage redundancy at Google). Artur didn’t pull an punches, he slammed nearly every hardware manufacturer, raid system and file system. The problem as they have experienced is that all hardware or software drivers fail at one time or another. Why should they waste money on buying more expensive kit ? Does price really contribute to reliability? For SixApart, it has been better for them to buy cheap hardware ie machines, memory, hard drives etc. and solve reliability and redundancy via software.
Software Tools
Arthur spoke a lot about four tools that they have developed to help run their sites and keep them fast!
Memcache is a distributed memory cache that reduces hits to your database. (I have it on good authority that Yahoo and Google are heavily using memcache to speed up their sites.) Memcache stores “objects” in memory for access within your application, thus with some smart design you can dramatically reduce the number of hits on the database. Arthur warned that memcache does require a little bit of care when managing the expiry of your cache objects.
MogileFS is a distributed filesystem. MogileFS is flexible ie. it requires no kernel extensions and it is filesystem agnostic and it provides “better than raid” as it manages the distribution of files not only across disks but also machines. In this way it has been designed so that there is no single point of failure.
Perlbal is a Perl-based reverse proxy load balancer and web server. Perlbal can be used to manage http connections among servers. It can act as a webserver or a reverse proxy. They have built in the smarts so that Perlbal can handle resuming downloads and connecting to another backend servers if one stream fails.
Gearman is a system to farm out work to other machines. It dispatches function calls to machines to do work in parallel and to load balance lots of function calls.
Wrap-up
It was an amazing talk to hear first hand what what worked and what didn’t and why they have developed the tools to make their site fast. The amazing thing is that Sixapart guys have released their software as open source for the benefit of the rest of us!
June 25th, 2007 at 2:59 pm
[…] Bergman (who has recently joined O’Reilly radar and gave a kick-ass talk about his experiences in scaling live journal at foo camp.) offers some sage advice for those people that are comptemplating using EC2 and […]