the economist

Selenium is an automated software testing framework for web applications. It drives a real browser automatically, as if a real person was doing the clicking and typing. We use it here at The Economist for functional / black-box / acceptance testing, the kind of tests that are repetitive and time-consuming to step through manually.

We’ve setup a system that runs our full suite of regression tests in parallel, using virtual machines in the Rackspace cloud, on every commit to trunk. The entire suite takes just under 8 minutes to run, which is the length of the longest test. A screencast of each individual test run is captured in full motion MP4 video, and made available as a build artifact in Hudson for anyone to download and watch. The video is captured at full resolution (1200 x 2000), but you can see a scaled down version below of an actual test running on Economist.com:

We have a fun concept here that we call a “human test instance”. Anyone can create an entire copy of Economist.com, running in the cloud, on a subdomain of their choice, on a branch of their choice, with just a few clicks in Hudson. “Human testing” implies that the instance is used by a human to do manual testing, however we use a human test instance to run automated Selenium tests, and each one also effectively tests the process of running the update functions every day.

Economist developers are a big, distributed lot. At one point we had 4 teams doing development in 4 different timezones from San Francisco to London. As you might imagine, this presents some extremely interesting challenges, like source code control and maintaining communication channels between teams to reduce the amount of toe stepping and work duplication going on.

The single tool that has had the largest positive impact on code quality and general sanity has been Launchpad. We switched from Subversion to Bazaar / Launchpad in March 2010 and although we didn’t do any official metrics, I would guess we have slashed the defect rate by at least half since doing that.

Running an out-of-the-box stack like MAMP may be fine for part-time tinkerers, but if you’re writing code for a serious site you’ll quickly realise that it’s important to develop against the same version of the stack that’s running in the production environment. You’d be amazed at how many subtle bugs emerge between two minor point releases of PHP, APC, MySQL or any other of the myriad components. Catching them early is crucial, and the easiest way to do that is to use a virtualised developer environment. A good choice for Apple OSX is VMWare Fusion: