Background for http://www.nzpost.co.nz
We are adding more moving parts to the New Zealand Post web and mobile sites all the time. Each time we make a code release, regression testing is critical to ensuring that none of the existing site functionality has been inadvertently broken. Twelve months ago we had no automated testing, and the effort to regression test site was around 2-3 person days per release.
Our current situation
During 2011, we have put a lot of effort into the creation and maintenance of automated tests. By automating much of the testing, that effort has been cut to around 3 hours per release, completely revolutionising our flexibility, responsiveness and ability to deliver.
Given that we release fortnightly (and would like to increase the frequency even more!), this automation also represents a major saving in time and resource. We now have in excess of 300 automated tests that run for every release. This is supplemented by functional testing and visual checks of the site to ensure that styles are OK.
We currently use Selenuim IDE and Firefox 10. The first year has been very much a setting-up process, and in the coming months we would like to upgrade the version of Firefox and increase the number of browsers covered.
Continuous Integration Testing
In addition to the tests we run for each fortnightly code release, we have also set up a Continuous Integration Testing (CIT) environment, over which around 250 continuous automated tests run several times a day. This environment is a stable copy of our development environment, and allows us to diagnose problems caused in the development process – before the code even hits UAT.
More to come…
I’ll be sharing more of our experiences and learnings in future issues.