Drupal Deployment part 2: how we do it at New Zealand Post

This is the second post in the Drupal deployment series looking at how we deploy updates to our Drupal sites nzpost.co.nz, stamps.nzpost.co.nz, and coins.nzpost.co.nz. In part one, senior Drupal developer Neil Bertram from Catalyst IT outlined the problem of Drupal deployments on large sites and looks at the foundations we depend on, source control and deployment with Debian packages. In this post he’ll review the modules we rely on, custom hooks, and how we deploy. In part three we think about seamless releases that we can be run on demand, and the three biggest issues we face with deployment.

The modules we can’t live without

There are a few modules we need to tip our hat to, as they have made what was once either impossible or very difficult now fairly routine. They are:

Drush: Drush has become the centre of our universe. Not only does it allow many operations that once needed to be done through the web interface to be scripted via the commandline, but there’s a vast and ever-growing collection of contributed modules that integrate support for it and make deployment far simpler. We use Drush for everything from packaging up the code for release to setting things up once the code makes it to prod. Many of the other modules below build on Drush, and we have a reasonable library of our own Drush commands that ease our development workflow.

Features: Although I suspect Features was never intended to be a deployment tool, this is basically all we’ve ever used it for. While Views made exportable configuration cool, Features (when paired with Chaos Tools) made it accessible and consistent. We use Features together with Strongarm to persist most configuration and Panels changes into easily deployable code. Features also integrates very neatly with Drush, which allows us to do the export and “revert” (upgrade/installation) operations from the commandline and scripts.

Environment: The Environment module is a relative newcomer to the Drupal scene, but fills a gap that we always used to plug with custom logic. Now there is a standard way (with Drush integration!) to make a site self-aware of which environment it’s running in, and configure itself accordingly. We use this to enable or disable things that shouldn’t happen anywhere but production, as well as swapping out certain configuration values that need to be different in staging or development environments. With some custom hooks, Environment can be used to clean up a database that’s being pulled down to a development machine to rid it of customer data that has no place being there. This module can also be effectively paired with Environment Indicator to make it easy for users and testers to figure out which environment they’re looking at, which can be a lifesaver when someone inadvertently logs into prod to make a change that they meant to make elsewhere.

UUID: While not strictly related to deployment, the UUID module helps fix a really big problem for anyone who’s needed to programmatically merge something content-like into a moving production environment. Because Drupal uses integer IDs on entities natively, it’s very difficult to write code that refers to a user/node/etc before that entity exists. In normal operation, the UUID module simply assigns an auxiliary UUID to almost everything Drupal creates, but when scripting a content import it’s possible to specify the UUID up-front. This means a custom module can know ahead of a release what UUID to expect to find a special entity at without having to know what numeric ID was assigned to it at release time. It’s almost as good as being able to specify a fixed machine name for an entity!

It’s worth noting that we do not use the Deploy module as it doesn’t fit with the way we work. It’s largely used for content migration, which is something we don’t do very often, and we have our own way to accomplish it that works for us (see the next section).

Custom update hooks for everything else

Unfortunately there are still some things that are hard or don’t work properly with Features, especially anything to do with blocks, roles, content or module maintenance. We’ve found this is better accomplished using special update hooks.

Each site has a special custom module installed that exists solely to perform scripted changes to state at deploy time. Anything that needs to be deployed that can’t go in a feature module is done through update hooks on this module. We’ve gradually accumulated a library of helper functions in this module to perform many common operations (some overlapping with what can now be done with Features), such as:

  • Create roles, and add permissions to them

  • Allocate or remove blocks from regions

  • Install or uninstall modules

  • Import content and/or menu structures

  • Configure modules that are configured through custom tables instead of variables (and don’t provide an Exportables interface)

  • Revert a feature as an update hook (in case another update hook depends on something exported in a feature)

Installing new modules is particularly important to us. While Features allows for dependencies, there is nothing that causes a dependency to be automatically installed when the feature is updated. Instead, the feature will just complain that a dependency is missing. We definitely do not want to have to be installing modules through the web interface, so we use custom update hooks to accomplish this.

The updates are applied using “drush updatedb”, of course. There are some gotchas with this, which I’ll discuss a little bit later.

How we do a deployment

Once we have a set of changes ready, features exported to code, and update hooks written, we need to create a release.

We do this using the drush dh-make command to build a Debian package for the release and tag it in Git. The produced package is then transferred to our internal repository server and digitally signed with the appropriate key so the servers will trust it.

Next, we use a custom shell script that knows how to deploy that package neatly to a staging environment where it can be tested. That script is responsible for restoring a recent backup of the production database and files to the staging server before deploying the package (which is actually handled by Puppet). It then runs a custom Drush meta-command that knows the required steps to deploy to a new environment. In sequence it’ll do:

  1. drush vset site_offline 1 to put the site offline

  2. drush env-switch to switch to whatever environment it is, which will clean up the database and configure the site for that environment

  3. drush updatedb to run any pending update hooks

  4. drush feature-revert-all to apply any changes contained in features

  5. drush varnish-purge-all to clear the Varnish cache servers

  6. drush cc all to rebuild caches, including rebundling JS and CSS files with Advagg

  7. drush vset site_offline 0 to put the site back online

You can see why we like Drush now, right?

Once the site is fully tested (you can read more about how we test in an earlier post), a production release is scheduled (usually very early in the morning, when the site is not too busy) and the same deployment procedure as above is applied to the production environment, with the same package that was signed off in testing. The release process is pretty much fully automated, but we usually have a small team of people awake to watch over it and to sniff out any problems that occur in case we need to roll back.

If we do need to roll back in production, unfortunately we would normally need to restore a database backup from just before the site was taken offline for the release, then roll the Debian package back to the previous version. Drupal unfortunately doesn’t provide rollback hook support like some other systems do. Because Drupal doesn’t allow any state changes (like a customer to submit an order) while the site is offline, it’s relatively safe to restore from a backup before the site is brought back online without losing any data.

One trick we have up our sleeve to reduce the impact of our outage windows is to use Varnish Saint Mode and Grace during a release window to continue to serve pages from the cache while Drupal remains offline. Larges swathes of our site, including most static content and the most popular tool on our site, the Address & Postcode Finder, typically remain accessible while Drupal is offline for a release. Even in the early hours of the morning there’s still a fair trickle of traffic to the site, and if we don’t have to inconvenience those users, we try not to. If a user browses outside the pages that Varnish has, they’ll receive a typical Drupal maintenance page.

Any ideas for us?

Are you looking after a major Drupal site? We’re always interested in hearing how others solve the problems we’ve faced.

Next time we dream about seamless releases that can be run on demand, and the three biggest issues we face when we deploy.

Thanks for reading!

This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.