Automating Deployments: Why bother?

At my employer, we developed a new software architecture. This involved developing and deploying several new components, many of them following the same pattern: A daemon process listing on a message bus (RabbitMQ, in case you're wondering) and also talking to existing applications: A database, an Active Directory service, a NetApp cluster or a VCenter, you name it.

Shortly after the development of these components begun, it was decided that a different team than before should operate the software we developed. The new team, although dedicated and qualified, was also drowning in other work.

As we had them deploy the first few components, it became clear that each new deployment distracted them from doing what we wanted most: build the infrastructure that we and our software needed.

As programmers, automating thins is much of our daily business, so why not automate some steps? We already had a Jenkins running for executing tests, so the next step was to automate the builds.

Since our systems run Debian GNU/Linux, and we build our applications as Debian packages, distributing the software meant uploading it to an internal Debian mirror. This proved to be a trickier than expected, because we use debarchiver for managing the Debian repositories, which doesn't give immediate feedback if an upload was successful.

After that, a deployment involved only an apt-get update && apt-get install $package, which at first we left to the ops team, and later automated too - though in the production environment only after a manual trigger.

Many of the manual and automatic deployments failed, usually due to missing resources in the message bus, so we automated their generation as well.

Reduced feedback cycles

So at $work, automating deployments first was a means to save time, and a means to defend the architectural freedom to develop several smaller components instead of few small components. Later it became a means to improve reliability.

But it quickly also became a tool to reduce the time it takes to get feedback on new features. We found it notoriously hard to get people to use the staging environment to try out new features, so we decided to simply roll them out to production, and wait for complaints (or praise, though we get that less often).

Being able to quickly roll out a fix when a critical bug has managed to slip into the production environment not only proved useful now and then, but also gave us a feeling of safety.

I'm writing a book on automating deployments. If this topic interests you, please sign up for the Automating Deployments newsletter. It will keep you informed about automating and continuous deployments. It also helps me to gauge interest in this project, and your feedback can shape the course it takes.