Provision your devops team

Laurent Arnoud

Feb 4, 2016 • devops, provisioning, and ansible

At TD, we are using Ansible to provision most of our servers. No matter if AWS, DigitalOcean or Geib IT.

Several months ago we started migrating from managed hosting to in-house DevOps using Ansible. This switch has been done for a single project in the first place (we were developing / maintaining 5-8 projects which were running on 10-16 different servers at that time). It turned out to be the right decision for a couple of reasons, but first and foremost for the ease and speed we gained maintaining our infrastructure, thus the time we won when we had to react to changes in application requirements. Another very important point was that we often had to setup similar infrastructures for the projects we worked on. Today we are maintaining an infrastructure with more than 30 servers and Ansible really helps us to take the pain out of maintaining them.

I will try to sum up a few of the best practises our DevOps team discovered during the past couple of months.

Reproduce the environments using Vagrant

Vagrant helps us to mimic real world environments like staging and production by setting up several VMs locally.

In a team which uses different development environments (Mac OS, Debian, Ubuntu, …), it helps a lot to standardize the way how VMs are accessed. The following plugins helped us to achieve that:

vagrant-hostsupdater, updates /etc/hosts automatically, also useful with aliases and testing deployments.
vagrant-cachier, speeds up provisioning by sharing a common packages cache among similar VMs.

Create a Vagrant stage for deployment testing

We mostly develop Ruby / Rails applications. For deployment we use Capistrano. Testing the deployment against a real world infrastructure is as simple as adding a Capistrano stage to your deployment with the hostname defined by vagrant-hostsupdater plugin.

This has been proven to be really useful for us. For example when new versions of puma / eye had been released, we added a new service to our stack or we had to reproduce and fix deployment errors.

Encrypt your credentials

We choose git-crypt over ansible-vault, because it is really easy to install, transparent for the user and much more integrated to our daily work.

One small drawback would be the .gitattributes file, which can get very long.

Create unencrypted fake credentials

This is useful if a developer needs to do a fix but doesn’t have the encryption key. With ansible we decided to use the common variable testing, this looks something like this:

- name: Load slack vars
  include_vars: "vars/slack.yml"
  when: testing is not defined

- name: Load testing slack vars
  include_vars: "vars/testing/slack.yml"
  when: testing is defined

Make your playbooks idempotent

This can be hard but it’s definitely worth it. It will help a lot to find problems with the playbooks.

Tag your provisioning

Tag using git or a monitoring tool (e.g. Datadog), will help you to quickly get information about the current state of the server.

Integrate to CI

Currently we are doing syntax checks and linting on the playbooks with ansible-lint.