Ansible enables infrastructure as code. It has been a valuable tool for my infrastructure@home project, helping me to setup the programs Consul and Nomad, to provide DNS for nodes and services, and an Nginx endpoint. Although this infrastructure was completely replaced by Kubernetes, Ansible is a tool that stayed and provides continuous support for system maintenance tasks.
Having worked with Ansible for a longer time, my scripts now reside in one well-structured directory. From here, I can run individual tasks as well as running the complete site.yml
to apply one consistent configuration state to all systems. It is also helpful to consider uninstallers that will remove unwanted configuration or software. And to achieve idempotency of your playbook, it is important to truly understand the Ansible modules that you are using. All of these is detailed in this article.
Effective Directory Layout
All ansible code is contained within one single directory. At the time of writing this article, this directory looks like this:
├── ansible.cfg
├── host_vars
│ ├── raspi-3-1.yml
│ ├── ...
├── group_vars
├── roles
│ ├── consul
│ ├── docker-arch
│ ├── docker-arm
│ ├── nfs-client
│ ├── nfs-server
│ └── nomad
├── scripts
│ ├── consul
│ ├── nomad
│ ├── configs
│ ├── jobs
│ ├── system
│ ├── update_packages.yaml
│ ├── tutorial
│ └── uninstall
├── hosts
└── site.yml
The directory is structured according to these principles:
- Global config files in directory root: The ansible config files
ansible.cfg
and the inventoryhosts
. - Global playbook: The playbook
site.yml
is an idempotent playbook that, when executed, configures all nodes with all the infrastructure systems that I have. Effectively it installs all the roles on all nodes and patches the nodes to the newest OS packages. - Global vars: The
group_vars
directory contains global variables, especially the IP addresses for the Nomad, Consul and NFS servers. In thehost_vars
directory, I include a file for each node that determines its Nomad/Consul/NFS role as being master or agent. - Separating roles and scripts: The
roles
directory contains playbooks that install infrastructure systems. These are the ansible roles I explained in earlier articles. Inscripts
are commands that are executed regularly, things like updating the nodes or restarting processes on the nodes. I also include the deployments of Nomad jobs, including config files for programs, inside the script directory.
Uninstaller
Ansible is concerned with providing repayable, consistent configuration for systems. But sometimes you need to undo these changes. If it’s just uninstalling software with the respective package management software, this is an easy task. But what about other configuration: Custom service files, configuration files, added configuration lines to central system files like /etc/mount
or /etc/ssh/sshs_config
?
There is no uninstaller in Ansible, but every change you make can be undone. Write this uninstaller yourself! Write it as soon as you are finished with the installation part, because its fresh on your mind and you can test it with your system.
I structure the playbooks into an install
and uninstall
block. These blocks are invoked by passing the uninstall=true
parameter to the run.
Here is a shortened example for installing dnsmasq
.
- block:
- name: Install dnsmasq
apt:
name: dnsmasq
state: present
- name: Configure dnsmasq
lineinfile:
path: /etc/dnsmasq.d/10_consul
create: true
line: server=/consul/192.168.2.201#8600
regexp: consul
state: present
when: uninstall is not defined
- block:
- name: Uninstall
apt:
name: dnsmasq
state: absent
- file:
path: /etc/dnsmasq.d
state: absent
when: uninstall is defined and uninstall
When this playbook is run as ansible-playbook ... -e "{uninstall: true}
, the output is as follows:
ansible-playbook site.yml --limit=raspi-3-1 --tags dns -e "{uninstall: true}"
PLAY [Configure DNS] ***********************************************************************************************************************
TASK [dns : Install dnsmasq] ***************************************************************************************************************
skipping: [raspi-3-1]
TASK [dns : Configure dnsmasq] *************************************************************************************************************
skipping: [raspi-3-1]
TASK [dns : Restart dnsmasq] ***************************************************************************************************************
skipping: [raspi-3-1]
TASK [dns : Uninstall] *********************************************************************************************************************
changed: [raspi-3-1]
TASK [dns : file] **************************************************************************************************************************
changed: [raspi-3-1]
Idempotent Playbooks
When a playbook is executed to configure a system, the system should always have the same, well defined state. If a playbook consists of 10 steps, and the system deviates in step 4 from the desired state, then only this particular step should be applied.
By its nature, Ansible tasks will only change the system if there is something to do. Most Ansible modules provide this idempotency. But some modules can be used in a way that breaks this pattern. I cannot cover all modules, but want to highlight this point with one example: The module Line in File.
Let’s consider the setup of mounting an NFS volume. On the clients, you need to add an entry to etc/fstab
for telling the system where the NFS server is. The first version of this change is this:
- name: Create fstab entry
lineinfile:
path: /etc/fstab
line: '{{ nfs_dir_server_ip }}:{{ nfs_dir_mnt_path }} {{ nfs_dir_mnt_path }} nfs defaults,soft,bg,noauto,rsize=32768,wsize=32768,noatime 0 0'
state: present
The particular output is dependent on three variables. If one of the variables change, and the playbook is executed again, it will create a new entry into /etc/fstab
. The mounts will not work!
You want only one entry for NFS shares in this file. And for this, you can use the additional regexp
parameter. The second version of this playbook uses the regexp
parameter to only and ever add one line to /etc/fstab
. This simple change makes the playbook idempotent.
- name: Create fstab entry
lineinfile:
path: /etc/fstab
line: '{{ nfs_dir_server_ip }}:{{ nfs_dir_mnt_path }} {{ nfs_dir_mnt_path }} nfs defaults,soft,bg,noauto,rsize=32768,wsize=32768,noatime 0 0'
regex: 'nfs defaults,soft,bg,noauto'
state: present
Whenever you use a module, take care to use it idempotently.
Conclusion
During continuous use of Ansible, my collection of individual scripts evolved into one effective directory structure. Knowing exactly where to put which playbook, variables or configuration files is essential. Also, when writing a playbook, always add an uninstaller so you can undo changes to the system. Finally, be sure to write idempotent playbooks so that repetitive playbook execution always results in one well defined, consistent system state.