So, what better to do on a lazy sunday than clean up some of the Fedora Infrastructure ansible playbooks to be idempotent. I ran into several common cases in our playbooks for something to not be idempotent, so I thought I would share them in case others run into the same issues.
First, for those of you wondering what ‘idempotence’ is, wikipedia describes it thusly: “the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application”. More practically in the ansible world it means after 1 run of a playbook to set things to the desired state, further runs of the same playbook should result in 0 changes.
In Fedora Infrastructure we run a ansible-playbook run of each of our host or group playbooks with ‘–check’ and ‘–diff’ every night from cron. Then, a handy report is mailed to our sysadmin-logs members showing all the playbooks that showed something would have changed. In the ideal state this report would be empty, but there’s various reasons things show up in there.
Of course the most common reason a playbook would show up in our check/diff report is that something actually changed and the playbook has simply not yet been run on the host(s). This could result from someone checking in a change to git and not running the playbook, some variable or global change was checked in and the person doing it didn’t realize they needed to run this particular playbook, someone made a manual change on the host (BAD sysadmin!), or perhaps the host was not reachable or up when the playbook was last run. This is the easiest case to fix up: just run the playbook(s) and everything gets back to the steady state.
Another common case I see is when a file is changed multiple times in the same playbook run. This can result from a simple copy paste error, or a conditional that isn’t correct (a ‘when:’ in ansible terms), or having the same template or file in multiple roles or directories and not realizing that you are setting it in two places. This can be fixed by only changing the file once, either by collapsing the task into one place, or adding conditionals so it only applies once on each host. Sometimes in rare cases you DO need to change a file multiple times (for example, put a admin config in place, run some admin commands, put the normal user config in place). In these cases you can use ansibles “when_changed” to just tell it this isn’t really a change and don’t bother marking itself as well. Do be careful using this.
Another case I see a good deal is when you run a command or shell (which is fine, –check simply ignores them because they always change) but register a variable thats used later in the playbook. In this case there’s a problem when running with –check because the command/shell is never run, the variable isn’t set and when you try and use it the playbook fails. In these cases you can use: “always_run: yes” and the command/shell will run. Of course you may also want to add a “when_changed:” so it doesn’t always show as changed.
I reduced our non idempotent playbooks today a good deal, but there’s still more work to be done. I do look forward to the day the report is empty, and I don’t think it will be all that far off. 😉