Etckeeper for version control on Linux

Version control for your Linux /etc/ directory

Version control for your Linux /etc/ directory
Image credits : 

Original photo by Rikki Endsley. CC BY-SA 4.0

One of the many advantages of Linux, UNIX, and similar operating systems is that everything is a file and that most of your configuration is done via text files, allowing you to easily read and write to them with any tool you choose. To monitor your configuration, you have many tools for automating configurations, providing you with detailed control over how your systems are configured. You can easily compare different versions of a configuration file to see any changes. Tracking specific configurations and changes over time allows you to:

  • See what changed recently when getting a wake-up page
  • Revert changes when package management stomps all over your finely crafted snowflake of a configuration
  • Restore a file when an inadvertent rm hits production rather than the test server you thought you were on
  • See what has changed when multiple people (and tools) work on a system
  • Investigate changes after a script kiddie gets in

For decades, we've been tracking changes in text files with version control systems (VCS). We have the tools and we use them all the time. Version control for your source code gives you features such as tracking file history, diffing versions, reverting to an old version, and branching.

Your /etc/ directory is where most Linux distributions store your system's configuration files, and tracking /etc/ has some extra challenges beyond what a version control system tracks. Etckeeper helps us with those extra requirements. Etckeeper prefers git, but also works with several other version control systems: mercurial, bazaar, or darcs.

Other tools such as configuration management and the distribution's package management also control configurations, but they don't track all the files in version control.

Configuration management systems (CMS), such as Puppet, Chef, and Ansible, do not track every file in /etc/. For many files, they only verify that certain contents are in place and ignore the rest of the file. For instance, Puppet will verify a user exists in /etc/passwd, but will ignore changes to accounts it's not managing.

Package files have base configuration files and maybe also scripts for automagical configuration, but they don't track modifications. Indeed, they might stomp over local configuration during a package upgrade. They shouldn't, but sometimes they do.

Etckeeper will track everything except the repo directory (for instance, /etc/.git) if it is not configured to ignore some files. The default configuration does ignore some files that don't need tracking, such as ephemeral files like /etc/mtab and cache files like /etc/ld.so.cache. Etckeeper uses the version control system's native ignore file and that file is tracked in the version control repo. For instance, /etc/.gitignore is pre-populated with the files that etckeeper will ignore, but /etc/.gitignore is tracked by git.

Some services are a little persnickety about configuration file ownership and permissions and even empty directories, all things not traditionally tracked well by a version control system. Etckeeper augments the underlying version control system by tracking this important metadata in /etc/.etckeeper. Once again, this file is tracked in the version control system.

Etckeeper also has hooks to work with the package management system and check in changes after package installs and upgrades.

Getting started with etckeeper

Once installed, use init to initialize the repo and then check in the current state.

$ sudo etckeeper init
$ sudo etckeeper commit -m "Initial checkin"

Once initialized, /etc/ is a repo in your version control system. Native version control tools can be used to manage files.

$ sudo touch /etc/testfile
$ sudo git -C /etc add testfile
$ sudo git -C /etc commit -m "A test file"
$ echo "10.10.10.1111 nextcloud" | sudo tee -a /etc/hosts >> /dev/null
$ sudo git -C /etc/ diff -U0 hosts
$ diff --git a/hosts b/hosts
index 97f1792..ecc187a 100644
--- a/hosts
+++ b/hosts
@@ -9,0 +10 @@ ff02::2 ip6-allrouters
+10.10.10.1111  nextcloud
$ sudo git -C /etc/ commit hosts -m "Added nextcloud box"
$ sudo git -C /etc/ revert HEAD -m "oops, wrong IP"
$ echo "10.10.10.111    nextcloud" | sudo tee -a /etc/hosts >> /dev/null
$ sudo git -C /etc/ commit hosts -m "Correctly added nextcloud box"

Etckeeper also provides a version control system wrapper. One advantage of using the wrapper is that sudo etckeeper commit will log the account using sudo rather than root.

$ sudo etckeeper commit
$ sudo etckeeper vcs status
$ sudo etckeeper vcs -C apache2 status

Want to see what configuration changes were made by an upgrade? Review your version control system logs.

commit 327f09703be712aab263b9565fede5c21f378732
Author: der.hans <etckeeper@example.com>
Date:   Wed Feb 15 07:24:24 2017 -0700
 
    committing changes in /etc after apt run
 
    Package changes:
    +vim-addon-manager 0.5.3
    +software-properties-kde 0.92.25debian1
    +vim-fugitive 2.1-2

Etckeeper is a great tool for capturing changes to system configuration over time. It complements native version control system capabilities with wrappers to track important metadata, such as permissions and ownership and work with package management systems. See the etckeeper documentation for further usage examples such as how to clone an /etc/ repo from one system to another.

3 Comments

AB

Nice article; another product that helps watch for changes is something like Tripwire.

It may be worthwhile to note that a lot of the benefit here is already managed by systems using BtrFS for the root filesystem. Seeing changes, rolling forward/back those changes (plus along with other directory structures where other important files, like the kernel, system libraries, etc., all live, is all magically handled with BtrFS snapshots. Integrate that with the system package manager and you have a powerful solution. This has been the norm for a year or two now with openSUSE and SUSE Linux Enterprise products. We've used them in our organization and these features really give us a lot of flexibility, both by tracking /etc changes and also by tracking the rest of the important directories.

Vote up!
0
Vote down!
0
lufthans

Yeah, tripwire will watch everything if you want. Last I used it was many years ago, does it have an option for keeping revisions and being able to view previous content?

I will have to think about snapshotting like with BtrFS. Maybe I can add that to my presentation on Saturday.

I think it doesn't cover all of the features I want, but it might provide sufficient features to cover the general use case. Like tripwire, it will also cover more of the system than just /etc/.

In any case, I really like etckeeper with snapshotting :).

Vote up!
0
Vote down!
0
Utku

Or just use a configuration management (CM) tool (such as Ansible, Puppet, etc.) and make all your changes to a repository of your CM tool.

Vote up!
0
Vote down!
0

Comment now