Etckeeper lets you store changes in /etc to git | Hacker News Reader

Etckeeper lets you store changes in /etc to git

github.com

107 pointsdz0ny38 comments

More great software from Joey Hess.

http://joey.hess.usesthis.com/

I liked the part about: I do everything on a netbook, a Dell Mini 9. I’ve been using it since 2008, and have worn out two keyboards.

He doesn't look like his fingers would be any smaller than mine, but I have tried to use a netbook and failed. I just ordered myself a new laptop primarily because I didn't like the keyboard on my ultrabook. Joey, I am humbled by you.

The cool thing about Etckeeper is that it plugs directly into your package manager. If you're using deb or rpm, a commit is made after each package install if they changed /etc.

Ah! Now there's a reason to look through the source code. I've being wanting to make update scripts for online projects that are not in my distribution's repositories. (I'd love it if GO updated when I told my computer to update).

Wouldn't a simple inotify[0] watch on /etc handle that scenario? Does it split up the commits by package?

[0] http://linux.die.net/man/1/inotifywait

Wouldn't that commit as soon as any file was changed, no matter who or what changed it, denying users the ability to supply a meaningful commit message?

That depends on the filesystem events you tie into (Open, Close, Write, Delete, etc) and what your responding script does. But, at the point where you want to put the entire system under some kind of change management, why not use something designed for that, like NixOS[0]?

[0] http://nixos.org/nixos/

inotify consumes resources (ram), which a hook into a package manager does not. you cannot watch all subdirectories within /etc with one handle, so you will have to recursively setup handles for all subdirectories. then, you might run into the max_user_watches limit (see sysctl fs.inotify.max_user_watches).

Acutally, using fanotify() you can watch a whole filesystem with one watcher; use FAN_MARK_MOUNT (inotify and fanotify both use the fsnotify subsystem in the kernel). So then you set a watch on either /etc (if it is separately mounted) or watch / and then ignore any paths that aren't in /etc.

small nitpick, it does not hook into dpkg and rpm but into apt and yum (there are surely hooks for other package managers). so when you install a package without these higher level package managers, it does not get recorded...

I've been using this for quite some time, but mostly as a 'fire and forget' tool. There were some rare occasions when the information from the git history was useful though, like finding out how the configuration looked for an older version of a program.

I have one big etckeeper repo for all of my personal machines with each machine being a separate branch. It comes in handy when you cant remember how you have something set up on another machine and or for when i get a new machine.

I've been using etckeeper on all my machines for a few years now. It's basically "install and forget about it", but it's been a tremendous help on a few occasions when something broke!

Does anyone recommend a good GUI for viewing Linux log files on Windows or Mac? I'd pay for an OSX-Console-like app that'll connect to a remote Linux distro and allow easy browsing/searching of /var/log/, or any paths you add to it.

The better architecture is to bring logs to a centralized place and do analysis on them that way. Your GUI tool doesn't scale for real work, particularly if the machine you are investigating logs upon has been compromised or is broken.

You could setup splunk free (or one of the OSS clones like logstache or greylog2) on the Linux host and use that web interface for visualizing and searching the logs.

I highly recommend Logstash and Kibana. The setup might be a little more complex then some would desire (but by no means is it difficult and both have solid docs), but the two are pretty powerful tools.

1 - http://www.logstash.net/ 2 - http://kibana.org/

I ve known people doing it with SVN/Git for at least 7-8 years now.

And with RCS for almost 30 years.

In a way RCS is a better architectural fit for this than git, because you can check /etc/shadow in and only one file is used to version it, unlike with git where the whole /etc/.git/ directory has to be locked down to avoid exposing its contents.

Oddly, RCS is about the only VCS not supported by etckeeper yet. (Well, and CVS and svn). It'd be pretty easy to write the 8 or so scripts needed to add support for it to etckeeper.

Doesn't Mercurial also maintain version info in one file below .hg per file in the tree?

Is there something like this that works for arbitrary files or directories? Basically, I'd like to have something that monitors files and/or directories for changes and then stores the diffs in git possibly via a cron job or a daemon using inotify? Anything analogous to this would be great.

I've contemplated writing something like this myself many times, but time is just not something I have a lot of these days.

http://sparkleshare.org/ is sort of a Dropbox clone backed by git, that might be what you want. There are a few other things in this space but Sparkleshare, by virtue of aiming to replace Dropbox, is the most polished IMO. There's also this script called gitwatch that does pretty much exactly what you want without the GUI: https://github.com/n3v1k/gitwatch . I guess if you don't need to push to a remote repository automatically this would be the better option (and you can easily stick in push/pull in the right place in the bash script if you want).

At first I thought this was a cute lead up to git-annex and then I realized it was a straight forward question and that you want incremental diffs. You should take a look at git-annex[1] one of joey's other projects if the diff feature is not a deal breaker.

"git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space.

git-annex is designed for git users who love the command line. For everyone else, the git-annex assistant turns git-annex into an easy to use folder synchroniser."

The only joeyh project that I use more than git-annex or etckeeper is moreutils.[2] (Obviously I'm not including joey's enormous contributions to Debian) Included in moreutils:

  chronic: runs a command quietly unless it fails
  combine: combine the lines in two files using boolean operations
  ifdata: get network interface info without parsing ifconfig output
  ifne: run a program if the standard input is not empty
  isutf8: check if a file or standard input is utf-8
  lckdo: execute a program with a lock held
  mispipe: pipe two commands, returning the exit status of the first
  parallel: run multiple jobs at once
  pee: tee standard input to pipes
  sponge: soak up standard input and write to a file
  ts: timestamp standard input
  vidir: edit a directory in your text editor
  vipe: insert a text editor into a pipe
  zrun: automatically uncompress arguments to command

The only program that I never have a use for is ts, I use tai64n[3] from djb.

[1] http://git-annex.branchable.com/

[2] http://joeyh.name/code/moreutils/

[3] http://cr.yp.to/daemontools/tai64n.html

Recently the git-annex assistant has gained the ability to check selected files directly into git, so you can have your diffs too if you want them.

http://git-annex.branchable.com/design/assistant/blog/day_22...

Been doing this with RCS for decades.

I'd been toying around with different versions of this for a bit; / and etc under git control with .ignore files keeping it limited to only the specific files I wanted. This sounds like a far more mature and robust solution, so I'll likely be switching to it shortly.

Having used this for about six months, can confirm that it, plus gitweb, have saved our bacon a good few times. Handy when, as it usually does, push comes to shove and you have to restore right the fuck now, and only have a browser on you.

A good analogy is etckeeper is to /etc/* as RANCID is to a (real) router and its configuration. So now if you know one, you know the other.

I worry about the security risks of saving your entire /etc to "somewhere on the internet"...

I use etckeeper, but only keep the repository locally. I use it so I can keep track of what changes there are to /etc, to help with that time when something breaks.

I'm still a little disappointed it doesn't ignore /etc/shadow by default. I can't imagine a valid use case for keeping /etc/shadow in source control.

Clearly there are reasons to keep /etc/passwd in version control. Changes to /etc/passwd often need to be synchronized with changes to /etc/shadow.

For something like professionally administered servers accessible only via ssh asymmetric keys, /etc/shadow may not be considered so sensitive that it's worth the risk of having it be an exceptional cases in version control.

That's indeed a bad idea. etckeeper creates a local repository in a protected .git directory inside /etc, don't confuse git with github or other hosting services. I use etckeeper on various machines, just to track changes in configuration files (both those caused by people and package updates). On top of that I use a conventional encrypted backup solution. I never push or pull the git repo anywhere, and I think that's the normal way of working with etckeeper.

hopefully /etc/passwd and all those other important security related files are excluded?

That is what I worry about uploading dotfiles as they may contain important bits about my system.

Yes thats excluded, however your .git is stored locally and it would need be root user to even look at directory.

yeah would be nice freebsd support.

Reply on news.ycombinator.com