Comment by hkhanna | Hacker News Reader

Comment by hkhanna | Hacker News Reader

Parent Full thread

The parent comment captures why systemd makes me uneasy, and why I moved from Arch to Debian when I noticed Arch had replaced init with systemd.

The Art of Unix Programming (the book) suggests that software should do one thing and do it well, with easy, plain-text interfaces to its input and output. Systemd tries to do everything -- replacing by default so many core system components like syslog and cron.

But what really bothered me was when I realized that systemd stores all the system logfiles in binary by default! I couldn't believe it. All the system log files: boot, kern, messages, whatever, were stored in some binary format. At that point, rather than research how this new binary log file was implemented, I abandoned Arch and systemd for the init system that had served me well for 15 years.

Log files don't need to be in binary. It means we can't use our standard tools: cut, sort, grep, awk, etc. It means we can't use scripts we have written built on those tools we may have built over decades of working with UNIX.

I guess my point is, systemd, by storing things in binary format and by trying to be the 'everything' daemon, violates two core UNIX programming philosophies: use plain-text interfaces & do one thing well.

Log files being in binary lets you do things like filter based on specific fields rather than on more complicated string matching. For example, you can get log file messages from postfix's smtp process (as opposed to its smtpd process) with one command (journalctl /usr/lib/postfix/smtp), which you can then cut/sort/grep/awk/etc.

You can also follow the logs with journalctl -f and go from there as well.

The problem that this solves is the classic 'what log file does this daemon log this error level to on this system', which varies a surprising amount between daemons, versions, distributions, etc.

I don't actually care which log file I look at, I just want to know what errors monit is getting. systemd and journalctl provide that handily.

I would agree with an approach to have systemd send log events to e.g. journald (or syslog-ng, or whatever the user wanted) to separate that functionality out, however.

That sounds handy, but I wonder if json could provide the same thing.

It is binary so you can quickly filter (indexes). E.g. on service, but any field is possible. You can output it in json. systemctl gives you the last lines of service output exactly because of the indexes + binary file. Using json for that just because "OMG binary" seems strange. Json is not readable.

Postgres can query inside json. To be honest I don't care that much about logging as long as it is saved to disk.

I don't mind journalctl in spite of my initial reservations, and as you point out with -f, it's not all that different from tailing a text log. I do suspect binary logging isn't the only reason hkhanna dumped Arch for Debian. It smells of nitpickery to me.

Using journalctl instead of hunting for the appropriate log files in /var/log seems to offer some degree of platform unity. Maybe it's not the right answer, but it works well enough. My primary complaint is how long it can take on a noisy system to filter a large log, but it's not exceptionally noisome. And most of the time, I just want to follow the log.

> I would agree with an approach to have systemd send log events to e.g. journald (or syslog-ng, or whatever the user wanted) to separate that functionality out, however.

While it probably isn't what you mean as you attach the caveat of task separation, it is possible for syslog-ng to collected logging from systemd's journal. It's a little flaky but it works. :)

>I do suspect binary logging isn't the only reason hkhanna dumped Arch for Debian. It smells of nitpickery to me

You are right, of course. Binary logging was not the only reason I ultimately switched. The big reason was that I needed a distribution that did not change so frequently (although I love what Arch is doing). Stability was so important for my use case, though, that I ended up switching to Debian Stable.

Although journalctl does work well, I admit, I wonder if the advantages could have been accomplished without the binary file format. So many people have homegrown log parsing tools that need to be re-written to handle a binary file format. Or at least re-written to call journalctl as an intermediary.

> The big reason was that I needed a distribution that did not change so frequently

Yeah, this is a perfectly valid reason and also why I tend not to use Arch on servers (as much as I would like to otherwise, though...). Both have their use cases, and I love Arch for desktops, but it's not a particularly great match for situations where rapid, near-constant change is unwarranted, unnecessary, or problematic. Arch is also a distro that requires a fair amount of love and care--failure to do so, as a friend of mine learned last year when I had to help him update a year old install, can result in an upgrade path that's nearly impossible. Fortunately, ARM was still up at the time.

There certainly is something to be said for stable distros with a strict upgrade path.

> I wonder if the advantages could have been accomplished without the binary file format.

I don't know. I'll be honest: I can't think of a time when I've actually made use of the filtering features of journalctl and systemd's binary format. Nearly all circumstances that have required me to examine the log are usually done by observing the last few entries or manual searches. To that extent, I'd agree that it bears a certain sense of overkill.

That said, it is still possible to feed syslog-ng entries from systemd. So I can't think of a reason to fret much over specific tooling or the likes that expects text logs.

IMO, the udev/systemd relationship, repo, etc. is probably a much more valid complaint, especially for distributions that aren't planning on using systemd in the near future as it likely has some impact on developer/maintainer workload. Although from a sysadmin perspective, the benefits of systemd are very real and material, at least in my experience.

Service introspection could probably use some work to simplify, so that's another complaint. I can never remember the commands half the time. ;)

I am not a fan of binary logs, either.

I am a fan of more consistenly structured/filterable logs that are easier to pin to their service. I'm an even bigger fan of the fact systemd goes to a more Unixy model of not making every daemon author re-invent the logging wheel for their double-forking daemon.

Bull, we've had a syslog facility for ages.

Which would be why pretty much ever web server and mail server choose to implement their own logging mechanism, right?

Or perhaps its that having to bundle your logs together into a comparative handful of categories and then sort through them is less than entirely optimal.

Daemons implementing their own logging in addition to syslog is a feature, not a bug. With journald, even more so.

My understanding is that for a number of services (postgresql comes to mind) the issue was that syslog isn't implemented on all platforms, notably Windows.

I too was put off at first when Arch moved from rc.conf to systemd as I found rc.conf simpler to use, though I trust there was a good reason to move as they are pretty good at managing the distribution.

Rachel mentioned why binary logging makes sense for complicated logs that require rather tricky state machines/regex. Apache for example, in her case.

http://rachelbythebay.com/w/2012/02/10/logs/

Reply on news.ycombinator.com