Hacker News Reader

Comment by ultramancool | Hacker News Reader

Am I the only one who's disgusted with this bloated, convoluted, dbus-dependent pile of crap? I mean, c'mon, binary log files? I'll pass, thanks. It replaces way more than it needed to.

I prefer the BSD-style philosophy, nice, simple rc.conf, used to run Arch till it got infected with this garbage too. It slowly progressed away from it's BSD-style roots. So recently, I just gave up and moved to FreeBSD. Not a single regret so far.

View on HN

I think I'm mostly fine with journald (as a concept). At least I can explain reasoning for it to myself. A switch from non-structured to structured data provides a significant advantage, and indexes are useful. At least I had too many times grepping a multi-gigabyte log file. Sure, relatively modern (RFC5424) syslog protocol has structured data too, but in my experience most software had never bothered to use it. So, forcing a switch through introducing another protocol that has structured data baked-in isn't a too terrible idea.

My only issue with journald's binary log files is that they're of homebrewn custom format, that's not accessible by any standard means. Plain text files aren't directly readable by humans, too, but we have cat, less or alike tools to pass such data to terminal (sometimes an iconv is required, say, if log entries contain filenames that has characters outside of ASCII range), and those tools are available on every modern OS out there.

Personally, I think, a compromise that'd satisfy me (YMMV) would be either an industry-standard log format (like, maybe, sqlite - it's fairly simple, universal and omnipresent nowadays) or, even better, storing data in text files, but having accompanying binary index and metadata ones that store non-human-readable stuff (like hash chain - bet, no sane human would ever check cryptographic log integrity by hand) and provide additional information for faster machine access.

But the heck journald is a tightly-coupled part of into systemd instead of being a separate project is beyond me. I can't reject that systemd has some good things about it too, but it's too terribly monolithic and unhackable compared to mostly-scripted init systems. And such negative points easily outweigh the positive ones.

The biggest problem with dropping the syslog protocol altogether is that there's a huge amount of other stuff that speaks syslog. Things like networking gear use it for the same purposes as a normal *nix box, and getting them to switch is going to be like pulling teeth. With your particular case, I'd say you should look at rsyslog and/or syslog-ng. Both of them have backends that talk with actual databases, so you can have all your tools readily available, and can additionally dump to plain text and/or email messages at the same time. As to the why for journald, it seems very much like the rest of Poettering's MO of NIH. He doesn't seem too capable of working with other project makers to get his goals handled, so just does everything himself, to the detriment of the overall community.

Well, journald has syslog compatibility layer and can talk syslog, so supporting any existing software is not an issue. It doesn't speak syslog by itself, so it probably can't forward logs to another networked syslog server, but I don't see anything that prevents implementing this, if necessary.

The point is, journald also introduces a new protocol that's oriented at logging structured data. This way it not just provides a feature, but forces developers to think about structuring their log output in a machine-readable manner. I think that's the excuse that I believe is the journald's raison d'être and that I personally accept.

Just my opinion, though.

The problem with the journal format is that Poettering used a completely new format for doing so, instead of one of the many existing formats out there that have an excellent design for such things. He could have used something like berkeley db, or sqlite, which have been used extensively to store machine-readable data, are fault-tolerant, and small memory footprints, and would have let developers and administrators use the many pre-existing tools that exist for both database types for log analysis.

However, my philosophical problem is that there's no escape from it. I'd be perfectly happy with it existing if there was a way to turn it off, and let me use my own syslogd program in peace. Instead, I have yet another binary on my system that's running, with all of the problems that can bring, wasting cycles while I hand off log data for actual processing.

Compatibility isn't the problem. The problem is the use of structured binary data itself for logs. Logging binary structures instead of raw text makes it very difficult to recover from crashes or misbehavior, since the logging facility (journald or otherwise) must ensure that the log's structure on disk is consistent at all times.

I've seen more than my fair share of journald corrupting its own log due to unclean shutdowns. If I'm going to be grepping the journald log file anyway to reconstruct it (possible, but not easy, since journalctl is useless here), then why bother using it at all? It fails at the very task it was built for.

> journald corrupting its own log due to unclean shutdowns

Exactly! What the binary log advocates seem to be missing is that those unclean shutdowns (often called "crashes") are probably the very thing you are going to want to search for the in log. In general, few people care how (or if) log stores that the cron daemon yet again ran the hourly maintenance without any errors. What everybody who has had to search a system log cares about is "what happened right before the crash happened".

The current data that needs to be committed to the log successfully and immediately almost by definition happens at a time when you do not have the time for the complexity of an atomic addition to a database. Often there is barely have time to any disk write at all.

The only way make the system log useful would be to make adding events synchronous. As nobody wants to deal with a syslog that is 10,000x slower (or worse), the only sane option is what we always did - make the writes simple and immediate, and defer any fancier feature.

Have they never heard of "log parsers" before? If you want it in a search able DB (which can be very useful), you do that from original log either as an async daemon or defer it with cron or similar.

Not sure it was built for the task we expect a log facility to fulfill. I mean the corruption issue is a built in feature of journald: https://bugs.freedesktop.org/show_bug.cgi?id=64116

I think I disagree.

Classic plain text log files are structured too - they're files of '\n'-separated records, without much else to it. It doesn't really matter (integrity-wise) whenever one's writing, say, JSON or mostly-unstructured plain English records.

I'm unaware on particular journald internal implementation quirks and issues. Maybe it's badly coded and has lots of bugs that corrupt data. That would be implementation issue. But the overall idea of using "binary" logs isn't that bad to me.

It seems highly disturbing to me how much push he seems to have obtained by writing what have in my experience been complicated and crash-prone tools. Especially when such tools seem essentially in direct conflict with system philosophy.

I'm just glad there are sane options available still.

>Sure, relatively modern (RFC5424) syslog protocol has structured data too, but in my experience most software had never bothered to use it. So, forcing a switch through introducing another protocol that has structured data baked-in isn't a too terrible idea.

I agree. There is a strong need for common interface(s), and that's a strong part of the motivation behind Fluentd/Kafka/etc. www.fluentd.org/blog/unified-logging-layer

sqlite (or other "real databases") need a fsync for every or at least many transaction to be data safe. Without that it is far more prone to losing data than journald, because with a write ahead log the window for losing data is far bigger.

And a journal that has another journal inside it would be somewhat silly. A simple write log can be done better.

I don't think fsync on every log commit is a good idea. This would be more a DOS attack.

That said I'm somewhat troubled by the cavalier attitude to log data safety in journald too.

I have some difficulty figuring out the use-cases for journald over (some) plainish text log files. Our disks are getting faster and bigger. If you need to consolidate logs you have many options (starting with rsyslogd). If you need to log gigs of data, that can't possibly be kernel/initd logs?! And if it's application logs why not pipe it through something for indexing? By definition if i need kernel/local/init log files it's because something went very wrong: in such cases problems with clean shutdown/startup seems very likely - and in such use - cases plain text to file/serial wins hands down.

I'm not against having a wrapper that magically slurps stderr/stdout to timestamped logs -- but if that can't be written cleanly with the apis we have, then surely what we need is to make the minimal improvement in our (probably kernel) api to make writing such a program a trivial exercise? Nothing I've seen of systemd has me convinced the project cares one whit about finding the simplest, least coupled solution to any problem.

Systemd is more like busybox than sysvinit. People bitch about Systemd because they're not foul. Systemd folks (mostly Redhat) have their agenda. I don't mind that. Heck I even agree on some of it. But pushing the whole bloated thing down nearly everyone's throat disguising it as just a simple init replacement is a bit too much.

> dbus-dependent pile of crap?

This seems to come up frequently. I'm curious: how alternatively would a local process communicate with the Init daemon?

I'm curious:

Why is the init daemon what a local process should communicate with?

A daemon that managaes other daemons doesn't need to be PID 1, even to reap zombies.

Secondly, daemons shouldn't care what process is managing them, a principled approach to communication between the daemon manager and a daemon would probably include handing off sockets/ports/fds, but probably not much else.

>A daemon that managaes other daemons doesn't need to be PID 1, even to reap zombies.

And get this, prctl(PR_SET_CHILD_SUBREAPER) has existed since May 2012, the original patch was created and submitted by Poettering, and yet we're still told that service management needs to run as pid1 in order to see all double-forked detached daemonized processes.

> Why is the init daemon what a local process should communicate with?

Because there could, and I know this is a crazy thought, be some benefit in having more meaningful information flow between the master of processes and the processes it manages?

> how alternatively [to dbus] would a local process communicate with the Init daemon?

The answer to that is easy: It's still signals, like it has been for decades, and systemd doesn't change that any more than anything else does. systemd defines some more signals, but that's about it.

Notice that you said init daemon. The errors here are in thinking that (a) dbus is the communications system, even in systemd, for the part of the system that does overall system state management and the stuff that the kernel requires of process #1; and (b) the part of the system that supervises daemons is in all packages run in process #1, the "init daemon".

systemd, the package, uses dbus in a number of places between a number of components, most notably logind, hostnamed, timedated, machined, and localed. But don't get that confused with systemd the program that runs as process #1, which is as constrained by the kernel (and others) into using the same signals as always, just as any other system manager is. There's an AF_LOCAL socket (/run/systemd/private) for systemctl to talk to PID #1 using a private undocumented protocol, and a public documented D-Bus API for units, but those are for the service management part of systemd the program.

The system manager and service manager in the nosh package are in the same boat. The system manager's API is that same set of signals again (augmented with some of the extra systemd-defined signals that fit the model). There's no other API because the system manager is not what the world talks to about service management. The control/status API for individual services is the filesystem, the service manager (that is not in PID 1) presenting the same suite of pipes and files as daemontools-encore. And there's an AF_LOCAL socket for system-control (and indeed anything else, such as service-dt-scanner a.k.a. svscan) to talk to the service manager for loading and unloading service bundles in the first place, and plumbing them together.

named pipes, posix ipc, signals, etc. There are quite a few existing communications methods that can be used for this purpose.

OK, and how do you format the data you send across, or find out which functions the daemon supports and trigger them?

Classic sysvinit or BSD-style init doesn't rely upon this sort of functionality at all. Just some simple scripts. No need to complicate it any further. A few signals are all that are needed.

Simple scripts? Have you looked at the shit in /etc/init.d in a modern linux distribution?

  $ wc -l * | sort -n | tail
   274 exim4
   286 apache2
   290 dnsmasq
   298 nfs-common
   350 clamav-freshclam
   364 udev
   386 checkroot.sh
   420 clamav-daemon
   465 clamav-milter
  9893 total

If you look into these, you'll find tons of near-duplicate code between scripts, and frequently every script reinvents the wheel in one way or another.

Using sysvinit doesn't require a domain specific configuration language to be known and understood. So I think simpler is quite valid.

Less verbose does _not_ mean simpler! In many cases it means quite the opposite.

UNIX to me is about simplicity. We don't need crap like binary logs and heavy RPC mechanisms to be polluting beautifully simple and minimal systems.

As several others have noted, the code duplication issue is solved in FreeBSD's init(8) with rc.subr.

  142 mysql
  142 sysfs
  157 kexec
  172 udev
  172 xdm
  182 apache2
  195 bootmisc
  232 named
  297 dmcrypt
  736 net.lo

This is on an OpenRC Gentoo system. There's very little duplicated code in the init scripts.

You might well complain that many these init scripts are substantially longer than the equivalent systemd unit files. Your complaint would be valid. Thing is, many of these init scripts do so much more than the equivalent systemd unit files. Ferinstance, the postgres and mysql unit files that I've seen permit no user configuration (like, such as, altering daemon listen port, config file location, and the like). [0] They also don't do any sort of housekeeping such as verification of the validity of the service's configuration file, checking and repairing mode and ownership of the same, and verifying the existence of the service's data directory.

I understand that OpenRC wasn't being considered for Debian Jessie, but it does a lot of things right, and is (IMO) head-and-shoulders above SysV init. (But then, isn't even bringing up SysV init kind of beating on a dead horse? We all agree that it really needs improvement.)

If you're interested, Gentoo's apache2 init script is here: http://pastebin.ca/2845519 . For reference, the apache2 systemd service file is here: http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/www-... .

For a look at a simpler init script, check out the script for dnsmasq: http://pastebin.ca/2845520 . Looks pretty simple, no? At least as simple as the systemd service files we've been seeing, yes? The config file for dnsmasq is a single line: DNSMASQ_OPTS="--user=dnsmasq --group=dnsmasq" .

OpenRC provides complexity when you need it, and gets out of your way when you don't.

[0] I don't have systemd installed, so I might be missing the user configuration facilities. If they exist, and the configuration files have any appreciable complexity, then their line count must be counted against the unit file's line length.

Yeah, I've got OpenRC on one of my Funtoo systems too... Let's try some different shell magic...

~ # for file in `equery files openrc`; do [ -f $file ] && echo $file; done | xargs wc | tail -1 23729 196573 7432232 total

Oh, and that's with lzma compressed man pages.

If duplicate code bothers you this much, then what stops you from abstracting the common routines into something like BSD's /etc/rc.subr, and then sourcing them in the service-specific init script?

You see crap. I see time-polished scripts.

I also suspect you're running RH. Under Debian script lengths are typically quite short:

n: 104 sum: 13553 min: 8 max: 1246 mean: 130.317308 median: 99 sd: 138.595750

That outlier, by the way, is xprint, part of CUPS. Never had to touch it myself.

Quite a few of those lines are comments, and the basic structure is a set of start / stop / restart blocks.

The comment makes more sense if you substitute "familiar" for "simple," a tactic I use a lot while reading arguments like these.

And on my Debian testing system with systemd, the longest service definition is 46 lines! (/lib/systemd/system/getty@.service)

Yes, modern linux and sysvinit are pigs compared to some alternatives (though systemd is worse). The choice isn't a binary choice between sysvinit and systemd. How about this? init + rc + all the scripts combined are smaller than your handful of daemon-specific scripts. And I understand exactly how all of this works (the kernel parts included). I can easily change any part of it, I can easily debug any part of it, I can easily extend any of it if I need special features. With systems that comprise hundreds of thousands of lines of code, just beginning to understand how all of it works together takes much more time...

Simplicity really does buy me something.

    $ wc -l /usr/src/sbin/init/init.c
        1450 /usr/src/sbin/init/init.c

    $ wc -l /etc/rc
        537 /etc/rc

    $ wc -l /etc/rc.d/*

      21 /etc/rc.d/amd
      11 /etc/rc.d/apmd
      21 /etc/rc.d/avahi_daemon
      21 /etc/rc.d/avahi_dnsconfd
      11 /etc/rc.d/bgpd
      15 /etc/rc.d/bootparamd
       9 /etc/rc.d/cron
      14 /etc/rc.d/cvsyncd
      16 /etc/rc.d/dbus_daemon
      15 /etc/rc.d/dhcpd
      11 /etc/rc.d/dhcrelay
      12 /etc/rc.d/dvmrpd
      11 /etc/rc.d/ftpd
      11 /etc/rc.d/ftpproxy
       9 /etc/rc.d/hostapd
       9 /etc/rc.d/hotplugd
      11 /etc/rc.d/httpd
      13 /etc/rc.d/identd
       9 /etc/rc.d/ifstated
      17 /etc/rc.d/iked
       9 /etc/rc.d/inetd
      17 /etc/rc.d/isakmpd
      17 /etc/rc.d/iscsid
      11 /etc/rc.d/ldapd
      15 /etc/rc.d/ldattach
      11 /etc/rc.d/ldomd
      11 /etc/rc.d/ldpd
      11 /etc/rc.d/lockd
       9 /etc/rc.d/lpd
      16 /etc/rc.d/mopd
      17 /etc/rc.d/mountd
       9 /etc/rc.d/mrouted
      18 /etc/rc.d/nfsd
      11 /etc/rc.d/npppd
      38 /etc/rc.d/nsd
      12 /etc/rc.d/ntpd
      11 /etc/rc.d/ospf6d
      11 /etc/rc.d/ospfd
      24 /etc/rc.d/pflogd
      11 /etc/rc.d/popa3d
      11 /etc/rc.d/portmap
      16 /etc/rc.d/rarpd
       9 /etc/rc.d/rbootd
     289 /etc/rc.d/rc.subr
      11 /etc/rc.d/relayd
      11 /etc/rc.d/ripd
       9 /etc/rc.d/route6d
      11 /etc/rc.d/rsyncd
      11 /etc/rc.d/rtadvd
      11 /etc/rc.d/rtsold
       9 /etc/rc.d/rwhod
      11 /etc/rc.d/sasyncd
      13 /etc/rc.d/sendmail
       9 /etc/rc.d/sensorsd
      11 /etc/rc.d/slowcgi
      13 /etc/rc.d/smtpd
      11 /etc/rc.d/sndiod
      12 /etc/rc.d/snmpd
      26 /etc/rc.d/spamd
      25 /etc/rc.d/spamlogd
      13 /etc/rc.d/sshd
      11 /etc/rc.d/statd
      15 /etc/rc.d/syslogd
      12 /etc/rc.d/tftpd
      11 /etc/rc.d/tftpproxy
       9 /etc/rc.d/tor
      32 /etc/rc.d/unbound
       9 /etc/rc.d/watchdogd
      11 /etc/rc.d/wsmoused
       9 /etc/rc.d/xdm
      16 /etc/rc.d/ypbind
      11 /etc/rc.d/ypldap
      28 /etc/rc.d/yppasswdd
      13 /etc/rc.d/ypserv
    1285 total

I think you also need to count the Bash source code there.

That's not really fair since the shell is an independent part of the system and could be swapped out for another implementation. Plus, it's not there just for the init system; it'd be there regardless of the init system. It's an already existing independent component, which is used for leverage.

Otherwise should we also count the C compiler, libc, all the CLI utilities and the kernel?

But here you go. Still smaller than systemd. Sysvinit needs a shell too...

    $ wc -l /usr/src/bin/ksh/*.[ch]
     127 /usr/src/bin/ksh/alloc.c
    1412 /usr/src/bin/ksh/c_ksh.c
     935 /usr/src/bin/ksh/c_sh.c
     560 /usr/src/bin/ksh/c_test.c
      53 /usr/src/bin/ksh/c_test.h
     201 /usr/src/bin/ksh/c_ulimit.c
      60 /usr/src/bin/ksh/config.h
     829 /usr/src/bin/ksh/edit.c
      86 /usr/src/bin/ksh/edit.h
    2163 /usr/src/bin/ksh/emacs.c
    1333 /usr/src/bin/ksh/eval.c
    1433 /usr/src/bin/ksh/exec.c
     107 /usr/src/bin/ksh/expand.h
     593 /usr/src/bin/ksh/expr.c
     984 /usr/src/bin/ksh/history.c
     438 /usr/src/bin/ksh/io.c
    1653 /usr/src/bin/ksh/jobs.c
      13 /usr/src/bin/ksh/ksh_limval.h
    1643 /usr/src/bin/ksh/lex.c
     132 /usr/src/bin/ksh/lex.h
     196 /usr/src/bin/ksh/mail.c
     786 /usr/src/bin/ksh/main.c
    1149 /usr/src/bin/ksh/misc.c
      91 /usr/src/bin/ksh/mknod.c
     285 /usr/src/bin/ksh/path.c
     267 /usr/src/bin/ksh/proto.h
     420 /usr/src/bin/ksh/sh.h
    1161 /usr/src/bin/ksh/shf.c
      82 /usr/src/bin/ksh/shf.h
     897 /usr/src/bin/ksh/syn.c
     231 /usr/src/bin/ksh/table.c
     183 /usr/src/bin/ksh/table.h
     420 /usr/src/bin/ksh/trap.c
     708 /usr/src/bin/ksh/tree.c
     141 /usr/src/bin/ksh/tree.h
      57 /usr/src/bin/ksh/tty.c
      37 /usr/src/bin/ksh/tty.h
    1210 /usr/src/bin/ksh/var.c
      10 /usr/src/bin/ksh/version.c
    2128 /usr/src/bin/ksh/vi.c
   25214 total

You cannot assume that /bin/sh always refers to Bash. There's a lot more to Unix than just Linux and Mac OS X. On FreeBSD /bin/sh is a 1989 rewrite of the SVR4 Bourne shell. As of 10.0-RELEASE it comes to 21167 lines, including its Makefile, ancillary shell scripts, and documentation.

What's wrong with old good Unix pipes?

The don't solve race conditions in peers trying to locate each other (surprisingly difficult). They don't solve a standardized marshaling format. They don't come with an implementation to integrate with main loops for event polling. They don't handle authentication (well, sort of). They have an inherent vulnerability in FD passing where you can cause the peer to lock up. You can get into deadlock situations in your messaging code if you aren't really careful about message sizes and when you order poll in/poll out. They aren't introspect-able to see what the peer supports. They make it super easy to not maintain ABI.

I could go on.

You seem to want pipes to bend over backwards to solve each and every one of your applications' problems.

> The don't solve race conditions in peers trying to locate each other (surprisingly difficult).

Not even sure what you mean here. Are you talking about peer discovery? Because DBus won't help you there either--peers have to be aware of each other's DBus object paths before they can rendezvous. Similarly, two peers need to know where the common pipe is to rendezvous.

> They don't solve a standardized marshaling format.

Nor should they. There are a ton of ways to skin this cat in userspace, depending on what your application needs. Protobufs come to mind, for example, but there are others.

Why do you want the pipe to enforce a particular marshaling format? Does the pipe know what's best for every single application that will ever use it?

> They don't come with an implementation to integrate with main loops for event polling.

It's not the kernel's responsibility to implement the application's main loop. That's what libevent and friends are for today, if you need them.

> They have an inherent vulnerability in FD passing where you can cause the peer to lock up.

Last I checked, you pass file descriptors via UNIX sockets, not pipes.

> They don't handle authentication (well, sort of).

Depends on your application's threat model. The kernel provides some basic primitives that can be used to address common security-related problems (capabilities, permission bits, users, groups, and ACLs). If they're not enough, you're free to perform whatever authentication you need in userspace to secure your application against your threat model's adversaries.

It is unreasonable to expect the pipe to be aware of every single threat model an applications expects, especially since they change over time.

> You can get into deadlock situations in your messaging code if you aren't really careful about message sizes and when you order poll in/poll out.

It's not the pipe's fault if you don't use it correctly.

> They aren't introspect-able to see what the peer supports.

Peer A could use the pipe to ask peer B what it can do for peer A. Why do you want the pipe to do peer B's job?

> They make it super easy to not maintain ABI.

Nor does DBus. Nothing stops an application from willy-nilly changing the data it serves back.

> Not even sure what you mean here. Are you talking about peer discovery? Because DBus won't help you there either--peers have to be aware of each other's DBus object paths before they can rendezvous. Similarly, two peers need to know where the common pipe is to rendezvous.

I suspect he was referring to socket activation and how that simplifies these kinds of messes.

> Nor should they. There are a ton of ways to skin this cat in userspace, depending on what your application needs. Protobufs come to mind, for example, but there are others.

Right... so that's exactly what systemd did. It used Dbus, which provides that standard serialization format. Not my favourite format, but very well established and tested and focused on systemd's problem domain.

The point is, in order to have loose coupling between components, something like unix pipes is just a starting point.

> Does the pipe know what's best for every single application that will ever use it?

Ah, now I understand the problem with systemd. I never realized it was trying to take over every application's communications protocol! ; -)

Seriously, I think it is perfectly reasonable (and necessary) to define a standard protocol for system even notifications... I say this because it has already been done... by the standard compoents like udev & dbus that systemd is building on top of...

> Last I checked, you pass file descriptors via UNIX sockets, not pipes.

Correct. People have a tendency to mess up their semantics though. If the original poster wasn't referring to unix domain sockets, than it is an even sillier question.

> It is unreasonable to expect the pipe to be aware of every single threat model an applications expects, especially since they change over time.

Yes, but you do need something more sophisticated than a pipe to manage secure communications between your systems components.

> Nor does DBus. Nothing stops an application from willy-nilly changing the data it serves back.

? D-Bus will drop you like a hot potato the moment you fire off invalid messages. You could send valid messages with fraudlent/misleading data payloads I guess, but at least a whole host of problems are addressed by tightening that up.

> The point is, in order to have loose coupling between components, something like unix pipes is just a starting point.

It's also an ending point. If each application gets to define its own IPC primitives, then there become as many app-to-app communication protocols as there are app-to-app pairs. This does not make for a loosely-coupled ecosystem.

> Seriously, I think it is perfectly reasonable (and necessary) to define a standard protocol for system even notifications... I say this because it has already been done... by the standard compoents like udev & dbus that systemd is building on top of...

This is circular reasoning. You're saying "we should use systemd's notification protocol, because systemd uses it." This says nothing about the technical merits of its protocol.

> Yes, but you do need something more sophisticated than a pipe to manage secure communications between your systems components.

Um, data within a pipe is visible to only the endpoint processes, the root user (via procfs and /dev/kmem), and the kernel. If you don't trust an endpoint, you should stop communicating with it. If you don't trust the root user or the kernel, you can't really do anything securely at all in the first place. My point is, data within a pipe is about as secure as it's going to get.

> You could send valid messages with fraudlent/misleading data payloads I guess, but at least a whole host of problems are addressed by tightening that up.

I think you will find that the bulk of IPC problems will come from dealing with data you didn't expect--that is, processes sending "fraudlent/misleading" data and other processes acting on it. DBus won't help you there--you always always ALWAYS have to validate data you receive from untrusted parties, no matter what transport or wire format you're using.

> t's also an ending point. If each application gets to define its own IPC primitives, then there become as many app-to-app communication protocols as there are app-to-app pairs. This does not make for a loosely-coupled ecosystem.

That's exactly why you need a more systemic approach to the IPC mechanism...

You can pretend that "oh this is just a stream so there isn't tight coupling", but the information that is communicated is the same. If you haven't imposed some structure and consistency to it, that's exactly how you end up with a ball of mud.

> This is circular reasoning. You're saying "we should use systemd's notification protocol, because systemd uses it." This says nothing about the technical merits of its protocol.

You misunderstood my point. I'm not justifying it on the basis that systemd is using it. I'm saying the fact that all the other systems have arrived at a similar, and in many cases the exact same mechanism, is pretty strong evidence that it is a reasonable design choice.

Basically all the Linux systems out there are already using udev & dbus. Most of the non-Linux systems do as well. Everyone's done it and made it work. That systemd is adopting arguably the most entrenched one in the Linux sphere is hardly as controversial as people seem to think it is.

> My point is, data within a pipe is about as secure as it's going to get.

I wasn't trying to suggest it wasn't a secure point-to-point communication mechanism (it has issues, but fine enough). The issue is that you need more integration in with the security model to avoid having a rats nest of security logic on top of it.

> I think you will find that the bulk of IPC problems will come from dealing with data you didn't expect--that is, processes sending "fraudlent/misleading" data and other processes acting on it.

There's a very long and glorious history of malformed and misleading IPC causing problems. Not that it is the only thing, but life becomes a lot easier when that problem is off the table.

> DBus won't help you there--you always always ALWAYS have to validate data you receive from untrusted parties, no matter what transport or wire format you're using.

Yes you will. However, DBus ensures that you don't have to write a ton of redundant code just verifying you are getting validly structured data and dealing with the nasty ways someone might try to exploit that.

Imagine having to write a secure REST service where the only thing you had to worry about in the entire network protocol stack was the validity of the data expressed in the payload.

> That's exactly why you need a more systemic approach to the IPC mechanism... You can pretend that "oh this is just a stream so there isn't tight coupling", but the information that is communicated is the same. If you haven't imposed some structure and consistency to it, that's exactly how you end up with a ball of mud.

So, you basically want to turn IPC into CORBA. It's a bad idea to have the OS impose too much structure on your IPC, in the same way that it's a bad idea to have the base class in an object hierarchy try to take on too many subclass-specific responsibilities. This is because over-specialization of a component needlessly constrains the designs of systems that use it.

That said, you are correct in that byte streams alone do not make for loosely coupled systems. Programs must additionally emit data such that other unrelated programs can operate on it without modification. But we already have this universally-parsable data format: it's called human-readable text. It's why you can "grep" and "awk" and "sed" the outputs of "ls" and "find" and "cat", for example.

Take a second and imagine what the world would be like if you had to write "grep" such that it had to be specifically designed to interact with "find," instead of simply expecting a stream of human-readable text. Imagine if "awk" had to be specifically designed to interact with "ls." This is the world that CORBA-like IPC creates, where programs not only need to be intrinsically aware of the higher-level RPC methods each other program exposes, but also intrinsically aware of the access and consistency semantics that go along with it. No thank you; I'll stick with pipes and human-readable text, where the data format, data access, and consistency semantics are universally applicable.

> Basically all the Linux systems out there are already using udev & dbus. Most of the non-Linux systems do as well. Everyone's done it and made it work. That systemd is adopting arguably the most entrenched one in the Linux sphere is hardly as controversial as people seem to think it is.

First, udev is Linux-specific--it uses netlink sockets to listen for Linux-specific hardware events. Second, not everyone uses udev and dbus. mdev, smdev, eudev, and static /dev are also widely used and have well-defined use-cases that udev does not serve, and plenty of servers (and even my laptop) get along just fine without dbus.

Trying to justify udev and dbus because "everyone uses them so you should too!" is not only an example of the bandwagon logical fallacy, but also reveals your ignorance of and insensitivity to other users' requirements.

> The issue is that you need more integration in with the security model to avoid having a rats nest of security logic on top of it.

I said it above and I'll say it again here. The IPC layer does not know and cannot anticipate the security needs of every single application. If you try to design your IPC system to do this, you will fail to encompass every possible case. This is because threat models are not only specific to the application, but also specific to the context in which the application runs.

For example, you do not send your bank account password over an out-bound socket unless it has first been encrypted using a secret key known only to you and your bank. Your reasoning implies that the IPC system should be tasked with automatically enforcing this constraint, among others. Nevermind the fact that the IPC system will see only the ciphertext and thus will not know that the data it's about to send contains your password.

> However, DBus ensures that you don't have to write a ton of redundant code just verifying you are getting validly structured data and dealing with the nasty ways someone might try to exploit that.

So do plenty of stub RPC compilers and serialization libraries that have been around longer, are more widely used, and are better tested than DBus. However, neither DBus nor any of these solutions will help you with well-formatted but invalid data. Your application has to deal with that, since the validity of data is both application-specific and context-specific (so, not something the IPC system can anticipate).

Again, what's so special about DBus, besides the fact that it's the New Shiny?

> Imagine having to write a secure REST service where the only thing you had to worry about in the entire network protocol stack was the validity of the data expressed in the payload.

If I'm writing a secure service of any kind, you can be damn certain I'm thinking about a LOT more than input validation! Security encompasses waaaaaaaaaay more than that.

Even if security wasn't an issue, there is still a LOT more to worry about than input validation. Things like scalability, fault tolerance, concurrency, and consistency come to mind. There is no silver bullet for any of these, let alone an IPC system that solves them all at once!

> So, you basically want to turn IPC into CORBA.

No, very much not, because we don't really need an RPC mechanism here. We want something in the event/messaging space.

But it isn't even a want. If you don't have it, you end up with each of the your components actually being very tightly coupled to all the other components it talks to and you've got a truly monolithic mess on your hands.

> It's a bad idea to have the OS impose too much structure on your IPC, in the same way that it's a bad idea to have the base class in an object hierarchy try to take on too many subclass-specific responsibilities. This is because over-specialization of a component needlessly constrains the designs of systems that use it.

This isn't exactly a new concept or a new problem. There are plenty of existing cases where this is happening (basically every platform I can think of right this moment, though I'm sure there are plenty of exceptions), including in the current Linux udev mechanism.

> But we already have this universally-parsable data format: it's called human-readable text. It's why you can "grep" and "awk" and "sed" the outputs of "ls" and "find" and "cat", for example.

/me falls out of chair.

Yeah, that's worked out great. Never had a problem with init scripts not extracting the right column or handling a new variant in how the output comes out (or even better still, the dreaded "value with embedded whitespace").

But you know what? DBus is basically human readible with a bit more imposed structure than generic streams. So, I think you are arguing in support of the systemd approach without realizing it! ;-)

> First, udev is Linux-specific--it uses netlink sockets to listen for Linux-specific hardware events. Second, not everyone uses udev and dbus. mdev, smdev, eudev, and static /dev are also widely used and have well-defined use-cases that udev does not serve, and plenty of servers (and even my laptop) get along just fine without dbus.

Very true. I was speaking in generalities. Point being, udev is out there and very thoroughly established as something that people seem to generally want.

> The IPC layer does not know and cannot anticipate the security needs of every single application. If you try to design your IPC system to do this, you will fail to encompass every possible case.

You have some ambitious notions for the systemd project that go well beyond the goals that critics say are overly broad in scope. This is for addressing a relatively narrow set of problems that wouldn't even come close to defining 1% of IPC on a LInux system. I'm not suggesting we replace the entire Unix toolset with a complete new set of interfaces and programs (nor are the systemd guys). This is specifically for managing the interactions between devices & daemons... It's a well established problem domain with some well established roles & responsibilities and some pretty well understood data message/event structures.

While it might use systemd/dbus/whatever to get notifications about various services and system events, YOUR BANKING SOFTWARE IS NOT SUPPOSED TO USE SYSTEMD TO MOVE MONEY BETWEEN YOUR ACCOUNTS!

> Again, what's so special about DBus, besides the fact that it's the New Shiny?

DBus isn't the new shiny. It's the old shiny. The new shiny would probably be 0mq or some of the new datagram protocols that people are experimenting with, along with various extensible binary protocols like MessagePack and Cap'nProto.

What's special about DBus is that it is already being used very broadly on Unix platforms for this kind of function and is well integrated in to the system security model. The one bit of additional coolness it brings to the table is the support for socket activation, which simplifies the complexity of start ordering and discovery tremendously, which is indeed a VERY nice benefit, but could no doubt have been NIH'd independently.

> If I'm writing a secure service of any kind, you can be damn certain I'm thinking about a LOT more than input validation! Security encompasses waaaaaaaaaay more than that.

Yes, but the point is one derives substantial benefit from trusted components in the system that take care of their part of the problem. You don't benefit from having to reimplement an entire security apparatus with each component. This is basic security compartmentalization 101.

> Things like scalability, fault tolerance, concurrency, and consistency come to mind. There is no silver bullet for any of these, let alone an IPC system that solves them all at once!

You appear to be simultaneously claiming there is no silver bullet and being terribly upset that systemd isn't one.

Yes, it is no silver bullet. It's not even the huge sea change that some people seem to think it is. Rather, it is an incremental improvement over existing practices that gets rid of a bit more of the cruft and stupidity in the existing infrastructure. Doing that kind of thing right can really make a big difference for the system as a whole, but it isn't the apocalypse.

Somehow you think I'm talking about systemd and init scripts and the things they do. I'm not. The original question I replied to was about why pipes (or any OS-level IPC) shouldn't try to solve application-level problems.

My arguments are that the OS's IPC should not enforce an IPC record structure, but should enforce a consistent set of IPC access methods (i.e. pipes, sockets, shared memory, message queues, etc.) defined independently of applications. I think we're in agreement about the latter--if the OS were to let each application have it's own IPC access methods, then there would be as many access methods as there are applications (leading to tightly-coupled "truly monolithic mess").

I don't think we've reached agreement on the former. I claimed that there is no "best record structure" for all applications, so the OS shouldn't try to enforce one. I also mentioned that human-readable text is the universal data format, which is both a manifestation of this principle (i.e. the OS imposes no constraints on the structure of bytes passed between programs) and a desirable outcome since parsing text is super-simple to implement (by contrast, take a look at the examples in dbus-send(1) to see how painful the alternative can be). You disagree--you think the IPC system should also handle things like serialization and validation.

The problem is that serialization and validation are both application-specific (and even context-specific) concerns, and for the IPC system to address them, it has to gain knowledge from the application. But this lets the application set IPC access methods, which we've already agreed is a bad idea! My (extreme) example to prove this point was that pushing validation responsibility from the application into the IPC system would require it to handle ridiculous application-specific corner cases, like defining a socket class that makes sure that your bank account password won't be sent to the wrong host (still not sure how you concluded that that remark was about systemd). The point is, if you want your IPC system to handle validation for you, you're just asking for trouble.

The same type of problem occurs when you put serialization into the IPC system. The serializer has to know whether or not a string of bytes represents a valid application-defined record. If you make serialization the IPC system's responsibility, it needs application-level knowledge on whether or not an inbound message represents a valid message (which also leads to ridiculous corner cases).

DBus not only enforces structured records (bad), but also lets applications define their own IPC access methods (worse). The RPC-like nature of DBus means that both peers must not only agree on the interpretation of bytes in advance, but also agree on the semantics of accessing them. Unlike reading from a pipe, accessing the value of a DBus object by name can have arbitrary side-effects which the requester must be aware of. In the limit, this puts us into the undesirable situation of having each application-to-application pair agree on an IPC access method, leading to the tight coupling nightmare.

Don't get me wrong--DBus has its use-cases. OS-level IPC isn't one of them. I wish systemd folks took some time to think about this, but they're too busy trying to make DBus into OS-level IPC with no regards to the consequences. See kdbus and the SOCK_BUS socket class it exports.

Now, nitpicks:

> But you know what? DBus is basically human readible with a bit more imposed structure than generic streams

/me falls out of chair too.

Now you're just being daft :) The more structure you impose on bytes, the less human-readable it gets. For example, I don't think I have to explain to you why this comment is more legible as rendered in your browser (unstructured text) than as raw HTML (structured records).

> DBus isn't the new shiny. It's the old shiny

CORBA is the old shiny ;) See also: https://en.wikipedia.org/wiki/Remote_procedure_call#Other_RP...

> The one bit of additional coolness it brings to the table is the support for socket activation

Not the IPC system's responsibility. See also: https://en.wikipedia.org/wiki/Xinetd

> You don't benefit from having to reimplement an entire security apparatus with each component

Of course--you use a library and an RPC stub generator for this. Not really part of the "design principles of IPC" discussion we've got going, though.

> The original question I replied to was about why pipes (or any OS-level IPC) shouldn't try to solve application-level problems.

That maybe what you read, but the context of drdaemon's statement was specifically in response to a question about communications with the init daemon, and of course everything I said after was as well... Glad we got that settled.

> Not the IPC system's responsibility.

Hmm... IPC systems need to have ways of matching up the parties in a conversation, and having one where you don't have to enforce who calls whom first and parties don't have to mutually agree upon the specific endpoints in advance sure seems like something an IPC system might want to have... particularly one employed in an init system...

As discussed here: http://0pointer.de/blog/projects/systemd.html

There absolutely is a ton of overlap between what systemd does with socket activation and what Xinetd has evolved to... but as with evenone else doing OS design, there comes a point where you leave Xinetd behind and let the full potential of that trick work in your favour.

> Now you're just being daft :)

Me and the folks at Wikipedia: http://en.wikipedia.org/wiki/Comparison_of_data_serializatio...

Don't get me wrong, I think a lot of the Wikipedians are pretty daft, but they are as reasonable a judge of human readability as I can imagine, given what they do.

> CORBA is the old shiny ;)

CORBA is the old shiny-my-god-we-dont-need-nearly-all-of-that-and-it-really-benefits-a-bootstrapped-systems-so-there-is-a-chicken-and-egg-prolem-here. But yeah, close. I don't think anyone has seriously considered that since the OS/2 & Workplace OS days... and even then.

That said, I would say that THESE DAYS (unlike in its heyday), CORBA is a pretty awesome robust, feature rich _general purpose_ distributed IPC system.

> Of course--you use a library and an RPC stub generator for this.

Ah, so it is much more modularized if it runs as an executable piece of code in process than a piece of executable code out of process. Got it. ;-)

> Not really part of the "design principles of IPC" discussion we've got going, though.

Well, that's the discussion you're having. I'm trying to talk about the design constraints and appropriate solutions for the problem domain...

> As discussed here: http://0pointer.de/blog/projects/systemd.html

Lennart Poetterring claims that you should use his software instead of someone else's software! I'm SHOCKED! Full story at 11.

Seriously now, did you honestly think that he would say to use xinetd over systemd? Do you honestly believe a developer will advocate the use of a competing piece of software over something (s)he produced?

> There absolutely is a ton of overlap between what systemd does with socket activation and what Xinetd has evolved to... but as with evenone else doing OS design, there comes a point where you leave Xinetd behind and let the full potential of that trick work in your favour.

Unless you don't feel like replacing small, simple, easy-to-use, well-tested xinetd with the 200K-line pile of C code that is systemd.

Besides, I've got your socket activation right here: Start the daemon, have the daemon open a port, and let the kernel swap it to disk. The kernel will swap it back in when it receives a connection for it.

Benefits:

* the daemon preserves state between "activations" for free

* the kernel gives you this feature for free

Security:

* the daemon doesn't have to trust another userspace program with anything

* the daemon can use mlock() to prevent sensitive pages from getting swapped

* if this isn't enough, you can encrypt the swap partition to resist offline attacks

Resources:

* If disk is too expensive, disk is read-only, you have no swap, you have no CAP_IPC_LOCK, the daemon would need to mlock() too much RAM, and you can't encrypt your swap, there's xinetd.

* Need to apply filters or QoS controls on connections before waking up the daemon? That's what the firewall is for.

Trivia:

* You can have xinetd trigger whatever event you want, since all it does is fire up a program and run it. This includes alerting other programs, like a service manager, that it got a connection, and maybe even sending along the message (or the file descriptor) if you want. There is no need for systemd to subsume this responsibility.

As you can see, "socket activation" is by and large a marketing gimmick.

> Me and the folks at Wikipedia:...

You think an article that compares data serialization protocols somehow proves your ludicrous claim that human readable text is less readable than marked-up text? Maybe daft was too nice a word...

> Ah, so it is much more modularized if it runs as an executable piece of code in process than a piece of executable code out of process. Got it. ;-)

Sir/madam, have you ever written an Internet-facing daemon? Obviously the bulk of the RPC logic lives in a shared library. You know, a logically distinct module that can be independently installed, loaded once, and independently maintained.

Besides, procedurally-generated RPC-handling code adds no technical debt to your project, anymore than the compiler's generated assembler output does.

You seem to want to replace the RPC shared library with a separate process. Not only will this make create a performance bottleneck, but also it makes it a single point of failure. If it crashes, all your daemons lose their connections. This is obviously highly undesirable, especially on servers.

> Well, that's the discussion you're having. I'm trying to talk about the design constraints and appropriate solutions for the problem domain...

I think I'm done with you. You deserve everything systemd will ever do for you.

> Seriously now, did you honestly think that he would say to use xinetd over systemd?

No... but I thought he might be able to pretty adequately explain how systemd exploits socket activation and contrast it with xientd....

> Do you honestly believe a developer will advocate the use of a competing piece of software over something (s)he produced?

Well, I've certainly done it, so it is possible, but I wasn't referencing him as a persuasive voice... Even if I was, that'd be such a flawed and pathetic argument...

> Unless you don't feel like replacing small, simple, easy-to-use, well-tested xinetd with the 200K-line pile of C code that is systemd.

You might want to look at the code. The socket activation logic is a pretty clean & tight ~90K chunk of code in a handful of files... and for the record, xinetd isn't that slim, with nearly 25K lines of code spead over well over a hundred files, and that's if you only count the C source files.

> As you can see, "socket activation" is by and large a marketing gimmick.

Sigh... I can see you didn't read the article. The implementation differences aren't terribly different, and Lennart already made your points for you... Systemd does have some little tweaks that open up a bunch of different worlds of advantages.

> Sir/madam, have you ever written an Internet-facing daemon?

Yes, but of course, in this context we're primarily focused on AF_UNIX sockets...

> Obviously the bulk of the RPC logic lives in a shared library. You know, a logically distinct module that can be independently installed, loaded once, and independently maintained.

it's very common, for example, for web apps to have a separate process that parses and validates inbound HTTP requests RESTful requests before passing them on to the main application process. You can and do run web apps that are directly exposed to the Internet, but nobody suggests this is to make the request processing logic more modular...

> You seem to want to replace the RPC shared library with a separate process. Not only will this make create a performance bottleneck, but also it makes it a single point of failure. If it crashes, all your daemons lose their connections. This is obviously highly undesirable, especially on servers.

I see you are familiar with Erlang. ;-)

You raise a good point. Often to reduce failure rates people employ load balancer that work with various HA protocols to avoid losing connetions. What do load balancers do again? Oh yeah, they are separate processes receive in bound RPC requests, parse and validate them, attempt to mitigate any in bound attacks before routing and forwarding them to the application itself...

And of course, a lot of web applications are largely front ends to a database, which means they themselves are processing RPC requests, formatting, validating and transforming them before forwarding them to a database for execution...

..and let's not get started about middleware... ;-)

> You seem to want to replace the RPC shared library with a separate process.

No. I really don't. I'm just pointing out that if you are looking for small, modular and loosely coupled components that are fairly resilient, it's not like someone is going to say that moving a component form a shared library to a separate process is going to get critiqued on the basis that it intrinsically makes for more tightly coupled code.

Or wait, are you suggesting that systems where all these libraries are rolled up in to one process would be more modular? [looks at critique of how systemd puts too much stuff in to one process...]

> First, udev is Linux-specific--it uses netlink sockets to listen for Linux-specific hardware events.

I was thinking about this comment, and I realized this is probably the source of most of your angst, which leaves some great solutions on the table.

systemd isn't really creating a much more significant break with the systems you like, because it's building on top of what Linux, which for the most part has already made the break.

The problem is, projects like GNOME, which have software you want to use, are integrating more tightly with Linux and specifically bits of systemd.

I think the obvious solution is a bridge/better interface. The contracts that GNOME is going to rely upon are at least going to be pretty well defined, and if you've got another system that works better, it shouldn't be hard for it to provide an equivalent, even compatible, interface.

If it really is demonstrably better, GNOME and other projects will likely adopt your interface/abstraction, and systemd will end up having to communicate through your interface. Even if they don't, it is a comparatively simpler effort for a software community to support a relatively small set of touch points that they want GNOME to be aware of, and maintaining a fork or compatibility layer is a perfectly reasonable solution (indeed, BSD already does this for Linux runtimes).

I can understand why it'd not be a perfect solution from your perspective, but if a bunch of developers contributing to a work you care about are going a direction you don't like, it's about as good an outcome as one could hope for.

> I was thinking about this comment, and I realized this is probably the source of most of your angst

No, what gives me the most angst is the arrogance of a certain segment of Linux+systemd users who think that just because they can apt-get install systemd and write some minimal unit files for some trivial services somehow makes them domain experts on OS design. And these people seem to think that other users' requirements don't matter, since if they're not using systemd too, they're clearly doing it wrong.

That's a lot of angst for someone that is clearly going to crash and burn in short order and has zero impact on the design and architecture of the systems you and I work with...

This is a pretty good example of how a lot of the contempt for systemd seems to stem from one or both of ignorance of how systemd works and/or ignorance about what a good solution might look like: https://plus.google.com/+LennartPoetteringTheOneAndOnly/post...

> The don't solve race conditions in peers trying to locate each other (surprisingly difficult).

There's only one PID 1 - it would not be hard to locate its UNIX domain socket.

> They don't solve a standardized marshaling format.

Fair point.

> They don't come with an implementation to integrate with main loops for event polling.

They're file descriptors so they work with select, poll, etc.

> They have an inherent vulnerability in FD passing where you can cause the peer to lock up.

Please elaborate. There's nothing inherently vulnerable with FD passing. In fact, dbus relies on it so if you can't make FD passing secure then you can't make dbus secure either.

> You can get into deadlock situations in your messaging code if you aren't really careful about message sizes and when you order poll in/poll out.

I think you're overstating the difficulty of doing this correctly.

> They aren't introspect-able to see what the peer supports.

I for one am OK with PID 1 not being introspectable.

> They make it super easy to not maintain ABI.

It's quite possible to maintain ABI compatibility, but still, how much of a moving target should PID 1's ABI be anyways?

dbus is a great solution for normal applications, but PID 1 is special. The generality provided by dbus is unnecessary. PID 1 should not be servicing requests from ordinary users, so any security concerns with using UNIX domain sockets directly are moot. If there are certain actions, such as shutdown, that need to be triggered from non-root users, then there should be a separate, unprivileged, process that listens on dbus, implements authorization logic, and then relays the command to PID 1 over a UNIX domain socket using a very simple and easily-audited interface. That's good security and reliability engineering.

> They have an inherent vulnerability in FD passing where you can cause the peer to lock up.

(It's kind of unfair but I could not resist): Just like dbus today? http://www.ubuntu.com/usn/usn-2352-1/

Exactly (since userspace DBus has to use sockets, kdbus uses memfd). Imagine if you had to fix that in every program that ever wanted to do IPC instead of just one place. Sounds like a security and maintenance nightmare.

Arguably dbus is basically built is unix domain sockets with a pub/sub routing mechanism (and a ton better security).

> So recently, I just gave up and moved to FreeBSD. Not a single regret so far.

It all depends on the use cases.

For me, FreeBSD is a no go given my desktop usage requirements.

what about the desktop oriented BSD such as dragonfly or pc-bsd ?

Also not, since I require access to specific 3D programming software SDKs (vendor specific), modeling, video editing tools and .NET (work).

This currently makes me a Windows/Mac OS X guy in what concerns desktop usage.

There are a lot of people who agree with you. I feel a bit torn on some of the issues myself, but I'm in a position where I'll be dealing with the enterprise Linux distributions that are adopting systemd quite a lot, so regardless of what I run on my laptop (another BSD fan here), I appreciate the people who are building good resources for getting up to speed on systemd.

Viewing the comments, no it appears you are not the only one disgusted.

I'm just glad for Slackware at this point.

Reply on news.ycombinator.com