IP Accounting and Access Lists with Systemd
0pointer.net
0pointer.net
Then, about two weeks ago, Red Hat decides they're gonna deprecate TCP wrappers, with a Red Hat employee saying:
> TCP wrappers is a simple tool to block incoming connection on application level. This was very useful 20 years ago, when there were no firewalls in Linux. This is not the case for today and connection filtering should be done in network level or completely in application scope if it makes sense.
Okay, fine. I still think it's useful as an extra layer but, nobody really uses it so, sure, let's get rid of it. I'm on board.
Now, all of a sudden, this functionality shows up in systemd (announced yesterday)?
Really!? This is (one of the) reason(s) why some of us greybeards loathe systemd (and just when I was starting to become "okay" with it, too).
You can also configure it without systemd. Julia Evans has a pretty good introduction to it: https://jvns.ca/blog/2017/04/07/xdp-bpf-tutorial/
On the other hand, it's another abstraction over rather new kernel APIs. Your average user may know of NetFilter kernel component, and the iptables userland counterpart. Some may even have heard of nftables. How many users know about eBPF/XDP ? More importantly, how many sysadmins know how to debug this when something goes wrong ?
I'm aware there's a lot of development around eBPF and a number of players (Facebook, CloudFlare, ...) are already using XDP because of the ridiculous performance gain it provides for their use-cases.
There's a myriad of frontends to iptables and yet they all suck. Each of them fail in mysterious ways and at that point the only way to debug what's wrong is to look at the ruleset with ... iptables.
A package could ship with IP filtering in its service file, which may not match what the user expected. The symptom "service does not respond to TCP connections" is well known. In that case, it's common to check the process is alive (using ps), it has setup listening sockets (using netstat) and that the system firewall is not blocking it (using iptables/nftables or some frontend). Figuring out it's a eBPF/XDP misconfiguration is going to be painful IMHO.
systemd attempts to cooperate with the system firewall. The subtlety is that there are now two facilities in the kernel that provide firewalling, and both can be on at the same time. To be honest, I don't even know which one has precedence.
Why exactly does systemd need to mess with the networking services ? AFAIK, its primary function is that of a service manager; i.e: reliably bring services up and down. I feel all the rest should not be systemd's concern.
After using systemd for quite some time, I've come to the conclusion that it works rather well when it sticks to starting/stopping services, assuming your service fits the systemd model. If it doesn't, debugging is extremely difficult and you often have to resort to dirty hacks to end up with a functionning system -- which is ironic considering one of systemd's motivations was to get rid of the dirty hacks in initscripts.
An example of a buggy use case is when you need to pass environment variables to your executable and there's escaping involved. The systemd documentation suggests you can pass pretty much anything in the ExecStart lines. However, the reality is that it does not behave as other environments. The solution is to store the variables in a separate file, and not involve systemd at all. My takeaway is: why does systemd involve so many features ? It increases failure cases as well as the cognitive load required when dealing with it.
It doesn't. All this stuff is opt-in. If you don't want to use it, don't.
What's a bit unfair though is that you imply that the various technologies where equivalent in behaviour and would be different only in performance. That's really not the case. The eBPF stuff allows configuration of per-cgroup, i.e. per-service or per-container access control. And that's a massive advantage. Classic Netfilter doesn't allow that: you can't tell it to permit all traffic to bittorrentd.service, because service/container/cgroup information is not available to it. Yes, people have tried to make that work, and there was some support for it in the kernel, but only for egress, and that made it pretty useless ultimately.
You know, if iptables would allow us to configure tables per-service, we'd happily hook that up. Such a design is in fact where we started from originally. But the fact is that such a scheme isn't really compatible with Netfilter's design and thus didn't happen. And that that's the case is really not systemd's fault, but a technical decision of the kernel community.
I know it is fashionable to blame systemd for everything, but seriously, eBPF+cgroups and all that kind of stuff is the result of the kernel's development scheme, and systemd only follows that and exposes what they came up with. If you don't like where kernel development goes, please complain about and to the kernel community, but systemd is just a consumer of what they do, and I couldn't care less if per-service firewalling is ultimately implemented based on netfilter, on nftables or on ebpf/cgroup. As it turns out only the latter ended up being capable enough to make this possible. And I accept that, and then made use of this.
Do note that the port forward hookups in nspawn (i.e. --port=) are actually implemented using netfilter, not cgroups/ebpf. I hope that illustrates a bit that we use what works and is provided by the kernel community, and don't really follow an evil scheme here.
Lennart
This sentence is getting rather long in a tooth. Cgroups? Kdbus? udev?
Same mantra that is used over and over again turned out to be at least deceptive if not outright lie, each time it was used.
"After udev is merged into the systemd tree you can still build it for usage outside of systemd systems, and we will support these builds officially. In fact, we will be supporting this for a long time"
http://article.gmane.org/gmane.linux.hotplug.devel/17392
"...this will effectively also mean that we will not support non-systemd systems with udev anymore starting at that point. Gentoo folks, this is your wakeup call."
http://lists.freedesktop.org/archives/systemd-devel/2014-May...
So when Lennart or somebody in the systemd team claims something to be optional you can bet everything it is not or soon won't be.
Some comments:
> It doesn't. All this stuff is opt-in. If you don't want to use it, don't.
I said I feared a package used it, because it made sense at packaging time. Let's be honest, upstream often has less than sane defaults. In that case, there will most likely be some hair-pulling on the user/sysadmin side before he figures what what's wrong. And then, he will definitely blame systemd (even though it's not really systemd's fault).
> What's a bit unfair though is that you imply that the various technologies where equivalent in behaviour and would be different only in performance. That's really not the case. The eBPF stuff allows configuration of per-cgroup, i.e. per-service or per-container access control. And that's a massive advantage. Classic Netfilter doesn't allow that: you can't tell it to permit all traffic to bittorrentd.service, because service/container/cgroup information is not available to it. Yes, people have tried to make that work, and there was some support for it in the kernel, but only for egress, and that made it pretty useless ultimately.
My bad. I mentioned the performance aspect because AFAIK, that's the main problem solved in the big players' case. I'm fully aware of Netfilter's limits. It is very powerful, no doubt about that, but it operates at the network level. Therefore, it has limited knowledge regarding the software running (unlike, say, the firewall integrated in Windows that can permit/deny depending on the process that attempts to contact the network). eBPF/XDP bridges that gap, which is an interesting feature for the user (as you rightly mentioned).
> I know it is fashionable to blame systemd for everything, but seriously, eBPF+cgroups and all that kind of stuff is the result of the kernel's development scheme, and systemd only follows that and exposes what they came up with. If you don't like where kernel development goes, please complain about and to the kernel community, but systemd is just a consumer of what they do, and I couldn't care less if per-service firewalling is ultimately implemented based on netfilter, on nftables or on ebpf/cgroup.
I didn't blame systemd for eBPF :-). I'm fully aware that the kernel community often has different concerns that what us user-peasants would like. I am bothered by the overlap between eBPF/XDP and Netfilter and more importantly, that the tooling seems to be lacking.
Using it in systemd is opt-in, sure. However, a user that opts into this feature is not aware of the technology used underneath. When debugging time comes, he will be wondering what this magic is.
> As it turns out only the latter ended up being capable enough to make this possible. And I accept that, and then made use of this.
Why does systemd need to expose it ?
As you said, it's fashionable to blame systemd for everything but I feel you're just making it easier by adding knobs for everything. What is the scope of systemd ? Where exactly does it stop ?
Since linux 4.10 you can attach eBPF programs to the cgroups: https://kernelnewbies.org/Linux_4.10#head-4f00d500db4f8e437a.... This allows you to do a really cool things, mostly firewalls attached to the cgroup and not to the system-wide netfilter stack. Imagine application-specific firewalls.
Systemd uses that to add "simple" packet counting eBPF application to the cgroups: https://github.com/systemd/systemd/commit/1988a9d12015990c14....
(eBPF is the same mechanism that is used with seccomp and filtering which syscalls can application use and which are forbidden - used in for example chromium sandbox on linux or with openssh)
Common response from the systemd groupies is to focus on technical bits and be precise but you can't focus on them if the concept/ideas/execution smell. Also, number of wrongly planned or wrongly executed bits counts in hundreds by now. Like cooking a frog.
Can you develop on some of the reasons that convinced you to run away from systemd ? I'm really curious to know why (honestly). Thanks :).
There are similar sources on the internet that claim the same thing in different forms. Every single bad systemd decision is not terribly bad on its own, but when observed as a process they result in a horribly wrong and very insecure system.
With systemd, there i crawling revolution of bad ideas, bad implementation and taking over of everything that was great in linux. When you try discussing any of those items separately you'll be redirected to 'technical discussion' and you can't really technically discuss conceptually wrong system. With bad concept it doesn't really matter how technically good or bad implementation is.
Maybe the best telling thread, in order to understand how pushy and deceptive whole systemd is, is watching how Greg Kroah-Hartman tried to sneak kdbus into a kernel: https://lkml.org/lkml/2015/4/13/645
It is very long mailing list thread and GH is not officially part of the systemd team -- only that he is, unofficially. He claims that kdbus brings great benefits and speedup only that he can't prove it. Every questioning of a very wrong concept is met by requesting of 'technical discussion'. If you have patience just read that lkml thread and you'll see the same sneaky politics of pushing things the same way that systemd people do it all the time.
I remember of this long article, that, technically demonstrates that systemd is badly designed, as you said: http://blog.darknedgy.net/technology/2015/10/11/0/
Didn't read it entirely at the time as it dives really fast into the technical stuff and provides an ever growing flow of informations that you have to absorb to understand it correctly. Seeing people like you being so concerned with systemd makes me inclined to re-read it fully.
And why FreeBSD and not voidlinux or something in this fashion ? More secure ? More mature ? Will never be subject to such issues ?
I am surprisingly happy with FreeBSD, userland packages are very fresh, which I like being former Arch linux user. Kernel works at least on the level of linux kernel, maybe even better. Zfs is something that I didn't completely understand before I tried it out, even with all the theory I knew. Zfs is just mind blowing.
I very much love linux and it is not impossible to imagine switching back, at least for some of my machines.
It is mostly matter of sanity and nerves, I couldn't follow and understand all the systemd flaws while pretending everything is OK.
I've been running this as my main dev machine at work for six months now without too many issues. The biggest problem was installing TeX packages actually :-)
I am not sure why you imply that there was some evil scheme behind all this. In reality, things are a bit different: one group at Red Hat decided to put an end to TCP wrappers (which btw is something other distros already did a while back, we are late at the party), and another group then offered to make a more modern replacement available, that could cover some of the uses, in order not leave users in the cold. I am not sure why you think that's an evil thing to do? Isn't it nicer to drop something with a suggestion how to replace it then to drop it without replacement?
Also, let's not forget: this functionality is implemented in the kernel, systemd merely adds knobs to expose this kernel functionality to the user. Hence, no, this functionality didn't really "show up in systemd". It showed up in the kernel, and systemd just wraps it to make it usable.
I mean, this is a dilemma, isn't it? Whatever we do you are unhappy, right? ;-)
Lennart
Personally, I am very pleased with systemd and thanks to its novelty, it puts me on a same level as all the "greybeards". Continue the good work and please, take special care of the man pages.
He didn't imply or say there was some evil scheme, and this reads like an attempt to play the victim card on your part.
If no one uses the functionality is the reason for getting rid of it, then why add it back in? I seems like a lot of useless busy work to explicitly claim that no one uses something only to replace it.
connection filtering should be done in network level or completely in application scope if it makes sense.
So it was done at the application scope, with a common config file format, and then it was deprecated because it should be done at the application scope? I can barely make sense of this logic they are applying.
The only place it wasn't done at the application scope (baring an invocation wrapper that handled it for applications that didn't directly support tcpwrappers on its own listeners) was in systemd, because systemd isn't the end application, so if the logic is that it should be in the application scope, systemd is the only place it should have been removed from.
IPAddressDeny=any
> It's a simple way to lock down distribution/vendor supplied system services by default. For example, if you ship a service that you know never needs to access the network, then simply set IPAddressDeny=any (possibly combined with IPAddressAllow=localhost) for it, and it will live in a very tight networking sand-box it cannot escape from. systemd itself makes use of this for a number of its services by default now. For example, the logging service systemd-journald.service, the login manager systemd-logind or the core-dump processing unit systemd-coredump@.service all have such a rule set out-of-the-box, because we know that neither of these services should be able to access the network, under any circumstances.
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
I'm sure this is useful but init is absolutely the wrong place for it.
>There was "a brawl" at the workshop about possibly replacing the nftables virtual machine with eBPF, but there is one major show-stopper in any such plan: nftables allows partial replacement of a firewall configuration, while eBPF, in its current form, would not.
Cloudflare, Netflix, Facebook, etc are all using eBPF via XDP https://www.iovisor.org/technology/ebpf
eBPF and XDP are supported seamlessly on kernel 4.12 and later.
I guess I'd prefer a single api, so I could sandbox (or not) netcat and Firefox the same way as samba and openssh.
It might not be a major point - but it would be nice if there was a single way to "run in env/with configuration" that didn't involve an extra (and somewhat opinionated and opaque) wrapper.
It's a little like the dot.desktop-file approach to dress up cli tools/interfaces in a desktop gui - it's brittle and doesn't really work all that well. And the cross-platform story isn't great.
On the other hand: attaching capabilities to process trees seems better that text-based matching etc. I suppose I think the bsd jail approach looks better - even though that to involves a "wrapper" - but it's a wrapper that's not artificially limited to "services".
[ed: the fact that any kind of name lookup is left out is also a bit odd - as demonstrated by the curl example, for many uses of whitelisting name resolution is essential. And for ldap/kerberos/ad being able to work with named resources seems to be the only sane idea. That's why we use dns and catalog services in the first place! This even more clear in an ipv6 environment, as the addresses are generally longer, and auto-assignment is somewhat better than ipv6/DHCP.]