Systemd-resolved DNS cache poisoning
seclists.org
seclists.org
Before you had a situation like this. ISP: Server and cache. Router: Client, server and cache. OS: Client and cache.
So if you ever encounter DNS failing, you only had 3 possible places to look for errors, or maybe not even errors, just stale data that may or may not correct itself over time. Flush the caches you can and hope for the best. Rinse and repeat.
But then came the browsers and decided that 3 levels of caching was good enough. Let's add more which can fail in the name of performance!
Combine this with mobile, mobility and data-roaming and now you not only have a DNS-cache (based on already tripple-cached data) which often is guaranteed to be wrong, your browser may actually have cached the wrong DNS-servers, so when you roam from Wifi to cell-data your precious LAN DNS servers are no longer available and your browser just looks stupid.
And now systemd adds to this. What the flying fuck?
All this nonsense because someone somewhere decided that a nice, shared, global OS level client and cache wasn't good enough. Handling this complexity once would just be too easy.
Can someone tell me why? Why wasn't it good enough? What was it that prompted someone to go reimplement all this wrongly, yet again?
Yes this is a rant, but if there are underlying technical reasons for this nonsense, I'm very, very interested to hear about it.
resolved: add DNS cache
Yes, that's the full commit message for what I think is the commit that adds the feature. I was trying to dig up an authorative source for the "why" you're asking, but I guess that failed. Perhaps it was discussed/motivated/described/something somewhere else, like a mailing list, I didn't search further.
If true (that it is a 4 word commit message), that is a disgrace. Seriously, something as important as this should contain a lot more information, if not just a pointer to a place that provides it.
Actually fuck that. Which is why I'm sitting here on a FreeBSD machine.
1. Solved problem. Contribute to something else.
2. Its another tentacle hanging off systemd. Some of us pay for our software (RHEL) so we'd in turn like to have some say in what we have to manage rather than prescriptive software development. We don't really get to choose to not use it because of the incestuous nature of the Linux ecosystem at the moment (I.e. Freedesktop/gnome/systemd).
3. Systemd is a buggy piece of crap. As are several of the "stable" bits of systemd I have encountered in RHEL7. I have quite a bit of experience with systemd and i really don't want it consuming any more things badly. Its been a complete pain in the arse to be honest. An init system has never been a problem until now. Its totally unreliable and incredibly difficult to debug.
4. Some of us get shit done, not use the ecosystem as a playground for egos and quote frankly incompetent twattery. This stuff can go on in the sidelines and mature independently, not knotted into the next "stable" drop because everything has suddenly been tangled with it.
Its a complete fuck up at all levels and I haven't even started on the technical marvels of introducing something with the complexity of COM into a process model that requires reliability.
This is not how to do software.
The guy going "mea culpa" or anything close to that is like spotting a polar bear in Sahara.
While I agree with you that these layers of caching can lead to headaches, Linux doesn't have a built-in cache (and distros often don't provide one by default). Systemd would be one way to provide this, instead of other daemons that have been used in the past, like nscd, dnsmasq or bind. So it's not adding yet another layer, just providing the layer at the local host.
As for technical reasons, I'm not sure what you want to hear, each layer implements a cache for performance and/or efficiency. No one throws in a DNS cache just because they feel like it.
What does the kernel have to do with this? No one is proposing that the kernel should have a DNS cache.
Systemd is a user process and now tries to replicate functionality that we already had a perfectly good solution for.
Since a DNS server is typically also a cache we already had this functionality (and more), simply make it listen to a local port and nothing else. That way you don't need to resort to rebuilding things we already have with incompatible interfaces.
Linux develops fewer things this way than the BSDs typically do, outsourcing a lot of userland to other projects like GNU, but it doesn't literally develop only kernelspace code. Imo it's generally an improvement; the situation with trying to get things like cgroups to work together with process management previously was terrible.
The reason those have to be kept in sync is simply because you don't want to run fsck on a filesystem that was built with a newer kernel that possibly has datastructures that your fsck (or other filesystem aware utility) accesses directly without going through the kernel. Essentially such programs re-implement a bunch of kernel functionality because they access file systems that are not currently mounted.
Systemd on the other hand (in the case at hand) replicates the functionality of another user process that we already have, know how to use and have hardened over many millions of hours of use. That seems folly to me, it duplicates functionality, makes troubleshooting harder and increases the size of the codebase without actually improving anything.
What's next? The framebuffer? Keyboard interface? TCP stack?
There are no really hard reasons why those could not be user programs. It's fine with me if we make linux a micro kernel but let's do that in a way that we do not need to revisit the last decade or so of security issues and ill thought out interfaces. Once we start to need things like DBUS and X in order to bring up a server we've really lost the plot.
I'm afraid so: http://lists.freedesktop.org/archives/systemd-devel/2013-Nov...
A userspace replacement entitled kmscon has already been around since 2012, by David Herrmann. It's a totally separate project.
consoled is a minimalistic fallback done by the same person. I'm not sure if it really needs to exist, but replacing CONFIG_VT in of itself isn't bad. Most people will probably still have kmscon primarily.
Why does systemd need to provide it? I already provides enough things.
Are there any bugs in :
http://www.thekelleys.org.uk/dnsmasq/doc.html
I had great success using that. Both installing it on my own or run it part of Ubuntu's base install.
(And the answer of "but it already provides everything + kitchen sink, might as well provide DNS caching too" would probably not be what most people would want to hear).
I doubt you will find anyone interpreting this as such.
http://www.informatimago.com/linux/emacs-on-user-mode-linux....
I'm not saying it needs to, or trying to defend systemd. All I'm saying that systemd is providing and alternative to running another resolver.
This is the worst case of feature creep in system land in a long time.
If a group is actively trying to ruin something, noticing that they're trying to ruin things isn't going to stop them. That's a useful strategy with accidents and friendly groups, but not with actively aggressive oppositional destructive groups.
Mostly I'm just angry about having to waste the labor hours moving everything to Freebsd from Debian. Its interesting, but not exactly productive.
Systemd is a cancer, it will end up killing Linux if this goes on unchecked. At least Torvalds seems to (most of the time) know his stuff, the way they replaced the memory management left me with a bad taste but in the end it worked out. What's happening with Systemd is absolutely unpalatable and I hope that Redhat and other distros take note, there are a lot more Linux systems on servers than on Desktops and if they believe that expending all this effort on Linux on the desktop so they can better compete with Windows they might just lose their dominance on the server because of it.
Red Hat is developing systemd. It has nothing to do with making Linux palatable on the desktop, but a severe case of NIH, which to be honest is hardly new in Linux-land.
systemd won't kill Linux. Linux development is out of the hands of hobbyists and enthusiasts now and corporate users and developers don't care about the holy wars. Red Hat ships systemd, Red Hats users will use systemd and that will be that.
Their existing server customers are likely to be sitting on RHEL7 for quite some time. And i suspect that the workstation customers, mostly military as best i can tell, are in part pushing for systemd. Logind etc after all is heavily focused on multi-seats.
I don't see FreeBSD's init trying to absorb most of the userland system utilities.
Whether or not it is in one repository isn't really the issue at hand here. It is that a bunch of desktop related stuff is worming its way into the systems to the point where you'll be running a whole slew of new, buggy and sloppy code on your servers if you're not very careful just so that the systemd folks get to scratch their NIH itches.
And even when in one repository i think you can grab the BSD utils without getting the kernel and whatsnot in one big tar-ball.
If you try to grab say just the dns daemon from systemd you can't do so directly. you would have to grab the tar-ball for the whole of systemd, and then attempt to extract just the parts dealing with the dns daemon.
Damn it, LFS have found doing this for udev so demanding that they have switched to eudev for their non-systemd edition (yep, they now have their own systemd edition going alongside the "traditional" one).
I don't think the kernel developers care that much about what systemd is doing (except for the notorious clash about systemd interpreting a debug option meant for the kernel and crashing the boot...), so you can't say the kernel is copying or doing anything.
The distros, however, are possibly the ones you want to blame for pushing and supporting systemd. Which is actually funny because the way it's going, systemd is laying foundations for a new kind of linux distributions to which distributions will certainly have to adapt if they want to keep it.
It's scary for me, because everything I've done has been in a Linux, but it's also educational for me, because everything I've done has been in a Linux. If the systemd invasion results in a lot of attention going to the BSDs, it's not entirely a bad thing.
Regarding sysadmin, I tend to disagree, I think it is the opposite, mainly due to great handbook they have (http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/).
As suggested reading "man hier" is a great way to understand the filesystem layout. I personally also really like clear separation between the system and applications. For example all ports/packages are installed in /usr/local including configuration files (in /usr/local/etc).
Your proposal would require a radical paradigm shift as to the fundamental project goals of systemd, in which case it'd probably become something like uselessd.
Without caching, all DNS queries degrade to worst case behaviour, which is a recursive query involving 4-6 or more round trips, starting at the DNS roots (which, good luck finding them, since you don't have their addresses cached, which is exactly what the root hints file is).
As for invalidating the cache when network configuration changes arise, that sounds like a bug report for wherever you experienced it. Certainly I've never seen this kind of problem on OS X, nor any time in the past 10 years on Windows.
errr, xxxxBSD don't. libc sends every DNS query to wherever /etc/resolv.conf says to send it.
The long answer is:
Often, systems do not come feature-complete. There are many cases of a platform, or OS, or service, or framework, or some other piece of software being released without every possible imaginable feature being implemented. When this happens, some other software may need to re-implement some core piece of technology in order to add functionality to it. Usually care is taken to not interfere with other systems or cause additional burden to the user. In the case of DNS, our use of DNS and different networks simultaneously requires additional functionality in order to process requests independently and efficiently. As the DNS protocol itself has an inherent flaw allowing for cache poisoning, it's non-trivial for people to implement without running into this security hole. And as someone else mentioned, the kernel and glibc don't have their own DNS cache.
I didn't downvote you for the rant because you weren't being mean or insulting and you genuinely seem to want to know what's going on. But next time you feel yourself posting out of emotional reaction, I recommend typing it out in a draft and save it for a half hour, and see if you might want to reword it later to be less angry/ranty.
Systemd here tries to be the official standard system level caching.
I agree though, about your extra caching sentiment to a degree. Actually as long the caches are RFC complaint (e.g. they obey TIL - looking at you nscd & java) or should be fine.
No.
But the solution is not to complain, but to use /etc/hosts, run your own root and your own cache.
Any software that overrides /etc/hosts is not worth using, in my opinion. There are no "underlying technical reasons" for not letting the user control their own name resolution that benefit the user. Of course, there could be reasons that benefit someone else.
It turns out that there's a very well-specified protocol by which clients can ask a local cache on their system to answer DNS queries. That protocol is called DNS :) I don't see why routing something DNS-like over dbus makes any sense in contrast to doing it using DNS itself on port 53.
Fedora is experimenting with running unbound as a local caching resolver [1]. This gives caching, DNSSEC validation, and all the benefits from the fact that unbound is probably much better hardened than the average libc or application-side DNS client implementation.
[1] http://fedoraproject.org/wiki/Features/DNSSEC_on_workstation...
Hell, i have recently seen a fedora bug regarding the use of su -. Where Poettering argued that people should not use su because it broke dbus.
Instead he seemed to advocate that people ssh into 127.0.0.1 to do their thing with a different account.
Why? This wasn't caused by systemd's complexity or having too much components interconnect poorly. It was a bug in one of their implementations that could have equally happened had it been a stand-alone DNS cache resolver.
> Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well.
I'd totally buy that if BIND didn't have such a checkered history. I mean with BIND 9 they scrapped the entire project and started again because it was so insecure...
Yet they're only attempting to do "one thing" so according to you they should have done it well.
> Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard.
This line oddly contradicts the line that directly proceeds it.
> For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well.
According to the previous line they aren't doing that one thing well...
Take DJB. He's an expert and djbdns has a stellar security record. He's not going to spend his time trying to make systemd's resolver secure. Instead systemd is going to end up repeating all the same mistakes BIND has made. At least BIND has a substantial head start over systemd-resolved.
So they would work on systemd then? Since it is set to have a far larger customer base in the near and long term than most independent cache resolvers (just see the list of distro's who have taken it up).
> Instead systemd is going to end up repeating all the same mistakes BIND has made. At least BIND has a substantial head start over systemd-resolved.
Potentially. But I don't think you've successfully shown that that is inherently due to systemd's design, complexity, or componentization. Instead it is likely due to systemd's relative immaturity of a codebase and that as discussed previously most cache resolvers have had a checkered history of security issues (and assuming systemd is "as bad" as everyone else, it will have security issues).
Is the systemd project passionate about DNS? Is the curl team passionate about protocols and files? Is the opensmtpd team passionate about e-mail?
I think two of the three are yes answers. The great strength of UNIX has always been people's ability to flock to places where their passion is shared and improve and learn (institutional knowledge) about their passions.
I don't think systemd will have passionate people for these extra services and it seems they don't have the institutional knowledge that others have gained since this is not a new kind of bug.
Tons of projects with a single focus have had countless security problems, some so bad the entire project was completely scrapped.
So the entire argument is based on a flawed premise.
Yes. And when it happened before, it was The Bug of the Century. Repeating that mistake is either stupidly incompetent or proof the systemd developers were literally born yesterday.
New development is expected to have some bugs, but it shouldn't sound like a tribute band playing the greatest hits of yesteryear. And they can't exactly fallback on the old standby of "oh, it's a legacy code base."
Do one thing and do it well.
It's like saying "package dnsmasq.deb is vulnerable, so sysv-init.deb is now vulnerable, they share an extension"
Even Debian has succumbed to Systemd.
Nope, that would be GSettings/dconf. It's limited to GNOME, thankfully.
The registry is fucking marvellous in comparison.
You see systems like Docker containers, which try to isolate a complete application and its configuration files together, and really that is just a hacky solution to the lack of inherent silo-ing within the underlying design.
While silo-ed apps have their own limitations (e.g. two "apps" using the same configurations) most of them can be overcome with tinkering for the edge cases.
I'd never design a system in 2014 with either Windows Registry (too messy, impossible to manage), or /etc (too inconsistent with itself, also often results in orphan files).
I encourage you all to please use the more appropriate comparison : Cthulhu / Cthulhud
Considering most users run on a backported 208-stable IIRC (I know RHEL 7 settled on that), I'm not sure how widespread this is. Still concerning that they felt to roll their own and not follow RFC guidelines, though.
Considering how plagued with vulnerabilities BIND was, I'd assume DNS is a hard thing to do.
That quote is gold. Most of DNS isn't implementing the protocol, but implementing all the protections against cache poisoning, many of which are not intuitive.
But as the oss-sec email says: "systemd-resolved does not implement any of the hardening recommendations of rfc5452."
I understand concerns about systemd's scope, but most of the "questionable" functionality is optional.
By the same author
"There's no such thing as a simple cache bug."
"Caches aren't architecture, they're just optimization."