Systemd Forward Secure Sealing of System Logs Makes Little Sense | Hacker News Reader

Systemd Forward Secure Sealing of System Logs Makes Little Sense

louwrentius.com

39 pointslouwrentius46 comments

This article is predicated on the assumption that an attacker goes from zero access to full root within 10 seconds. While there are vectors which have allowed such rapid escalation, they're less common now than historically thanks to Linux's security looking more like an onion rather than just a binary state (root or user).

Also the author picks apart a quote, but if the assumptions in the previous paragraph is taken at face value (that an attack can have sup-10 second root with automated attacks on the logs), then even remote logging isn't a secure vector as they often aren't sent instantly so an attacker could intercept or alter them.

However I suggest that such a scenario is uncommon. Most attackers have to start outside and fight their way in which will generate plenty of logs for either external logging or FSS.

> It seems to me that systemd now also wants to play the role as some crude intrusion detection system.

FSS isn't a "intrusion detection system." It is an attempt to make logs immutable, nothing more. The nice thing about FSS is that it can be coupled with an actual IDS and the IDS can look for attempts at altering the FSS.

> If the systemd author would just have implemented sending log events to an external log server, that would have been way more useful security-wise, I think.

It isn't a binary choice. They can implement FSS and remote logging. Also it is open source so you yourself can contribute external logging.

>This article is predicated on the assumption that an attacker goes from zero access to full root within 10 seconds. While there are vectors which have allowed such rapid escalation, they're less common now than historically thanks to Linux's security looking more like an onion rather than just a binary state (root or user).

You mean like a disgruntled system administrator? There are countless scenarios in which an attacker goes from zero access to root within 10 seconds. Predicating your security on that not being a common scenario shows that you've spent next to no time in a real corporate environment. IE: Poettering.

Nonsense – defence in depth and all of that. FSS is useful because it provides a layer of audibility for users that wouldn't have external log servers in any case, and because it provides some minimal guarantees about the ability to tamper with logs. That's useful, regardless of the fact that it's not a bulletproof solution!

I didn't say FSS was a step backwards or wasn't useful. You completely ignored the content of my post. When the project founder tells people "hey, you don't need external syslog servers" it shows an absolute ignorance of WHY we have external logging servers. And that kind of ignorance is dangerous in a project leader. ESPECIALLY when his solution to being called out on his ignorance is to ignore the person who was critical.

I don't know a whole lot about this stuff. Why would one set up remote logging and not send logs instantly?

Sometimes you have a lot of throughput, and it simply isn't feasible. Could potentially saturate your network, or overload your remote aggregator. Of course, all of this is fixable, but often times when every node just pushes logs out, it's hard to throttle that.

Because sending logs has a lot of overhead. Much of that overhead can be mitigated by waiting even a few seconds, packaging several logs together, and sending a batch.

On a LAN where the sender and receiver are close this type of overhead is much lower than over the internet. It is also worth noting that on some systems several logs can be generated each second, several hundred a minute, so even modest overhead can become costly 24/7/365.

It is also worth noting that some people have remote logging configured NOT for security reasons (primarily) but for system health monitoring. In this scenario the speed isn't as vital as keeping the overhead low.

>they're less common now

Source?

>arent sent instantly so an attacker could intercept or alter them

intercept where?

>Most attackers ... generate plenty of logs

Source?

>They can implement FSS and remote logging.

Point is, thus far they have not, and their first choice was not what the author views as more secure.

> Point is, thus far they have not

This has been possibly through forwarding to rsyslog right from first release.

> Source?

Experience. I've set up SELinux, AppArmor, etc. For example back near 2000 a lot of Apache servers I worked on were running as root, now few are, and better still many have limited permissions even as their user/group.

The same can be said for many Linux services today. And better still the few that HAVE to run as root often come shipped with some level of permission controls to limit exposure.

I can link you to a random book on Linux access controls and security if you want an actual third party source. Or be a total dick and link to a man-page.

> intercept where?

It depends which system you're using to send logs remotely (and how it is configured). Sometimes the filesystem, sometimes IPC, etc.

e.g. http://www.rsyslog.com/doc/queues.html

>> Most attackers have to start outside and fight their way in which will generate plenty of logs for either external logging or FSS.

> Source?

Source on, what? That attackers who break into systems generate logs? You REALLY need a source on that, REALLY? Come on now.

> Point is, thus far they have not, and their first choice was not what the author views as more secure.

Alright, and then criticise them for that. If the article had been about systemd's lack of remote logging that would have been one thing, but it was complaining that FSS was bad because systemd lacked remote logging. FSS existing is irrelevant.

Also, again, open source. Go implement shit.

>>arent sent instantly so an attacker could intercept or alter them

With Reliable Event Logging Protocol [0], interception is somewhat mitigated. You can configure and white-list certificates on the client and server so that even if a network attacker inserts a syslog relay with a certificate that's signed but isn't the one expected, you can't intercept the messages. Though, if you have that much control of the network, you can just drop syslog packets entirely. Having a dedicated management NIC and network helps mitigate that.

And I think there's a big difference between 10 seconds, and any delay inherent in to external syslogging. 10 seconds is enough time for an automated exploit to penetrate multiple layers of the security onion, but external logging should be able to squeak out a few messages between pwnage and syslog clobbering in the same situation (YMMV).

[0] http://www.rsyslog.com/doc/relp.html

(Not OP, but I'll reply anyway.)

>Source?

You need a reliable kernel exploit, which isn't always easy to download and compile within 15 minutes (or 10 seconds, or whatever you set the interval to). But I agree that this is not a very good mitigating factor.

>intercept where?

If you have root on the box and logs are being sent to another server via syslog, you can intercept the traffic in all sorts of ways.

>Source?

Read any incident response book or blog.

>Point is, thus far they have not, and their first choice was not what the author views as more secure.

Yeah, I side with the blog author in this case.

You make a good point. FSS isn't an IDS, it's a metric to be used by an IDS.

However, that raises the question of what the hell it's doing in the init system, and not implemented by the IDS. That's a SOLID violation.

It would appear that systemd left behind SOLID engineering principles a long time ago.

This is EXACTLY my complaint.

“Security” is not something you can relegate to an IDS. Security must be embedded in each program from the start.

> Security must be embedded in each program from the start.

Agreed. Wholeheartedly. However, the way most of us interpret that bit of conventional wisdom is that security should be added to existing features, for example, by cleansing user input and not passing on tainted input (shellshock is a classic example of this, in the vein of a system-based XSS vuln). It does not mean that we should develop "security" features as addons to existing systems.

In this case, they use an untested cryptographic primitive (SSKG) which was published by Poettering's brother almost a year after the feature landed. That's negligent and reeks of nepotism.

I wouldn't recommend that any sysadmin use this feature in a security context (there are existing security products that provide log-signing, with security notifications and patches).

The answer to that is that systemd is much more than an init system.

This starts off looking like a reasonable criticism of systemd, or rather that systemd claims to do more that it (or any alternative) actually can guarantee. It quotes the author of systemd to prove that the claim is made, but it does not give a link or say from where the quote was taken (so we don’t know the context or age of it, i.e. we don’t know it is still relevant).

But then it veers off: “This whole observation is not original, in the comments of the systemd author's blogpost, the same argument is made by Andrew Wyatt (two years earlier). The response from the systemd author was to block him.” No links are provided, so we can’t know what any of them actually said or did.

This smells like deliberate character assassination. Especially the fact that no references are provided seems fishy to me.

> It quotes the author of systemd to prove that the claim is made, but it does not give a link or say from where the quote was taken (so we don’t know the context or age of it, i.e. we don’t know it is still relevant).

I'm critical of the article also, but they in fact did. If you click the "Forward Secure Sealing (FSS)" it takes you to a G+ page with the quote on it from Lennart Poettering on Aug 20, 2012.

It was not stated that that was the source of the quote, and the link was not given anywhere near the quote, but I guess you are right that there was a link…

So this quote is from Aug 20, 2012. Has Lennart’s opinion of this changed during the more than two years that have passed since? Does the documentation of systemd make this guarantee? Did the documentation ever make the claim that FSS is an complete replacement for remote logging?

Now that I think about it, and have the context of the quote, I think that I (and the blog post author) maybe misunderstood Lennart. FSS is not a complete replacement for remote logging, but: Among other things, remote logging used to be the only way to get immutable logs. If this was your only reason for using remote logging, then FSS is a viable replacement.

EDIT: Now that I can actually read what Andrew Wyatt wrote and how he wrote it, I can see why he was blocked. He was condescending and vague on details, and (according to Lennart) Andrew had a habit of doing that. The comments to the blog post actually cover what I wrote in my previous paragraph, so there is no excuse to dig this up now, two years later, and deliberately misunderstand it in order to gain points with the systemd haters. This is FUD.

I found the comments from Andrew Wyatt - They're on the same G+ page linked in the article actually (Though it would be nice if the article pointed this out). Here's a full copy of the relevant comments:

Andrew Wyatt:

> Sigh.

Lennart Poettering:

> +Andrew Wyatt Sigh?

Andrew Wyatt:

> +Lennart Poettering I'm not going to waste my time explaining why.

Lennart Poettering:

> +Andrew Wyatt cool, then what you doing is not helpful, and just noise, and hence waste of time, and hence why do you post this at all? You are contradicting yourself there...

Andrew Wyatt:

> +Lennart Poettering I forgot that this wasn't a free internet, forgive me for sighing in public about something that doesn't really solve the problem that it aims to solve.

> One of the principle reasons to ship logs is not just to protect and preserve them, but to remove the control of the logs from the administrators of the systems - as sometimes the attack happens from the inside.

Lennart Poettering:

> +Andrew Wyatt Cool, I'll translate this for everybody else: +Andrew Wyatt has no clue what he is talking of, but is full of negativity, hence tries to spread FUD, and be as unspecific as possible, making wild claims he makes up of thin air, to just make this negativity stick.

Lennart Poettering:

> +Andrew Wyatt I never claimed that FSS tries to solve the problem of attacks from the inside. That is a fundamentally hard problem, since it is entirely against the protection model of Unix: the admin is in power, and this gives him the power to stay in power. If you want to fix that problem then an external log server won't help you at all. You need to change Unix from the ground up, and not just that as the untrusted admin problem is primarily a social problem that you can only do so much about solving technically.

Andrew Wyatt:

> +Lennart Poettering it is actually a pretty easy problem to solve, I should know as someone who has solved it. Log syslogd events including starts and stops, and ship every log message to a server or cluster of servers that administrators don't control. A good forensics auditor would know who was on the server when the syslog kill event occured, and if the nodes are configured right - at least some of the steps executed prior to.

Lennart Poettering:

> In case anybody wonders, I have blocked +Andrew Wyatt now. If people get personal I am more than happy to remove their comments, and ultimately block them.

The last reply claims they removed some comments, so we don't really know what got said between those two posts (the last two). Maybe nothing, maybe something.

So, in one fell swoop, Poettring demonstrates both his own immaturity as well as a practical example of not having an external log server--had that been the case, the comments couldnt've been deleted by a "disgruntled admin" (Poettering). :)

Why the hell would you trust a software community with this guy? He doesn't seem reasonable at all.

Granted, Andrew Wyatt was acting immature as well, but his last response was decently reasonable and seemed like a fair point. I could have understood ignoring him, but the fact that Lennart was clearly provoking him isn't what I'd want to see from a project leader. That's simply not a way to treat people who seem to have legitimate gripes. I could understanding calling him out for his 'Sigh' comment, but I don't understand claiming he has no idea what he's talking about and is just spreading FUD and 'making wild claims up out of thin air' after he explains his point. That comment was simply completely unnecessary and just Lennart being angry.

If Andrew did keep acting like he was at the beginning and Lennart answered all his questions/complaints, then I could understand either blocking him or discontinuing the conversation. But the fact that Lennart either never responded or removed all the responses after that is disconcerting, and honestly just annoying. If he had just answered it and left the responses up then most complaints would be gone and this article wouldn't even exist.

I've updated the article with a link, it's the same link as at the top of my blog post but originally it may not have been clear where I got this quote from.

An interesting thing to note is that systemd's FSS feature is based on a FSPRG [1] (forward-secure pseudorandom number generator) contributed by Lennart's brother Bertram Poettering, based on the latter's research paper entitled Practical Secure Logging: Seekable Sequential Key Generators [2].

[1] http://cgit.freedesktop.org/systemd/systemd/tree/src/journal...

[2] https://eprint.iacr.org/2013/397.pdf

Oh, goody, so it's homebrew crypto as well?

I think the systemd complaints really boil down to two very real problems:

1. The authors of systemd aren't dealing with criticism well. They've allowed it to turn into a PR nightmare. Agree or disagree with them on specific issues, but they've allowed this to grow out of hand. There are nearly daily popular posts on HN, Reddit, etc. Most of these are bikeshed arguments, but if you have so many bikesheds that it becomes a daily issue on news aggregators, then you have yourself a culture problem.

2. Perhaps as a result of the above, there haven't been any limitations of the scope of systemd. This means that there is a lot of feature creep as stuff gets added in, and criticism gets swept away without considering it.

Unfortunately, I don't see an easy way to fix the culture problem. But it needs to be addressed.

EDIT: formatting

Posts like yours are one of the first steps. At this point we need to address the culture problem and the people contributing to it instead of focusing on the distracting fallout.

I've tried to keep an open mind about all things systemd but after finally reading some of the exchanges people have with the systemd folks, I'm surprised systemd isn't more widely rejected as a viable alternative simply because of the attitude of some of the people currently maintaining it. You can't be an butthole to folks trying to offer valid criticisms and expect those same people take your project seriously. That just does not inspire confidence, it inspires skepticism and distrust.

I wonder if it's simply a critical mass problem, if once enough devs get peeved they will fork systemd and lead it the way they see fit sans any caustic contributions or contributors.

It has been forked [1].

> uselessd (the useless daemon, or the daemon that uses less... depending on your viewpoint) is a project to reduce systemd to a base initd, process supervisor and transactional dependency system, while minimizing intrusiveness and isolationism. Basically, it’s systemd with the superfluous stuff cut out, a (relatively) coherent idea of what it wants to be, support for non-glibc platforms and an approach that aims to minimize complicated design

He could use better marketing, but I think removing the kitchen sink aspect of systemd and keeping the service file syntax is a step in the right direction.

[1] http://uselessd.darknedgy.net

No security professional will be bothered to critique "FSS" because anybody worth their salt know that any log ON a server is, by definition, not trustworthy. They've known this for at least 20 years (if not longer), and everyone uses syslogd or some variant thereof.

So, in that sense, there is no controversy here, and this feature is, for the most part, completely useless.

In the comments below, Andrew Wyatt is absolutely correct and this, "Lennart Poettering" individual doesn't know what they are talking about.

This is a neat feature, which I didn't know systemd had, and which I recall having wanted in other contexts. The criticism about the 15-minute interval is valid, but that's an implementation detail which, from the sound of it, can be fixed if someone puts in the effort. So while the author may have intended this as a pure criticism, it didn't really come off that way.

So the argument is: it's not good enough, so we need to throw it away?

No, I think the author's point was that people shouldn't misrepresent FSS as equivalent to logging to a remote server. They can compliment each other, but they are not the same, and thinking they are can lead to serious security snafus.

Exactly.

What I find troublesome with journald is that they ignored everything that was done before in the log signing space. There has been some research done, even an RFC published. They implemented their own solution incompatible with everything else. Any questions about their design was met with, at best, indifference.

Since then rsyslog gained a log signing solution, sadly incompatible, but which at least to my eyes look a bit more conservative in design. We don't really have any second opinions on journald. That would be fine if it was just a hobby project, but I find it problematic to include by default in stable enterprise distributions.

Systemd is starting to show the signs of feature creep which IMHO is good. Failing early means the ship can be turned around if there's enough will, or at least (newer) replacements of systemd(in case it goes really off the rails) will learn from others' experience.

This was added in 2012.

You mean the article or the feature? The article is dated by 22/11/2014. The feature may have been added a few months/years ago, but the point still stands about systemd showing signs of feature creep. Note also the DNS resolver issue[0]

[0] - http://seclists.org/oss-sec/2014/q4/592

The feature was added in 2012 or something, whenever the log support came.

The DNS issue affects not just systemd but any stub resolver.

The DNS issue is worth in systemd's resolver because it is NOT merely a "stub resolver" - it implements a cache. From the same thread:

"The DNS specification does not require rewriting of upstream responses to filter out parts for which the queried server is not authoritative. This means that a downstream caching resolver will tend to poison its cache if it adds data from such responses that are not directly in response to the QNAME."

http://seclists.org/oss-sec/2014/q4/602

Adding entries (including additional records, which are common in DNS) without validation is just asking for a DNS poisoning attack.

Won't happen. It's turning ugly (really the ship sailed on that with the container stuff), but it's never going to "go off the rails". It's quality software and it works.

I... guess I'm sort of in the "anti-systemd" crowd, but in the mildest sense. For the overwhelming majority of use cases, it really just isn't appreciably better. It's better in lots of measurable, small, and ultimately more or less irrelevant ways. Runtime system software maintenance just wasn't a problem worth solving, IMHO.

But... it's quality software with solid features and it works. The same logic says we should just shut up and take it. So I do.

I don't have any dogs in the systemd fight, but it looks like you're moving from 'anger' to 'bargaining' in Kuebler-Ross.

Log signing is nice to have, but ultimately not that reliable. Systemd has quite a broad attack surface, and if the attacker has root, you're fucked anyway. It might be a useful tool for poking local sysadmins.

compromising systemd, forcing it to re-sign the logs (assuming that practicable.)

Potentially you can have an alert that checks the signature, which might be useful. However you can do something similar now, but comparing log files (local vs remote) although its not that easy or elegant.

I've yet to use systemd in anger. I'm sure when we update all 25k machines to it next year we'll become intimately familiar with all its pains.

> compromising systemd, forcing it to re-sign the logs (assuming that practicable.)

This is completely unfeasible. The point of FSS is that you sign the logs, create a new key using your old one, and then permanently throw away the old key. Once this is done, no matter what control you have on the system, you cannot re-sign the logs without the tampering being obvious.

Reply on news.ycombinator.com