Log4j 2.15.0 – Previously suggested mitigations may not be enough
isc.sans.edu
isc.sans.edu
(The whole discussion why a logging mechanism should support the whole JDNI stuff is still perfectly valid).
It's not cool, but far less severe than the original issue.
The java ecosystem is full of these abstractions, that are not outright security problems in themselves, but give rise to complex interactions that makes it hard to judge what the implications are.
This particular logging system has been used by millions of developers for the better part of a decade before any one single person realized the actual implications.
It has been used by millions of developers for the better part of a decade before anyone disclosed the implications.
In 2.16.0 message lookups have been completely removed: https://github.com/apache/logging-log4j2/pull/623
Lookups now only work in configured patterns. Thats IMHO the way it should have been in the first place.
But 2.16 disables JNDI lookups entirely, so that they cannot be triggered via any lookups, including context lookups. But context lookups can still trigger other non-JNDI lookups.
The log4j vulnerability shares some common traits, but it's completely different at the core. It's not a problem of sanitization, and I'd argue it's not a problem of escaping either (though escaping could fix it). It's a problem with an obscure feature that is left with insecure defaults, and is (was?) unknown to the vast majority of developers who integrate the library in their apps.
Another example: A "transformation" that's a "sanitation" but not "escaping" would be replacing all occurrences of "<" with "<" (among others!). It surely doesn't add escape characters (e.g. \), but instead replaces the problematic substring with a replacement string that makes the string safe to display on a website. Of course you'll want to replace user-supplied "<" with "&lt;".
(btw, thanks for that it's "sanitation" and not "sanitization" ^^).
Now a Google search finds instances where sanitization/sanitation also includes techniques beyond filtering: https://www.webopedia.com/definitions/input-sanitization/ https://hack.technoherder.com/input-sanitization/ https://developer.wordpress.org/plugins/security/securing-in... https://stackoverflow.com/questions/129677/how-can-i-sanitiz...
But there are also results where it isn't really clear, or where the only sanitation technique considered is filtering. So I'd say "yeah, it's unclear and poorly defined".
Buuuuut: I still like my definition more, as I have a word for "all techniques that aim to make an input safe for processing" (sanitation/sanitization) while I can still refer to "destructive elimination of substrings" as just "filtering", which is a again different from outright "rejection of input" by using an "allow list" or "deny list". :P
I agree that splitting data and code is the way to go, if that's an option. But I didn't talk about that in the post you're answering to, so I'll ignore that ;-)
Agreed that this is not a sanitisation attack, but i think that means it is a format string attack:
https://owasp.org/www-community/attacks/Format_string_attack
It requires a couple of other loopholes, related to JNDI, to work, but that is the first step which goes wrong.
Now, I think format string is not entirely right, either. That's more related to `printf`, which splits code (the format string) and data (the varargs). But log4j actually mixes code and data into one string. In the OWASP frame work, I think the more generic variant of format string attacks would be https://owasp.org/www-community/attacks/Code_Injection (that's actually linked under "related" for the format string attacks).
OWASP lives mostly in the web world, but links to CWE-77 (Command Injection, https://cwe.mitre.org/data/definitions/77.html), which is pretty generic. And log4shell matches the description just nice: 1. Data from untrusted source: yes; 2. data is part of a string that's executed: interpreted, which I think qualifies as a yes; 3. the execution gives capabilities the attacker would not have other wise: oh yes!
So it's probably safe to claim that this could fall under CWE-77 (too bad CVEs rarely use CWEs).
Now, CWE-77 (Command Injection) is a child of "CWE-74: Improper Neutralization of Special Elements in Output Used by a Downstream Component ('Injection')" https://cwe.mitre.org/data/definitions/74.html. Which states "The most classic instantiations of this category of weakness are SQL injection and format string vulnerabilities." - that's probably why the both of us thought of the the two of these! :) And even if CWE-77 is to specialized (this could be argued) CWE-74 should be a good match, quoting CWE-74: "The software constructs all or part of a command, data structure, or record using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify how it is parsed or interpreted when it is sent to a downstream component" [end quote].
Funnily, CWE-74 is a child of https://cwe.mitre.org/data/definitions/707.html - "Improper Neutralization". Which says neutralization can be done by (among others): "[...] transformation of the input/output to be "safe" using techniques such as filtering, encoding/decoding, escaping/unescaping, quoting/unquoting, or canonicalization [...]"
:) Thanks for triggering me on this.
Also I said "similar class", not "strictly equivalent". Both instances are a case of "user input handled as code, not as data". Now with SQLi the problem is widely known (but sadly still happens) and we have techniques like parameter binding to outright avoid the problem. The same techniques are among the <edit>theoretical</edit> possibilities to avoid the problem with log4j.
Though I agree, the problem are the insecure defaults (JNDI enabled) and that seemingly every java dev used log4j but wasn't aware of the "${}"-expressions. I just pointed out that this "trend of neglect" [sry, non-native speaker here<edit>, but I hope you get what I mean?</edit>] seems to continue: Everyone and their CISO is now aware that attackers can abuse JNDI and the whole mitigation recommendations (<edit>e.g.</edit> remove JDNI classes) so far was primarily focused on that. Still, even if JNDI wasn't a thing, the "${}"-expressions can still be abused (it's just more specialized now and not a huge pwn fest), requiring mitigations beyond disabling JNDI. Just look at the reporting for more examples of ignorance: When this started to explode, there were some who thought blocking the string "${jdni" in their WAF would be enough (and maybe it helped against script kiddies), but a smart attacker just used/uses a string like "${${env:randomstring:-j}dni[...]" defeating the simple filters...
This is a form of sanitization.
How are they different?
I've heard people talk about this problem with analyzing windows event logs using Java-based software.
* probably not too hard to come up with a way to break one of the many "grok" pattern regexes https://github.com/logstash-plugins/logstash-patterns-core/b...
Which, if you're still on 1.2.x, also has an RCE against it.
More like blindly trusting that a logging library will just log the strings that you pass to it.
I speculate will be a cluster of newly discovered vulnerabilities that follow a similar pattern, possibility across a number of familiar Java libraries.
But some projects are little more interesting in that they include libs like this, or include libs that include it. It is part of their foundation/framework. So you can still put in override and ignore what the lib wants.
But now when it comes time to upgrade that top lib maybe there are 6 newer versions of the one problematic lib you were concerned about today, but now it is a year later and you forgot about it. What is the mental context to say 'oh yeah go back and check that'. You may have got used to ignoring the 'hey you are overriding that jar file'. Are you going to do that for all of the libs/jars this top level framework drags in?
It is not just java that has this sort of issue. NPM (though you can float the versions at least), nuget, rust, C++ containers, etc. A lot of projects out there have taken on this maven central style building. Which is a huge productivity boon. But in many ways has not helped us with dependency hell, and in some cases has made it wildly worse. I see simple projects pulling in 150+ items in some cases. What is the mental context of actually managing that? And have real fun if your project depends on a 'dead' project where there are little to no updates.
This one is at least high profile enough that many people are revisiting old build chains that have long ago been forgotten and updating them. That will kill out many other vulins that have been lurking. But many will just do that one jar set and call it a day. But you are probably right we are going to get see a few of these style of attacks.
Is there actually evidence for this? The 30 year arc of the industry to date that I am familiar with so far shows little sign of engineering standards in software codifying around anything.
If we are in the infancy stage in 2020, we are one very large infant. More likely to me is that in the future, software will change so much again that what we do today is unrecognisable; not because of codification but because of the inevitable tech churn between now and then.
However, log4j seems to be adjusting their design rather quickly, so they should be out of it rather fast.
Link to the project: https://github.com/gredler/aegis4j
It's not a lot of code, but it uses parts of the platform that I think are a bit unusual for most devs, so it was quite interesting to implement. Happy to discuss details, ideas, and concerns.
One idea for a possible improvement is to make the feature block list adaptive, i.e. watch what the application uses in the first few minutes of execution, and then shut down all unused "dangerous" features for the remaining lifetime of the VM. Not sure how reliable this would be though, especially for services which have background jobs that might only run once a day.
We don't even use log4j and yet are fielding a steady stream of requests to reassure clients that we've mitigated against it, which hasn't been an issue because we don't use it (although we do use log4net with Stackify and it's certainly a possibility that may be compromised in future), and of course are having to audit all our own suppliers.
I encourage companies to post information on their websites about this so we don't all have to keep emailing you (I'm trying to get some info up on ours detailing mitigations, for similar reasons; once it's there I can send some all company comms and hopefully never see another email in my inbox with "log4j" in the subject line).
Of course, this whole issue is crucially important to deal with, but it's also frustrating because it's an entirely non-value-adding time-vampire.
It can be "fun". :)
Oh, and the thingy you put your ticket in get the stamp was out of order on all the buses.
Made me wonder if that was due to some component involved was using log4j? I don't think it's very likely - why would the bus display or the ticket stamping machine be affected in a way that requires turning them off? But who knows, maybe somebody overreacted, or maybe these depend on some backoffice infrastructure that is temporarily offline due to log4j.
It was weird, though. This is the first time I remember a security vulnerability made the national news (in Germany, if it matters). I don't think even Heartbleed got that much attention outside IT news outlets.
It would be far worse if this exploit existed in slf4j or commons-logging.
log4j is in widespread use, directly or indirectly.
Most people don't give a crap about the exact Java logging library, the only places that actually care are those that bike shed, since functionally Java logging stuff is generally quite robust and full featured.
The worst part about my previous phrase is that the places that don't care about the Java logging library frequently don't do security audits, so prepare for a ton of exploits and data leaks in the following months and years.
It's not just your code that is vulnerable.
Our in-house software doesn't use log4j, but a number of third-party components we use do and needed to be patched.
did they also map the jndi feature ;)
https://github.com/cisagov/log4j-affected-db <-- that's a (probably incomplete) list of possibly affected vendors and applications, and in many cases the vendor can't even say yet whether they're vulnerable.
Oh, but it does turn out that we do have the elastic apm agent, and it has log4j as a shaded dependency. So not as free and clear as we thought, and damn hard to detect that.
In the case of the transient dependency, it's further complicated because we've seen that some packages repackage the JARs they depend on... that means that statically analyzing for log4j is very difficult because you can't use hashes (even if you unzip the jar and hash class files directly).
I've been working on a scanner for this stuff on GitHub[0], and it's a real pain in the neck lol. Especially for Vendor software that you don't control.
0: https://github.com/lunasec-io/lunasec/tree/master/tools/log4...
I prefer to package all my projects into a single fat jar, since it makes distribution so easy and clean, but if I was creating jar files with the expectation that other projects would use them I would make sure to package everything separately.
In this case using fat jars made things really easy to check for the effected classes and luckily for me I also didn't have any log4j-core classes in my fat jars.
My completely-unverified guess would be that there are more people immune to this issue because they never migrated past log4j 1.x than there are who are immune because they picked up Logback or something similar.
> You shouldn't need to pour over code to figure it out.
This is true but as sibling comments have pointed out, a lot of other software you might be deploying without having written or configured the logging for are written in Java.
Log4j2 never quite got the same status that v1 had. V1 should be considered a bit obsolete at this point. It still works of course but it has some performance issues that both log4j2 and logback try to address.
The issue with high profile vulnerabilities like this is that there are a lot of projects where dependencies are rarely updated.
I update aggressively on my own projects to stay on top of changes and keep the effort related to mitigating compatibility issues at a minimum. A nice side effect is that you get all the latest security, performance, and other fixes. In my experience, updates get harder the further you fall behind. So, the longer you wait, the more likely you will have a lot of fallout from updates and the more likely it is that you will be exposed to pretty serious bugs that have since been addressed upstream.
If you are like me, I can recommend the excellent refreshVersions plugin for gradle. It makes staying on top of dependency updates a breeze. I run it every few weeks to spend a few minutes updating misc libraries, and verifying everything still works. Run the command, update to the suggested versions, create a pull request and merge when it works.
Occasionally there are issues with specific libraries but 95% of the updates are completely painless and the remainder are usually pretty easy to deal with. And if there are show stopper issues, I want to know about them and document them why we can't update.
I would recommend doing the same for packaged software. I work with a lot of customers running ancient versions of whatever for no other reason than that they seem a combination of fearful, ignorant, and indifferent about what will break because they can't be bothered to even try. Mostly updating them to more recent versions isn't that big of a deal and it tends to address a multitude of performance, security, and other issues.
I did not know this, thanks for letting me know!
> I update aggressively on my own projects to stay on top of changes and keep the effort related to mitigating compatibility issues at a minimum. A nice side effect is that you get all the latest security, performance, and other fixes. In my experience, updates get harder the further you fall behind.
This is absolutely a best practice, though I think people struggle with it for all sorts of reasons. In general one of the downsides of maintaining a diverse codebase is that this constant update cycle becomes more and more difficult, and it's one of the things that I find drives towards more consistent tooling within a team.
> I work with a lot of customers running ancient versions of whatever for no other reason than that they seem a combination of fearful, ignorant, and indifferent about what will break because they can't be bothered to even try.
While I agree this is something people need to get over, we have to take some blame for this as an industry. A lot of people have bad experiences with upstream Shiny Object Syndrome.
Yet we have seen a large number of high profile hacks already, and they keep coming. Turns out that not only are there many ancient java versions in use, the aren't very compartmentalized either.
https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sm...