Third High Severity CVE in Log4j Is Published

This wasn't just a predictable scenario, it was predicted. Or more accurately, it has occurred already repeatedly in the NPM ecosystem, but for some mysterious reason those incidents were simply ignored by security teams world wide.

Instead of chilling them to the bone, they simply shrugged their shoulders and said "Well, we don't use NPM... I think. Probably?" and went on with their paper-pushing or whatever it is CISOs do these days.

If you think Log4j is bad, wait until Rust gets popular and has something similar happens with a commonly used crate. How would you scan for a vulnerability in compiled code that doesn't even have separate module files?

I had to help an organisation with log4j that had on the order of 5K distinct executables/applications across 3K servers on two clouds and three on-prem networks. At least with log4j it's a simple matter of finding JAR files and scanning their contents. They're just zip files! With languages that output a single binary by default such as Rust and Go, we would have been screwed. No way to scan, no way to self-help update.

Introspection and post-shipping updates for security are mandatory. Rust and Go like to pretend they aren't, because they originate from organisations that are in total, end-to-end control of the software on their own network. Google famously uses a monorepo and can build everything they run from scratch in short order. The shortcuts they can take with their post-release management will never work for ordinary organisations. Never.

So let this be a lesson: Log4j was made for a language that at least allowed us all to find the issue and fix it ourselves. We have Sun's forward-thinking and the enterprise-friendliness of the Java ecosystem to thank for that.

Githubs of the world could just gate downloads, pull requests etc. behind a payment to see what is the real valuation of open source software; I imagine it'd mostly settle around $0 excluding couple of big projects.

Security teams are stuck with securing the tire fire; they didn't choose the library or platform. If anyone should be advocating for supporting open source, it's the developers who benefit by using open source libraries.

Log4j2 partially exists because of pushback on adding features to v1

And that's a point in time audit. To maintain that value we'd have to redo the audit periodically.

It's just not gonna happen.

It was especially difficult for us because we’d shipped so much code that used the library, and replacing the library was unthinkable.

It's just not as simple as "security audit finds all the vulnerabilities, then you fix them." You invest X in the review, you get the results that X/(hourly rate) finds. This is a lot of software with a ton of configurability-- that's a lot of variations to review and test.

Now that someone found the first lump of gold and gave it away, there are thousands of eyes searching for the next one. These recent findings are all abuses of this same chain of functionality, just along different sets of settings. In another month we might have half-a dozen more of varying severity and scope. That _still_ won't prove that the overall library is then safe, but we will probably have a a little more confidence in this particular bit of crazy template formatting flexibility. Maybe not as much as we had had three weeks ago, but more than we do now.

Because "java:comp/env" is the way for a J2EE web application to get values or objects from the configuration outside its container. Yes, nobody actually uses the remote parts of JNDI anymore, but the local variant (within the same process) is still in use.

And that's what led to this vulnerability: someone wanted to be able to get these local values within the logging configuration, which is a valid use case. Not realizing that this opened the door to remote values and objects.

yea literally any library/dependency can introduce risk, including stuff coded internally for those purposes

There is also lots of stuff you might want a logging library to do that while not very hard would be annoying to do yourself for every project like log file rotation, switching log level while the application is running, optimizations (like in c/c++/rust you might compile for INFO level and drop all the DEBUG and TRACE level logging from the binary), different log levels for different parts of the app (including dependencies), etc

Transitive dependencies for one thing. I rather enjoy having my DB driver, connection pool, kafka client, and web framework all emit their own logs, which is not possible if I wrote my own logging library.

> I get that the tradeoff it is worth it for complicated things (e.g. crypto libraries). But logging, really?

Logging at scale is more complicated than people give it credit for. Personally I've had my services suffer noticeable performance degredations due to poor logging framework config. We've ended up with a fairly complicated logging config that allows us to carefully balance collecting as much developer useful information as possible against the need to drop excessive logs during periods of heavy activity. Oh, and it's gotta be converted into JSON for aggregation into our observability tools, and decorated appropriately so that our logs can be correlated with our application traces, because that is incredibly useful during incidents.

Oh, and it all needs to work no matter what threading configuration you throw at it. That matters a lot too.

Do we need all those things? Arguably no, but then again they are helpful tools to have. Without them, the team would have a harder time developing on our systems. You have to understand the use case that people need to fulfill before dismissing the entire thing as unnecessary and pointless.

> Software development culture today is too quick to adopt a huge tree of dependencies of unknown quality, rather than thinking about how to minimize dependencies to only those truly necessary. The leftpad fiasco was but an extreme example of this, but I see it all the time, and it seems probable that there are hundreds (maybe even thousands) of similarly severe problems out there in widely used dependencies that we just don't know about yet.

I see this argument a lot, and I think it's a great example of "grass is greener" syndrome. People don't think through the practical consequences of drastically reducing their library usage. I personally worked on a project that had basically no external libraries due to the language choice, and we wasted inordinate amounts of time trying to chase down bugs we wrote in things like our database library, localization library, and yes, our logging library. Looking back I can say with very strong confidence that what we were doing was a massive waste of the company's resources, and rewriting it into Java or C# would have been the right call, even with the complexities those ecosystems bring to the table.

Yes, using external libraries comes with tradeoffs. Fixing bugs like this is one unpleasant possibility. But let's be clear that the alternative is a massive drop in developer productivity to write basic functionality that already is implemented elsewhere. Having been on both sides of this equation, I can confidently say that you'll spend way less time fixing security issues as they arise than you will reinventing the metaphorical wheel.

(I do agree that leftpad is an example of going way too far with libraries, but I think the library culture for NPM and the Java ecosystem are quite different).

Logging facades like slf4j help a lot, but then you still need to implement the required interfaces. For something like the specific Log4J CVE, things would be significantly worse if it weren't centralized in a library. They have a bug in deserialization when talking to LDAP servers, allowing a compromised LDAP server to execute code remotely. It's likely that a subset of libraries would likely have rolled this functionality independently, and then you have many libraries that all need to be patched and updated instead of one.

In terms of features, one that comes to mind log rotation.

I dunno, I suppose all together I’d answer your third paragraph to the affirmative — I’m more productive and the libraries improve significantly within crucibles like this.

The larger issue is in javas dependency system are built with dependencies on these libraries, so if there is no commons logging (or bridge, ala slf4j), then the program won't run.

There are a few other techinical traps with the jvm, like using system.out logging can cause issues with threaded code, or having one giant static logger being imported all over the place conveniently (this is before inversion-of-control became the dominant paradigm) had a few shortcomings, especially as programs grew. Ultimately, though, if you were going to use some third party libraries or app servers, you were almost certainly going to get pulled into the commons-logging vortex, which means you're now configuring that AND your homespun logger.

There is also the case for things like SOLR, where it might be just being used out of the box, but the distribution includes the affected JARs, even if you're just posting and retrieving documents with a python script doing print-logging.

Honestly, I didn't even know log4j2 was a thing, I've moved as much over to slf4j/logback as I could ages ago because of the madness of the JDK logging ecosystem.

(1) Coordinating with dependencies requires at a minimum a common API (which, in a statically typed language, means a library providing at least the interface).

(2) Configurability to log to different destinations and with different formats, which you eventually tend to want on major or long-lived projects, can be fiddly, as can the specifics of different formats and targets. Each of the bits is simple, but added up it's a lot over time, and reinventing the wheel, the axle, the tire, and the air pump for it just doesn't make a lot of sense. Especially given the need for a library for #1, it's best to get the core functionality together with it, and the peripheral pieces either together or as plugins, and only solving unique problems, if you have them.

Sir, you might have just extended all of our careers by at least a decade. We salute you.

I agree that developers are too quick to put themselves at the mercy of 3rd parties, but it's a risk-dollar trade off we'll continue to make because reinventing the wheel each time would sap our margins.

A lot of these libs separate logging into layouts and parameters. If you log input parameters as layouts. You might break the logging. I have seen this at every company I worked with.

The only really surprising thing about log4j is the remote code execution.

Log4J has been around 20 years. Log4J inspired and was lifted into the JDK. JDK logging is essentially an inspired copy of Log4J. The Sun coders were no better or worse than the Apache coders. I use JDK logging personally simply because it's one less dependency. JDK logging hasn't change since JDK 1.4, and is weaker than, well, pretty much anything else. But, with a simple wrapper I've used for 15 years, it does most everything I want from a logger.

That said.

Log4J has been, and is still, a boon to the Java community. Arguably, Log4J is the root of a tree of vast array of logging frameworks, across languages. Java server developers essentially live and die by their logs. It's routine for developers in dark rooms with screen lit faces to pouring through logs with endless stack traces. Thank heavens for Java stack traces.

Logging is part and parcel to the Java server side experience, and even the client side, and we can place much of that on the shoulders of giants like Log4J because it made logging easy and set the stage. It's so helpful, so useful, so flexible (obviously, perhaps, a bit too flexible), and so powerful.

Because Log4J inspired the other loggers like the JDK Logger, we have logging shims. Shims like Commons Logging, that act as intermediaries that can have adapters written so that we can use other logging frameworks. Log4J was one of the first, was, and is still, dominant in the community, but it's not alone, and thus a bad choice for things like libraries. Instead, those choose the shims like Commons Logging that developers can use to configure to route through Log4J or JDK logging or any of the others. That said, even programs that use Log4J directly can be routed to other loggers.

This is all entrenched. It's part of the flavor of server side Java, configuring the different logging shims to write to your logger of choice on your system. Just the way it is. But, that's what happens when you don't live in a mono-culture. Feature, not a bug, and it helps empower the vast array of software that millions of developers and applications rely on everyday. Nobody designed it this way. It didn't start this way, it just evolved this way. It's a very "Java" thing.

I watch Stack Overflow questions and 90+% of the time when someone asks "How do I" what they mean is "What library do I need", not "How can I write this". This is the sign of the times, and Java is not alone. The beauty of Java is that it made this kind of sharing REALLY easy. REALLY REALLY easy.

Apache has a solid reputation for good projects and good code and good stewardship. It's not a back alley transaction to grab an Apache Java jar file and shove it in your project. Is it all perfect? No, but what is?

The problem with rolling your own is that, for a non trivial technology like a producionised logging framework or some encryption library, you're highly likely to write a massive security hole yourself. The great thing about battle-tested open source software, used by 1000s, is that years of real world use, development, bug fixes and general scrutiny from a large number of people makes for a high level of reliability normally.

Obviously here we have a problem with this approach but avoiding third party dependencies carries its own risks and obvious costs too. Let's focus on how reliable so many of these libraries seem to be and how rare problems of this scale are.

I think a bigger problem here is the fact that so many people are using so few libraries. Maybe we need more open source libraries to counter single points of failure. As with how we deploy our code on the same clouds and so on, single points of failure abound these days. Things fail however well they're engineered and therefore we need diversification of everything.

Rolling your own logging library to reimplement it is not trivial, even if you ignore the more esoteric output formats.

Sure, in retrospect they went too far with some of the dynamic logging features in log4j2, but it is no exaggeration to say that log4j revolutionized logging in the Java world when it first came out, and heavily shaped later solutions like logback and java.util.logging.

That said, I do agree that there is a problem with large dependency trees. It seems building a simple hello-world app with Maven ends up downloading about 1/2 the internet. :)

Okay, more seriously... this is why I am currently staying away from Rust. Because sometimes I don't even know if the random number generation crate I imported for my banking app came from Rust team or some random dude in zanzibar.

(Sorry, I know HN loves Rust, so do I. But I currently that the cargo/crate system is a bit too easy to use and misuse).

Then there's the user-agent header, which also can be pretty valuable to debugging and other operations. Although this one is easier not to log.

In my current logs, I'm seeing attempts to compromise log4j vulnerability in both URL query string and user-agent. (Which of course I would not know about if I weren't logging them). This particular application has no Java involved, so is not vulnerable.

The fundamental problem is that, pretty surprisingly, log4j was running the same parsing and logic on user-entered strings as they did format strings. That's essentially the root cause of all this - log4j shouldn't be attempting to parse this data at all.

So many severe vulnerabilities are due to the complexities of parsing potentially malicious user input - just look at all the severe iMessage bugs. While many other apps do have a fundamental requirement to parse this input, there is no reason a logging library needs to do it.

Log4j is not part of the JDK and is an open source project with only few regular contributors. There is really no reason to infer any correlation to other parts of the Java ecosystem.

No, it’s not.