Comment by bob1029 | Hacker News Reader

Comment by bob1029 | Hacker News Reader

Parent Full thread

I have taken some flak over the years for rolling my own basics like logging, telemetry, tracing, authentication, session management, etc...

Incidents like this remind me that it is sometimes OK to ignore those who constantly order you to "vendor it out" over some notion of principled development excellence wherein one never reinvents a single hypothetical wheel. Arguably, writing text to a log file on disk is a simple thing you dont want to keep implementing over and over. But, it is also a simple thing that doesn't take a whole lot to do correctly in many use cases.

DIY (in aggregate) would represent one of the largest barriers to any attacker producing a wormable/DDOS exploit of this log4j-style vulnerability. Imagine ten thousand plus unique implementations for how to get a string to disk and I can almost assure you that not one of them would contain an attack vector like this. Even if one of them did, the other 99.999% of implementations would be unaffected and nothing would have made the headlines.

Depending on the code paths that others have developed is the biggest liability in all of software engineering. We are going to have to get better at evaluating this cost relative to the benefits. This doesnt stop with Log4j. There are other things more ubiquitous and more complicated out there still. IMO, the best we can do is to stop producing new software that is exposed in these ways.

Well it sure does suck to find out that you have a RCE vulnerability in your application thanks to a dependency, but I’m not sure that means it was wrong to decide to use Log4j based on the information available at the time. Having to rewrite a bunch of library code in-house is a significant burden, without a guarantee you’ll get it any more right than the Log4j developers did.

I don’t think this event is sufficiently bad to overturn the principle that code reuse is a good thing.

The better lesson is to learn how to contain the behaviour of your dependencies so you don’t end up with ridiculous situations like backend database servers being permitted to make JNDI calls to the public internet.

To take the example of log4j, I can see this going two ways.

On the one hand, I've worked at a couple places that used homegrown logging, and it's been fine. They covered all the features I use/want out of a logging framework, in only a couple hundred lines of code. I suspect that both logging frameworks even had lower TCO than their respective platforms' name brand loggers, because the greater simplicity paid recurring dividends across the whole organization.

I suspect, sometimes, that arguments against DIY being too complex have an unstated major premise that you need to implement a significant fraction of the off-the-shelf library's features. In my estimation, it's more like 10%, and then the code is only maybe 2-3%, because you also don't need any of the code for managing how the features you do need interact with all the features you don't need.

On the other hand, it's not like not using something like log4j yourself is any protection from things like this. We don't use log4j where I work, but I've still had my hands full dealing with all the other software we use that relies on it.

> I don’t think this event is sufficiently bad to overturn the principle that code reuse is a good thing.

Are we open to the idea that code reuse is a delicate spectrum of possibilities?

I am also curious what your threshold would be for something that is sufficiently bad to force a review of these principles. Last I checked 10.0 is the highest CVSS score available.

Exactly. Judging a decision solely by the outcome is called "resulting" and should be avoided.

On the other hand, you don't know what vulnerabilities are hidden in your own code and just haven't been discovered yet, and you're one fuzzing attack away from a vulnerability being discovered/exploited.

It really depends on the code. Proprietary custom code of course is less reviewed, but can also be much simpler and have a much smaller attack surface compared to a sprawling code-base like log4j.

For me the trade-off is when understanding the complete architecture and important subtleties (often ill-documented) of a third-party library takes more time than writing reasonably straightforward custom code restricted to the current needs of the application.

For context, the log4j2 repository contains over 3,000 files and 175,000 lines of code. log4j-core alone has 1,100 Java files containing 82,000 lines of code.

That's a pretty lush habitat for gremlins.

What if there are none?

It's not like library code is some magical stuff that only senior principal ninja 10x developers can comprehend

Not everything is don't roll your own crypto.

> Not everything is don't roll your own crypto.

Even for crypto, I have had people give me trouble for daring to use Microsoft's public & documented implementations of SHA256 and CSPRNG to derive session secrets. It's almost as if directly interacting with any crypto primitives is tantamount to rolling your own crypto these days.

But splat 20 lines of obtuse openssl commands into a bash script and no one bats an eye!

>What if there are none?

The problem is that "There are no bugs" approaches an unproveable statement as a codebase grows into even a modest size. It doesn't matter if you're a 10x developer or not, you aren't always going to immediately see all the implications and edge cases of a few hundred++ (at least) moving pieces the first time around with new code, and each additional piece adds a non-linear amount of complexity.

That doesn't mean you never roll your own version of something, it just means that you should either be very very good or have a lot of experience (or both) when making that decision for anything that's non-trivial. (And deciding what is/isn't trivial is its own decision too)

you're in responsibility land then. That's not where they buy IBM to avoid getting fired.

You get paid not for being smart, but for just doing responsibility for standard engineering. Plain craft. And do it seriously, not for admiration but payment.

Ah, responsibility land. More freedom to do as you want, and more risk because you can make mistakes and then have to face the consequences. Though it also often means you get to work on things that are a bit more interesting as well.

youre doing something incredibly wrong if there is a threat that your system can be taken over by logging. and java enterprise object store late bound remote naming system registry garbage non-solutions to non-problems are the thing that is being done incredibly wrong here. your opinion is literally just an admission that you misunderstand basic software engineering principles like pretty much all programmers

edit: on second thought, your sentiment makes sense in your world where most programmers live where most libs are doing asinine stuff because some enterprise dude needs it for his use case which could have been solved 500 other more saner ways. in that world, yes, you should always be afraid of anyhing, even a printf (hint hint DOTS soon)

You don't know what my world consists of, so I'm not sure where that comment comes from.

My comment was addressing the general sentiment from the GP regarding the potential presence of bugs in software, not the log4j issue in particular-- plenty if other have done that and I won't rehash it.

And of course your comment seems to ignitre the part of mine where I say that rolling your own version of something can be okay, but it's a decision to be made very carefully, along with my implication that trivial things are a different category. (your printf straw man would be about as trivial as it gets, so you're not making a string argument against my comment here)

And this is not just for issues of bugs, but also time and resources. Why spend your time writing a complex library for something that is not part of the core of your business if a well established OSS alternative that has had countless eyes reviewing the codebase, and it can do all you need? I don't go rewriting grep or other utilities. I don't roll my own equivalent of a nearly decade old library like python Requests. I use well established tools from trusted sources and then get down to my real work.

This is pretty much how most professions go about their jobs: use established tools, get to work. If there's a difference it's that many other engineering disciplines have had more than ~75 years to refine their established processes and basic tools.

Also, code written for libraries often need to factor in far more use cases and edge cases than code used for one part of your app. Its not too far fetched to say that there might not be any when you write your own because it might be significantly less encompassing.

Logging isn't an application that typically results in RCE (outside charlatan code, and yes I'm referring to people who call themselves senior enterprise architects).

On the other hand, if you roll your own logging but import a dependency for something else (you generally can't roll your own everything) and that dependency uses log4j, isn't that strictly worse from a security perspective than just using log4j everywhere? Whatever the probability of any given logging utility exposing an RCE exploit, the probability that none of your logging utilities expose an RCE exploit decays exponentially to zero as the number of unique utilities you're using increases.

I don't think that's necessarily (or even probably) true in this case, since the vulnerability only exists where attacker-controlled data gets logged. If a library is just using log4j for some diagnostic messages with no user data, there's no issue, right?

> if a library is just using log4j for some diagnostic messages with no user data, there's no issue, right?

Depends on the diagnostic messages and how you use the library. If exceptions are being logged with any level of detail, and an attacker has the ability to provoke exceptions while varying error message content, then the thing that seems safe was just made quite unsafe.

The way I would look at it - If log4j methods are being invoked on any logical threading context which is serving a potential attacker, then you have something you need to investigate for safety. This is not a complete scope, but it is the most immediate because of the feedback loop being instant for an attacker. Other flavors of this attack could involve nightly log shipping and other batched procedures being poisoned with nasty messages, but these have longer cycles and will take more time to compromise.

> If exceptions are being logged with any level of detail, and an attacker has the ability to provoke exceptions while varying error message content, then the thing that seems safe was just made quite unsafe.

Right, I would certainly include that in "user data" though. I was mainly responding to the idea that using two logging libraries is strictly worse than one: when log4j is only used for logging safe data, and another library elsewhere, this is better than using log4j everywhere.

Only if you send just as much untrusted input there as to your DIY logging. If it's a transitive dependency that's unused or that's only used for some logging happening on calls to that library, the risk is lower.

Given many of the engineers i have worked with and much more importantly the PMs and Managers that work over them, I would expect vulnerabilities like this to proliferate all over the place if people were to roll their own logging and telematry and especially authentication.

I mean SQL injection is such an easy known mitigation yet is still on the OWASP top 10 even after so many years.

Yes, people still forget to

  ; delete from comments where id = 29544262 */--

sanitize their inputs.

...still there?

The humor here is that sanitizing inputs is not the proper mitigation.

but? It think it's far deeper than mitigation, it's the solution.

Edit: I apologize for getting 'sanitation' wrong. Don't do it.

For databases you simply parametize the inputs so that code is code and data is data and there's no mixing of the two.

Sanitization is a defence of last resort when you simply can't separate code and data. Usually used for user content on the web since HTML has no formal mechanism to separate code and data because the angled brackets that do this separation are also valid user input.

But databases do have a way to separate the query from the data. Parametize your queries.

Indeed. That's enforced system boundaries.

The proper solution to SQL injection is parameterized queries, not input sanitization, to my knowledge.

The irony here is that if you use the log4j equivalent of parameterized queries, parameterized logging strings, you're still vulnerable to this CVE, even if you did everything right.

> sanitize their inputs.

For anyone confused about why "sanitizing your inputs" isn't the right approach, please read (shameless self-promotion, but I think the concept is important): https://benhoyt.com/writings/dont-sanitize-do-escape/

I've pretty much always taken 'sanitize' as a catch-all for all of the things you need to do.

Fair enough on that part. But it's the "their inputs" part that's just as problematic: whatever massaging you do to your "inputs", they'll always be unsafe in some contexts. You need to encode/escape your output.

it's all about system boundaries. Would you have arbitrary racoons in your closet?

It seems like you're saying that I should let the racoons out of my closet. I'll consider it, but then getting my clothes every morning will be a lot less exciting.

I'm not sure that an argument from monoculture is the right way to go here. It's merely a form of security by obscurity, and probably shouldn't be the dominant factor in whether or not you take some amount of mythical man hours to roll your own code vs. adopting a piece of very well established OSS that has had countless eyes on it over the years.

How do you know yours isn't vulnerable?

you're advocating diversity against monoculture. Only nature has muse enough for such.

I feel pretty good about betting on nature in the long term. Life in general has a pretty decent track record of surviving, even when complex life forms don’t.

The argument of monoculture vs diversity may actually be a short term vs long term advantage discussion.

But it’s been too long since philosophy classes, and not enough of them in any case. :-)

To be fair, log4j offers more than to write a file on the disk. Logging is also quite relevant for performance as these I/O operations are a bottleneck in many applications.

I think we cannot prevent bugs like this, but mitigation has to become easier.

Reply on news.ycombinator.com