Log4j 2.16: Certain strings can cause infinite recursion | Hacker News Reader

Log4j 2.16: Certain strings can cause infinite recursion

issues.apache.org

226 pointsscblzn173 comments

We just released Log4j 2.17.0 which addresses this issue: https://logging.apache.org/log4j/2.x/download.html

Good job! I really don't envy you right now, being in the center of attention like you are now. Take care!

Awesome work.

I read a lot of negative things about log4j these days, but people forget how great log4j was compared to whatever the JDK provided us with. It served a useful purpose for most developers, and inspired many libraries in other languages.

Take care!

This looks worrying, but if you read the issue thread it seems that this can only be triggered if you can edit the pattern string of the logger. This is something you can only do on the server itself (if made configurable) or in the build artefact that you deploy.

From a quick glance at the comments this looks like a minor issue due to the attack vector being very, very small — i.e., the attacker must have access to where the logging pattern is defined, and if that is the case, this attack is probably not the most worrisome they could pull off.

I hope I'm not wrong, otherwise we'll be patching everything again.

> This looks worrying, but if you read the issue thread it seems that this can only be triggered if you can edit the pattern string of the logger. This is something you can only do on the server itself (if made configurable) or in the build artefact that you deploy.

Plenty of existing code does things like:

    log.info(“foo: “ + request.getFoo());

Rather than using fixed format strings and {} place holders. You’re not supposed to, but it’s far from uncommon.

Up until last week, I thought you’re not supposed to do this out of performance reasons (string needs to be evaluated even if not logged). That log4j does anything else on this string than writing it to some log destination is the real problem and the cause of all these bugs…

It's also a very bad idea in e.g. C, where printf has all kinds of exploitable behaviour if the attacker controls the format string passed into it. Format strings are a kind of code, and should be treated as such when considering injection attacks (by far the worst aspect of the log4j bug what that it would expand these strings on the final formatted strings: this was completely unnecessary for the features they were aiming to support and made the vulnerability even worse).

Since version 2.16, log4j doesn’t perform such substitutions on the message string anymore, unless explicitly requested by the log pattern in the configuration (i.e. by `%msg{lookup}`).

Oh yes, I’ve seen several programs that spend ~30% of their CPU cycles formatting strings that are immediately thrown away because the log level is not high enough. Now you can also include vulnerabilities with no extra effort!

In Java? Surely one of the benefits of the JIT is to compile logger.debug() into a no-op if the log level is not high enough.

You would still be doing the job of concatenating together the error message, possibly rendering some complex objects to a string, that is then fed to the no-op. The point parent was making is that there is also a performance aspect to this, in addition to the security aspect.

> You would still be doing the job of concatenating together the error message, possibly rendering some complex objects to a string, that is then fed to the no-op.

Why would you still do be doing that work? If a value goes into a no-op, then the value isn't computed. (In theory - I'm sure it doesn't always work out 100% of the time in practice.)

Unless the JVM is sure the logic is not effectful, it couldn’t eliminate it.

The JVM knows what string concatenation does.

But what if one of the arguments is a function call? Then it isn't easy to prove there are no side effects

If it’s too big to in-line then yes.

That’s says absolutely nothing about whether the function has a side effect.

If you can inline it into the compilation unit, then you can see if it has side effects or not.

Isn't the point that you don't generally stuff your log of things that are only strings but of things that can become strings? (Asking since I know of your work with truffle)

I assumed things like the names of resources that are already strings because you’re using them in the actual program?

A typical logger.info("user {} did the thing", user) can skip the actual string interpolation and user value stringification, if the log level is not > info. However, logger.info("user " + user + " did the thing") cannot avoid at least the execution of user.toString(), even after jit optimizations, unless the jit could prove that toString does not have side effects. But I don't believe the jvm jit tries to do that. Am I wrong?

If user is a string, then #toString() is a trivial method, will inline, and become a no-op.

However, if user happened be an user-defined User object it would be hard to guarantee that it could be inlined

Yes

`"hello" + myFooObject.toString()` Can you eliminate that? It's impossible to say without knowing whether or not `myFooObject.toString()` is effectful. Maybe the JVM can make that assessment. Maybe not. I don't know, but it's not as trivial as "knowing what string concatenation does".

A straightforward way to do this is for Java to assume built in types like strings are side effect free, but custom types may not be. This would likely cover a good portion of log lines but not all, obviously.

The noop determination is made during runtime within the log method called. So, it's LOG.info(obj.toString()), wherein that method inspects the log level setting, then decides whether to ignore.

Are you saying that Java does or should "optimize" in such a way that it branches to the logic within each method called during runtime, inspects which code would be executed per possibly mutable runtime values (like log level), considers possible side effects (in-method and in parameters) then decides whether to even invoke the method at all?

There's an even simpler reason why the JIT compiler can't prune it: it's possible to dynamically change the logging level at run time.

That is actually a reason that a JIT compiler can prune where an AOT compiler can't. JIT can deoptimize when assumptions change.

One of the bigger wins with the JVM is assuming that a virtual method can be called statically if there are no derived classes. That's a huge win for every leaf class in the class tree. The assumption can change every time a class is loaded of course. The optimization is called devirtualization and it can be combined with inlining to get even bigger wins.

Fair point... but it seem like, even so, doing that to monitor whether a single integer variable might change sounds like a lot of added complexity. Are there other use cases that would help to justify it? Reducing the impact of failing to follow logging best practices doesn't seem like an obviously sufficient cause.

It's a standard part of the JVM https://docs.oracle.com/javase/7/docs/api/java/lang/invoke/S....

Logger levels are mutable to allow switching them at runtime, JIT can not make an assumption that the log level will stay the same.

Sure it can, since the JIT has access to runtime information and all the code. It could reorder the steps so first the log level is checked, then that block is run.

log.debug(“foo count: “ + fooHandler.incrementAndReturnFooCount());

Should the JIT call incrementAndReturnFooCount if debug logs are disabled? This is a long-recognized pitfall of C preprocessor macros that look like function calls but may simply be defined away, causing unexpected behavior.

It should call it if it has side effects, otherwise not. Java isn't C; the runtime can be a lot smarter.

I guess I’m not up on the state of the art with respect to the JRE’s ability to determine whether arbitrary code has side effects. That’s quite a broad problem. If I write fizzbuzz but use log.debug for output, does it optimize my main() to be empty unless I run it with debug enabled?

I believe the problem is just that the toString methods implicitly called when concatenating strings and objects, can execute side effects. And the JIT has to keep those effects if it cannot prove they are just heating up your CPU

That struck me at first as bizarre - but is it because one can override a toString method with something that has side-effects? (Or give a user-defined class a tosSring() with side-effects, I suppose.)

Well, yes, you could simply put a System.out.println in there and it would be valid java

I'm confused. Do people not overload toString() on a regular basis along with .equals() and .hashCode() anymore? Common case was prettyprinting of deep object structures, and yes, technically, you could update state while doing it, but dear God, why? Hell, API method names don't actually mean squat as far as the JVM is concerned. It's just the consensus of the community to go with convention over config. If everyone does the same thing with methods of the same name, it decreases cognitive overhead.

Did I fall into an alternate dimension where 90% of Java seems to only use preexisting objects now, and people hardly ever write their own code?

(No disrespect to the asker; it's just such a surprising question to stumble across, that mentally speaking, I had to deoptomize my mental model of the world to accommodate for people who may not have worked much with Java).

I get functional is all the rage, but wow. Default toString() impl is inherited from Object, and basically gives you class and instance number.

> JIT can not make an assumption

The whole point of a JIT is that it can make assumptions. 'Assumption' is literally the term used in the JIT to track things like this.

Just because a JIT is "making assumptions", it does not mean it could or should make the assumptions needed for what you are calling for here. Generally speaking, optimizers should only be making semantically-neutral assumptions (optionally under the further assumption that the code fully conforms with any constraints the language spec. puts on program behavior, which is somewhat controversial in C.) Furthermore, a JIT can only spend a limited amount of resources in determining whether skipping the evaluation of an argument expression would be semantically neutral. (At this point, functional-programming enthusiasts are rolling their eyes...)

If logging was part of the language, one could simply rule by fiat that arguments shall not be evaluated unless logging is enabled, but log4j is just another collection of user-defined classes, and gets no special treatment.

You might want to refresh yourself on who you're talking to here on this one. I'm sure that there are certainly people more familiar with the JIT behavior of HotSpot and Graal out there, but I don't know that any of them post here on the topic.

I'm not saying that when it comes to the JVM's optimization behaviors that I'd jump without checking that there's water on Chris's assurance, but I'm not saying I wouldn't either.

I am duly chastened, and ready to learn what boneheaded mistakes I am making. For one thing, I take it that that the analysis is tractable for most practical cases.

It occurred to me today that, somewhat ironically, JNDI+LDAP calls appear (at least to me on first sight) to be prime examples of expressions that would be difficult for a JIT optimizer to analyze for side-effects, given that this is a mechanism for making remote calls. If I am mistaken in so thinking, I would be genuinely interested in learning about how it is done.

To be clear, I am not suggesting that these sort of expressions are inherently insecure - that would depend, I think, on whether they involve user input in a way that allows an attacker to take control of the call. A hard-coded JNDI lookup from the arguments list of a log4j call might be inefficient, but no less secure than if made anywhere else.

You might want to google who Chris Seaton is lol.

It can only make safe assumptions.

Assuming a global mutable variable will never change is generally not a safe assumption.

I suppose, at least in principle, that something like the GraalVM AOT compiler has the option to scan all of the code to verify that it never is mutated. But HotSpot cannot, because it only JITs code as it is loaded.

> Assuming a global mutable variable will never change is generally not a safe assumption.

Put your logging level behind a switch point and C2 will treat it as a constant but still let it be changed - that works today.

Obviously with your background you know this offhand, but is there a mere-mortals reference for JVM JIT behaviors and capabilities to keep in one's back pocket? I have a pretty strong understanding of a lot of the Java universe (particularly around the JMM) but modern JVM JIT behaviors are a black box to me.

Actually this is not true. Hotspot can deoptimize code. It optimizes making some assumptions. If these assumptions stop holding, code gets deoptimized, measured for a while, and reoptimized with new assumptions.

For example, an interface implemented by only 1 class might get the class inlined. If a second implementer pops up (which can happen at runtime for e.g. some dynamically generated class), all of this will get undone.

There was a series of small articles with all these things, but I can't seem to find them right now.

>Logger levels are mutable to allow switching them at runtime, JIT can not make an assumption that the log level will stay the same.

Java HotSpot can make and verify that assumption. And switch code when the level changes.

So you've moved the IF statement that was checking the logging level from the program code into the JIT. Great. No saving. (And I doubt it actually does this - as it's not gaining anything)

You didn’t move the IF. you literally removed it.

If (a) DoExpensiveStuff());

Becomes

When modifyA() RecompileIfWithNewAValue();

As long as you call the initial if more often than you modify (a), you’re fine (and the JVM was able to see that you called your if 10k times without modifying the value of (a) even once)

Respectfully, your post is very wrong :) JIT compilers are very fast and run once to prep the code that will run many times. If you had flat code that was always run exactly once the games would be lesser, yes

How could it? The log level is not a constant. It is evaluated at runtime, every time. You can’t compile that out. Am I wrong?

The JIT works by making assumptions using runtime information, and by discarding compiled code when the conditions change and the assumptions are not valid anymore.

JVM is not that smart.

Years ago we had a system that was performing really badly. It turns out that it spent over 50% of its time on

log.debug(xmldoc)

The debug method took a string, and Java was converting the xmldoc to a string. This was back in the 1.4/1.5 days.

That makes no difference for the ${…} substitutions, which also aren’t applied to messages anymore. The new issue is about log patterns defined in the appender configurations.

I thought log4j performed no escaping so even if you use its logging calls that take parameters you’re still vulnerable?

That was changed with version 2.16. The lookups are not applied to either the message format or the message arguments anymore, only to log patterns defined in the configuration.

This is why Python's logging module treats the argument as a literal string without performing expansions if you only give it a single argument. That avoids such issues.

This is really making me feel better about badgering our contractors to use format strings for effective logging to Sentry.

Using format strings makes no difference for the log4j vulnerabilities. The stage at which the vulnerable substitutions are applied is after the application-level format string has already been formatted. The vulnerabilities occur at the level of applying the Appender log pattern, which is a mechanism separate from the application-level format strings.

Ah, the person I was replying to had conflated the two? I retract my feeling of relief, as is the course for this week.

To be fair, most descriptions of the vulnerabilities and of the fixes do a bad job of explaining when and where the substitutions are performed and how they (don’t) relate to application-level format strings.

It seems like the ability to declare a parameter to be a constant string (no runtime computation allowed) would have been useful here, or in any printf-like API. If you really want to do that, it could be using a different function.

I think Go's log package, and similar, have the best take on this:

    log.Print(fixed_string)
    log.Printf(format_string, args...)

Simple, unambiguous.

Well, almost. If you change Print with one argument to Printf, the compiler doesn’t complain. Maybe lint or vet will warn about it.

If it were a special type then the compiler could do it.

Just for the record, it is possible to type check format strings with regard to their actual arguments. OCaml does this, for instance, probably since its inception ~30 years ago or so.

Yeah, that's how I read it. If you're on 2.16 then you have to put a crazy substitution in your config, which no one is going to do.

So if this is true:

> In 2.16.0: if the suspect string is put in the PatternLayout, then that specific patternLayout will crash when loaded and replace itself with a PatternLayout that just logs what is handed to it with no formatting

> logging the suspect string seems to have no affect, and it is passed untransformed to both System.Out or the file I specified as expected.

Then not a big deal. I'm sure there's a number of ways I could mess up the config file and end up with no formatting. Should I be submitting CVE's for those too?

There's a new CVE filed for it, just now. And here are the others for reference.

12/18 - https://nvd.nist.gov/vuln/detail/CVE-2021-45105 Score: -

12/14 - https://nvd.nist.gov/vuln/detail/CVE-2021-45046 Score: 3.7

12/14 - https://nvd.nist.gov/vuln/detail/CVE-2021-4104 Score: 8.1

12/10 - https://nvd.nist.gov/vuln/detail/CVE-2021-44228 Score: 10.0

CVE page for 45046 says that the score is being revised. Log4j self-assesses 9.0/10 as the new score: https://logging.apache.org/log4j/2.x/security.html (see under Fixed in Log4j 2.12.2 (Java 7) and Log4j 2.16.0 (Java 8)).

And the new CVE-2021-45105 is self-assessed to have a CVSS of 7.5/10 (see the same page above).

What surprises me is the scoring... for a non-default config. Seems a little high, but these seem inflated, maybe just because of the attention.

The scoring has little merit in general.

Particularly with the `${ctx:...}` vs `%X{...}` distinction. For a plain DoS, which only affects `${ctx:...}` usage? I wouldn't panic, fix it if someone manages to actually exploit it...

It's goods new so far that with more people/time and attention paid to the log4j exloits, the vulnerabilities are just getting narrower in scope and lesser in impact.

I think the scoring is done from the assumption that many users may be concatenating a log string instead of formatting it. If you format your log strings, 7.5 is definitely too high.

This is nothing surprising. I'm not talking about the bug, but the report.

If you shine a thousand spotlights at a problem, you'll find more problems.

Glad this library is finally getting the code review it deserves. Hope the whole SDK gets the fine-tooth comb treatment it is overdue.

Like it or not... 50% of the Internet runs on Java (and my statistics are 50% accurate. I swear, 50% of the time.)

Definitely there's no reason why Java libraries couldn't be as polished as in other popular languages. (Or why every Enterprise pattern should be present.) It's easy to just consume libraries and complain when they fail instead of contributing. Although I realize that a large part of the user base is corporate and just getting a permission to contribute during work hours can be problematic.

It's not just getting permission--and maybe I'm generalizing way too far here--but it seems much of "enterprise" java culture is oriented around commoditized, interchangeable cog people who meet KPIs just like it's composed of commoditized, interchangeable components which implement interfaces. So they get cog behaviors, as designed.

Polished, masterful, crafted product is hard to KPI for.

I think you’re generalizing way too much. Java is old. The libraries are old. There is a lot of bad old code out there, and very little glory in fixing it.

> Like it or not... 50% of the Internet runs on Java

Highly unlikely. Most of the internet runs on PHP, WordPress more often than not (there are stats on that, but I'm on mobile and can't check right now).

Get away from languages like C, they said. There are double free, dangling pointer, undefined behavior, they said.

Wow, a LOGGER engine that execute arbitrary code. wow

I think the natural point of comparison with C here would be printf().

Printf at least doesn't recursively expand the format string.

And one of the good things in Go is that you can pass arbotrary crap to fmt.Printf and it wont crash or overflow anything. So logging errors don't kill you (unlike c or python).

Ironically, tho, my tool to scan all our jar files for log4j has revealed a panic in archive/zip, something to do with a zero length file name.

Rust rewrite when?

The weakness is in the logger itself here, not Java per se.

I think it's more about the odds than that "there will never be an arbitrary execution bug in python/javascript/java/rust"

The comments show that nobody can prove a reliable attack vector of DOS in 2.16

The risk for a denial of service by a malicious programmer are much lower than the DOS risk posed by a distracted programmer. Looking at the typical attack surface in an enterprise app a distracted programer can stumble on to trigger a DOS the risk of stumbling on the log4j recursion by accident is minuscule.

The author of log4j abandoned it and wrote logback. That plus slf4j seem like a better path than sticking with log4j.

I'm not sure there was really a need for log4j2, and I think the name itself is what drove most people to it.

Avoid frameworks and libraries whenever possible.

The last time I benchmarked java.util.logging though, it lost out to log4j by a wide enough margin. Has anyone done any benchmarking lately?

Yes, let’s reinvent the wheel, waste time and resources every time!

Who needs decades of battle tested, proven methods and tools? Who needs thousands of human hours behind thousands of GitHub issues and pull requests?

Let’s do everything from scratch!

Hm, why is there JavaScript fatigue and 10 new frameworks every month?

Didn't log4j show us that "battle tested, proven" means nothing?

Nothing in programming is proven since everything changes all the time. Otherwise we would still be programming in Cobol, Fortran, LISP and APL without changes from the 1950's. It's like saying tanks from WW1 were good enough to last forever. Change means you have to prove things over and over, and generally the pace of change is too fast for anything to be proven before it is obsolete.

Finding and fixing this bug is part of the battle testing.

Truthfully most log libraries are not going to have such a flaw.

That may be true, but that doesn't mean that battle tested means nothing. There are plenty other logging libraries that have also been thoroughly tested. If you look for obscure logging libraries in c, c++ and even bash I wouldn't be surprised if you found rce bugs. In Java it's probably less common.

"Program testing can be used to show the presence of bugs, but never to show their absence!" - Djikstra... meaning, just because something is battle-tested, doesn't mean that we have proven it has no defects.

Not every time, "whenever possible".

I'm sure there's 0 chance your custom logger won't have any bugs either? At best you get some security through obscurity, but your chances go way up on writing a bug ridden custom library. No software selection paradigm will be 100% secure, that's the one guarantee you'll have.

I like java.util.logging.

Unless you're doing something very special, is there any reason to not use Javas build in logger? I don't know any Python developers that doesn't just use the logger in the standard library.

I'm not a Java developer, so I don't know, but is there something "wrong" with java.util.logging?

JUL is weird and came too late to stand a chance against commons logging and log4j. Log4j's API façade was successfully abstracted away into Slf4j, whose usage became ubiquitous in the Java ecosystem. You certainly can use JUL as an Slf4j backend though.

There's nothing wrong with java.util.logging. logback/log4j are better in almost every regard (configuration, usability, speed), but if you don't mind to write some boilerplate, jul is absolutely appropriate for most projects.

So, let me get this: Log4j is disabling JNDI, fixing various string substitution issues and who knows what else, but the root cause of the whole mess - that Log4j attempts string substitution on the actual parameter values remains untouched? Why?

Reading stories like this makes me very sad about the state of my profession. Something that is supposed to be a simple stupid logging library is on the front pages of the mainstream media due to all the havoc it's causing. We really have a long way to grow up as a profession.

Hardware is a mess, operating systems are a mess, programming languages are full of poor design decisions, foundational libraries are a mess and maintained by unpaid volunteers.

Our field is royally screwed, whole thing needs to be rethought.

And yet here we are taking over the world.

Indeed, how scary is that? ;)

Software architecture is a non-pure field where other fields, sciences and art routinely intersect to meet human factor and impact real world (ideally, improve it and solve problems).

Any such field is in varying degrees of messiness (medicine, agriculture, architecture, you name it). Mistakes are made constantly, have real consequences (people die), and hopefully are discovered and admitted sooner rather than later.

All we can is (cliché warning) do our best, keep growing, act responsibly and be humble.

I agree but the supposing part seems a bigger problem... log4j is not small or simple or understandable and this is easy to discover if you glance at it, I think the problem is more in underappreciating simplicity (aka overtolerating complexity).

This is the root issue. There are so many libraries that do way more than they should. I get it, too. It’s really hard to say no over and over and over. It gets tiring, but it’s the only way to rein in these kinds of things.

And if you advocate for avoiding complex deps, you'll get hordes of people defending them. Someone needs all these features, they are there for a reason! The authors of the library must be experts on the subject matter and anything you'd write yourself would inevitably suck. Why reinvent the wheel badly. Why make what you can take. Etcetra.

It's quite depressing.

The question is not about presence of complex tools. The question is about lack of very simple tools. I'm the person who likes simple tools. I need just a little bit more than simple System.out.println from logging. But I'm expected to use commonly accepted libraries and frameworks, nobody will be excited if I'd replace logback with my 200-lines "library". And every commonly used library in Java is complex and feature-ridden.

Sometimes I wish I had more time to rewrite all Java stack, from logging to http server with dumb simple inefficient but understandable code.

May be I should just move to Go. It seems to better reflect that approach. But I like Java language...

Perhaps I missed this, and I get there are backwards compatibility issues, but can't a version ship where default is that the logged strings (not formatting strings) are not parsed at all. This seems like a major design flaw - I don't want my logging library doing any parsing of the logged input.

Sometimes I wonder if it would be possible to estimate the probability of the presence of nasty vulnerabilities like this one on a software stack.

At one point, it seems that "everyone use this so it must be secure enough" replaced "we're a large company, did we spend enough time reviewing code of the open source stuff we use?".

It seems the Linus's quote "given enough eyeballs, all bugs are shallow", is not really true.

Imagine if a well funded agency like the NSA employed at least hundreds of full time developers whose job would be to sniff for those vulns. I'm pretty sure you could automate searching for those vulns, and that only the NSA has such tool.

That’s what Google’s project zero is.

The problem is, there are a vanishingly small amount of engineers talented enough to find these kinds of things.

And if NSA wanted to hire a few hundred they’d have to go the defense contractor route which will inevitably lead to them getting 1 great engineer and 499 extremely mediocre ones.

I would look at things getting parsed and throw some fuzzers at it. Maybe data mining Github to use some real world strings for it. I think with that approach even my normal sized brain would find problems.

It is about the willingness to invest, not brainpower. If I was tasked to "improve security" at my current project, I could instantly tell you things that could be looked into, even though reasonably precautions are already taken and there are already processes around it in place. I bet it is the same for most programmers.

>I would look at things getting parsed and throw some fuzzers at it. Maybe data mining Github to use some real world strings for it. I think with that approach even my normal sized brain would find problems.

Given that you're mentioning fuzzers and a decent source of inputs you could use, I think you're either underestimating yourself or overestimating the actual "average" of software developers.

Just hire good engineers, and train them inside the NSA.

> It seems the Linus's quote "given enough eyeballs, all bugs are shallow", is not really true.

Well that's the issue isn't it. Very few have read the log4j code, even though thousands of developers have incorporated it into they projects. Right now all eyes are on log4j and the bugs a showing up quicker than the log4j users can patch.

Then again, if Log4j wasn't open source, the NSA could still run said tool while most of these bugs would go completely unfound.

The current bug was either found or first exploited by Minecraft trolls, to bring down servers and clients via chat.

If you want to find issues in a code library, embed it somewhere in the hot code paths of a game or DRM system to maximize the number of eyeballs looking at every single instruction.

Honestly there's an excellent chance they already discovered it and are using it right now, or at least up til the past week or so.

> Linus's quote "given enough eyeballs, all bugs are shallow",

I think it is called Linus' Law but the quote is by ESR

I wonder if this won't be a boon for them in the long run. Lots of eyes on log4j security nowadays, it will come out stronger if with fewer users. I imagine a lot of user have switched to other logging solutions now or plan to in the near future.

Lets make a program that can print strings and do log rotation. Then overly complicate it, put everything in it. wcgw?

But I want my logger to brew me a coffee via the network card in my coffee machine. A logger that's not Turing-complete is definitely under-featured.

That's what we're missing. Anyone knows log4j enough to write a proof that it's Turing complete? How about it passing the Turing test?

I want my logger accessible/controlled from my nintendo switch so I can check logs while I game.

You just need fast print strings. Leave log rotation up to the OS and it’s tools.

How does it work? Do I need to close/reopen file after every log message?

No, just close and re-open the file when you receive a specific signal, generally SIGHUP, from logrotate or any equivalent tool.

I don't think that Java can handle UNIX signals, at least without some internal API.

I wish I could tell you how many times I've seen a machine run out of disk space because logrotate had been failing since the last time someone pushed out a config, or a new log file appeared with a software upgrade. It's gotten to the point where I give /var/log its own disk partition now, and hunt down any logs in /opt and bind mount them into /var/log. Screw the logs, I want my system to keep running.

I am 1000% in favor of apps managing their own logs. Generic system agents can collect and ship arbitrary logs remotely for storage/processing.

That's why you set your OS to rotate everything inside /var/log. And yes, chase down the other logs and place them there.

> It's gotten to the point where I give /var/log its own disk partition now,

Putting /var on its own partition has always been recommended.

When default installer/partitioners fail to do so, I assume it's a habit picked up from desktop-oriented Linux distros.

Logging to /opt would clearly be a bug of the brown M&M variety -- it would make me question other decisions made by the developers.

If you have a standard cloud image and want it to use all the disk given to it, one big xfs partition is easier than a bunch of various little partitions for the sake of not filling the root partition. Hopefully everything log rotates well and you have disk full metrics/alarms when they don't. And the architecture should be if the node goes bad for any reason just replace it (with hysteresis so you don't kill all the zookeepers at once when new traffic and bad rotation/rentention fills the disks all at once)

Your "easier" has a lot of complicating qualifications. :)

I'd describe single-partition as "slightly simpler, but requiring additional support mitigations, and less effective at the primary job of keeping systems running".

I prefer:

One partition for / which is required to have an operable system for diagnosis of problems

One partition for /var which is expected to grow with limited predictability

One partition for the rest, which should be largely static depending on function, but sometimes surprises you regardless.

...

With elastic scaling compute nodes and centralized logging, most of these issues are subsumed into the enormous support infrastructure and can be ignored. But some applications don't map well to that environment.

Well, TIL that log4j does log rotation by itself. Really, why the hell does an application need to rotate logs by itself? It can only lead to bad decisions. Now I understand why java servers never seem to have all the information about bugs on their logs.

And looking at the documentation, it has a very Java idea of rotation, where it supports every possible use case as a main case, and the examples expect that you will set it with the most insane defaults from day zero.

One reason to adopt microservice architecture. Stop import logging library, and build a logging service instead.

I’m going to stop you right there.

http://www.slf4j.org/

The gift that keeps giving!

Feels like a bad comedy movie.

At this point I feel like just shutting down anything Java-based until Log4j reaches version 2.20 or something.

2.20 would stand for 2^20 vulnerabilities.

I decided to just ditch it and write from scratch something api-compatible, but extremely cut-down on "features".

Maybe someone would release something along this line. I can't.

You could consider alternatives such as logback instead of writing your own.

A case of scare of old codebases. Decision was made to have at least some control.

this is basically what the openbsd community as done with several protocols/systems.

LibreSSL is a complete rewrite of the openssl functionality with drastically fewer features. Same goes for CARP.

Maybe running software with minimal defaults is a good thing, as it forces the users of the system/library/whatever to think about its behaviour and usecase.

LibreSSL is a fork, not a complete rewrite.

Use logback, spring boot default.

Logback isn’t entirely immune: https://cve.report/CVE-2021-42550, although access to the configuration file is required.

I wonder how difficult will it be to increment the major version and turn off all features that can lead to these vulnerabilities.

Time to fork and start removing features.

https://tinylog.org/v2/

I still try to understand why anybody would want a logger that executes embedded code and loads remote code from aribtrary web urls. That's like having a toaster that needs regular tire changes so it doesn't run me over.

Not again

Do they just, like, not have a QA department over there? Have people really still not understood that they amount of resources spent on QA must be exponentially proportional to dependents count? And that QA, not software "engineering", is the most important job that requires the most highly qualified people?

"they"? It's an open source software. What exactly do you expect?

I expect open source developers not to release projects or updates that they reasonably expect many others will choose to depend on unless they are willing and able to do extensive, thorough, sophisticated QA on them. Contributing to open source is a privilege, not a right

Three releases for essentially the same bug in a week is not okay. Just shut this project down. It’s seemingly a cesspit of bad code, bad testing and general incompetence.

Every project has bugs. Log4j just also happens to have literally the most scrutiny on the planet right now.

That's actually a great thing for the long term health of the project - it's getting a whole lot of free auditing right now. Many millions of dollars worth.

We have systems still using 1.2.x. Do you think if they closed the doors today, the problem would magically disappear? We'd have it ripped out of circulation, replaced by christmas, and everything would be magically easy?

There's an old joke, goes something like: “Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?”. That's what's happening for log4j right now. A lot of eyes, a lot of attention, stuff's going to come out - and long-term it'll be better off for it. Shutting it down and moving to something else that hasn't been tested in battle isn't the pay-off it sounds like.

That's fairly normal. I'd rather patch everything 5 times in a row to get rid of one RCE at a time, rather than running with a 10/10 vulnerability for 5 days so the application can make sure to root out all possible problems.

Reply on news.ycombinator.com