Hacker News Reader

Comment by marginalia_nu | Hacker News Reader

I don't get why people get so theological about logging.

If logs bring you some sort of value, then log away. If they don't, then don't. In some domains you absolutely crucially need logs, in others they're largely a waste of time.

Logging is probably the most foolproof way to extract debugging information from your application. The killer feature of logging is that standard out very rarely breaks, no matter how bad things get it works. It will work from the moment the application starts, it will work in an OOM situation, will work if your network switch is on fire, when the DNS records are wrong and your technicians can't even enter the building, it will work no matter how bad things are. If logging does somehow fail, then that is on you, in that scenario you have fucked up your architecture somehow.

View on HN

> I don't get why people get so theological about logging.

Because the reason to log is about people, not about technology.

There's no technical reason to make logs: Just don't make bugs, and you won't need the logs to find out what went wrong because nothing will have gone wrong. Go to the beach!

This is hard though, and bugs happen for all sorts of reason that have nothing to do with the programmer making a mistake-- the customer can be unsatisfied because they didn't know what they wanted. Programmers learn very early in their career to distrust the requirements, because they know that saying but that's what the ticket says gets them nowhere besides an argument: They have to do what the business/customer meant.

Put another way, the user doesn't want logs: The user wants to know if everything is okay or not, and presumably they have some way to figure that out, but if it isn't working correctly (or they merely think it isn't), the programmer will probably want those logs to help them.

In many environments, the loop is larger than just those two people, there's QA support people, and internal experts/operators, and triage, and all of those people are going to have different needs from those logfiles. People might also have a second use for those logs involving anticipation of failures and tracking capacity.

So there are a lot of non-technical, non-functional needs for logs, and once you understand this, it may be a little easier to understand why people would have almost rabid opinions about what those logs should look like (or if they can get the cure they need another way, what they might prefer instead of logs)

> [standard out] will work in an OOM situation

If you use write() which doesn't allocate memory, that's almost true, but if you use printf() it might allocate memory, and it might not work in an OOM anyway depending on what "standard out" actually is.

> will work if your network switch is on fire, when the DNS records are wrong and your technicians can't even enter the building

And yes you can write logs that nobody will ever be able to read, but what's the point of that? When your manufacturing plant is a smoldering rubble, someone is going to want to launch an expedition for those hard drives and try and figure out what caused millions (or potentially billions) of dollars in damages, so if you just wrote everything to standard output and washed your hands of the rest, you might have to find another job anyway.

I've been programming for over three decades at this point, and I'm convinced there is no "easy one-size-fits-all answer" to whether to log or not, or if to log, how to log, so I find it useful to learn about this bit of theology so that I'm better equipped to talk to others about this, and maybe convince them of my opinions.

> And yes you can write logs that nobody will ever be able to read, but what's the point of that? When your manufacturing plant is a smoldering rubble, someone is going to want to launch an expedition for those hard drives and try and figure out what caused millions (or potentially billions) of dollars in damages, so if you just wrote everything to standard output and washed your hands of the rest, you might have to find another job anyway.

Reducing the number of points of failure is almost always a good idea. The network being down is one problem, not being able to ever know what your applications did when the network was down is another problem added to the first problem.

> Reducing the number of points of failure is almost always a good idea.

Reducing the single points of failure is usually a good idea.

You can easily reduce the number of points of failure by reducing redundancy and removing sanity checks from your process. Doing that does normally not improve your reliability.

Sending the logs to your SAN for it to write in a couple of disks arrays is much more reliable than writing to stderr (stdout is not a reasonable place to log failures) and redirecting it to your local disk.

> Reducing the number of points of failure is almost always a good idea.

I think it is good to think about what happens when things fail: Things fail all the time and most of those failures are benign -- and even those that aren't can often be remedied by turning it off and on, but some of them aren't, so what happens then?

If you run out of disk space, writing a log line to the disk complaining that you're out of disk space is no help, but what else can you do? You could send a network alert, or print something to hardcopy (if there's a printer nearby), or ring a bell if you have one attached -- all of these things are also potential points of failure, but they can also help prevent a meltdown.

And if that sounds surprising, you might also be shocked the number of problems that can be solved with:

    <img src="cdn1/img.jpg" onerror="(function(t){t.onerror=function(){};t.src='cdn2/img.jpg'})(this)">

It's for this reason (I think) that people usually qualify your statement as reducing the number of "single" points of failure, instead of the way you put it, because having two single-points of failure isn't exactly like having two weak links in a chain, and in many cases, it's nothing like it at all.

In any event, there's another advantage to thinking about the failure-process instead of trying to merely avoid failures:

The biggest reason reducing points of failure is a bad idea is when it is more expensive to do so than to simply tolerate the failure: One thing that dominates the attention of startups looking to improve their availability beyond 99.999% uptime, is "removing the failure points" to get higher uptime, but if Amazon is down, what are Amazon's customers going to do? Are they going to buy from someplace else? Sometimes it is simply that more-established brands have the benefit of ignoring some problems, but you can be sure a large company like Amazon has someone who knows exactly how much more they can spend per second of uptime they may desire, and I think it is worth it for a small company to get good at that exercise as well, because sometimes the juice just isn't worth the squeeze.

As a specific example: My ad servers (HTTP event collectors) are critical, and I want 100% uptime for them because missing a click means missing money, but the reporting dashboard (so the publishers know how much money they made, and the advertiser knows how much they owe me) doesn't need that, so as long as the user got redirected to the landing page correctly, and I can generate line charts and excel spreadsheets once a day, nobody cares that the reporting UI might be down for a few minutes a day-- they will simply press refresh, not knowing if it is their internet or mine, and combined with aggressive javascript-side caching and automatic retries, I am not quite sure any of my customers even notice my downtime.

If instead, I focused on reducing the things that could go wrong (because as you say, it is almost always a good idea), I might spend time trying to improve the availability of my database simply because that's the thing that fails instead of whacking a setTimeout into the JavaScript since the impact is that the user would just retry on failure. That setTimeout saved me the cost of a whole server (and power and heat and electricity) and in the early days of my ad server, that would've been a lot of beer.

> "When your manufacturing plant is a smoldering rubble, someone is going to want to launch an expedition for those hard drives and try and figure out what caused millions (or potentially billions) of dollars in damages, so if you just wrote everything to standard output and washed your hands of the rest, you might have to find another job anyway."

That's the normal state of life; when a building burns down, forensic investigators pick over the evidence trying to work out what started a fire. They don't ask everyone to record every time two bits of metal touch, every switch switched, every use of electricity, every flammable material entering the building, just in case there's a fire one day. Use a teaspoon in a metal cup? Log it. Post-it note? Flammable, log it. Don't know if your shoes are flammable? Log them anyway just in case.

And yet the cause of the fire is not likely to be any of those things, it's likely to be the normalization-of-deviance fireworks next kept next to the open gas tank where the employees take smoke breaks which people were reporting for ages until they were told to shut up about it.

Beverley Hills Supper Club was known to be overcrowded, badly designed and ignoring fire department reports for years before it burned down. Cocoanut Grove nightclub had flammable decorations and an employee using flames for light. Rana Plaza had big cracks appear in the building before it collapsed, employees were told to go back to work or be sacked. The World Trade Center collapsed because a plane hit it. Titanic sank because it was going too fast and ignoring reports of icebergs and chose nice looking corridors over high bulkheads. Grenfell Tower burned while "everyone" knew the building was coated in flammable coating and nobody in power wanted to pay to do anything about it. Station Nightclub burned because someone accidentally brought outdoor fireworks indoors, and the sound damping foam was flammable not fireproof and it killed people because the emergency exit doors were locked.

We have a culture of log every order the captain makes, every response from the crew, every movement of the rudder, every radio signal sent and received, and at the same time "why did the front fall off?" "a wave hit it" "is that unusual?" "Yeah, at sea? Chance in a million."

> "you can write logs that nobody will ever be able to read, but what's the point of that?"

Cover your ass, cargo culting, habit, defaults are powerful, the cognitive bias effect like psychic powers where the one time people predict a bad thing happening and it happens is very memorable but the all the times people predict a bad happening and nothing happens are very forgettable, it's easy to remember one time logs helped and ignore all the times they didn't and the costs associated with them. For some possibilities.

> They don't ask everyone to record every time two bits of metal touch, every switch switched, every use of electricity, every flammable material entering the building, just in case there's a fire one day.

It's industry specific. In my line of work, yes, they do ask you to make extensive records like that and it is why we log nearly everything. Though I've mostly been in safety-critical and aerospace work (space now, previously just aero). I imagine medical, pharmaceutical manufacturing, and others have similar requirements. If you're making systems that don't matter (in the sense of, if they fail no one cares except a few possibly angry users), then don't bother logging anything.

My personal reason for having strong opinions on logging is that nearly every performance issue I have come across has had an easy win of 2x-3x improvement (small sample size, to be fair) by simply disabling logs. Not that this tends to stick because people fear the lack of information that comes from not logging, but you need to be aware of exactly how expensive logging to a synchronized output is, and how expensive that serialization is compared to the rest of the critical path.

People treat logging as if it's free, when the reality is that it very often has a significant impact on performance, exacerbated by the fact that it's treated as free.

Add in the fact that today's distributed systems approaches add complexity to the logging paradigm and make it much harder (vs something like Jaeger/OTel) to understand the context of a single log message, and I would rather logging disappear almost entirely in favor of metrics (fast, coarse) and tracing (slower, fully contextualized), with some ability to fall back to just printing metrics and traces to stdout or other local debugging options as a fallback.

Yes there is performance cost, but removing logs removes your ability to solve problems that happened in production. At best without logging you will have a snapshot of your program at the instant the problem was detected such as a process dump or even just a stack trace. Logs tell you what happened before and while things were breaking.

> but removing logs removes your ability to solve problems that happened in production

No it doesn't. Distributed tracing can do a far better job. As do metrics.

Metrics tell you bupkis about what causes an error, they only tell you that something has gone wrong.

And that's exactly where tracing comes in. Except instead of writing a dozen log queries in your ELK/datadog/splunk/etc system and switching contexts that many times, you have a single view that centralizes who called whom when, and even aggregates text metadata that you may have chosen to include. In other words, logs, but you don't need "started xyz" because that's just the natural beginning of each span.

I worked on a real-time telecom system, and had to debug issues that heavily depend on timing. Regular logging impacted performance and changed timing, but when we implemented logging properly we fixed this. Our solutions was in-memory ring buffer that stores only data and pointers to formatting functions, logs are formatted and printed on-demand.

A kindred soul. This is exactly my preferred approach to logging. A ring buffer gives you enough information to debug an issue when it occurs, and nothing otherwise.

Visibility into programs running in environments you can’t access is unreasonably important, but any code that creates visibility will impact your performance, too. The “right” solution is to balance those two needs for your situation, which means there is no “right” solution for everybody.

> logging to a synchronized output

There's your performance issue, logging doesn't need to be persisted or even processed by your main program thread or any thread/process actually doing the work. Who's logging straight to stdout or directly ::write'ing a file in a production application?

Uh oh it got silent in here :)

Many many people

Right, and depending on the actual buffer implementation when logs are buffered, given the massive volume of logging I've seen, it's likely not all that much better than synchronous because those buffers are going to fill quickly and need flushed frequently.

My comment wasn’t about buffered writing in a main thread, it was about moving the work out of your threads actually executing your program and having them push raw log actions into a queue for another thread to do the formatting and writes to file from.

If you’re logging so much that you’re saturating a dedicated thread that just reads from a queue in shared memory, does log formatting and writes to a file then you have bigger problems than figuring out how to log because there isn’t going to be a solution that lets you log with a deterministic latency without skipping events or wrapping around your queue.

Of course, the approach scales fine if you can have multiple log files per process - e.g. a normal log file and a message log or transaction log, because you can give each file its own queue and thread.

Me, not write(2)’ing though, mostly because you don’t want to syscall-block every few bytes. Modern service managers redirect 1,2 into logging facilities (or just files if configured such way), and 1,2 are no different than 3+. As if there was a regular way for data to escape a program through non-fd (mmap/shm/fork doesn’t count).

That said I’m not sure what context we are discussing. I work on nodejs apps now where blocking is a non-issue. I’d obviously think about it more in C’s fprintf() or a similar block-by-default environment.

Edit: also, my area is fintech with $0.1-$1000 range per request, not telecom. I log everything because my employer can ask about everything, and it’s always money.

I get the idea, but then how would you decide when to enable tracing and in which area?

IMO, the advantage of logging is that it some degree of information about actual failure events - in contrast, monitoring only gives you aggregated information and tracing is too expensive to be applied everywhere at any time.

Tracing should be sampled at a rate that makes sense for the dollar impact of an error in the service, versus the dollar cost of your tracing system.

$thousands per transaction? Sample 100%. Cheaper to refund a customer than debug the problem every time? Then sample. You'll have plenty of sampled traces with the bug/error shown as soon as the problem is expensive enough to matter.

Metrics are great for demonstrating a problem exists (and alerting), and tracing is great for helping you understand how and why it happened once it's at a big enough scale to matter.

Beyond that, with a lot of what's coming out of Open Telemetry, you can get fancier like sampling a higher percent of traces to a localized processor middleware that then filters out any traces you doesn't care about (e.g. dropping non-errors or low-quantile latencies).

Yeah, but once again, this requires you to already know there is an error, have a dollar impact calculated, etc.

There is a lot of talk in this thread about the performance impact of logging. I don't see how sampling 100% wouldn't have a far greater performance impact.

Definite up vote here. Theological is the perfect word for this. It would be helpful if the author substituted the word "blog" for the word "log" in the article, and took the new meanings that result seriously.

> I don't get why people get so theological about logging.

> If logs bring you some sort of value, then log away.

I have been on the other side of the problem for years now. Logging has a cost. And the cost benefit is also not great. It requires a human to read it(either that or a regepx hellscape awaits you). This can be fine if you are working with a small service, but it quickly gets out of hand once scale increases.

I've had to deal with double digits(approaching triple) terabytes of logs daily. Most of the information logged was, like the author suggests, completely irrelevant. It makes it much harder to see what's actually relevant. And, for whatever was actually relevant, generally not enough context was provided (something the author also argues).

What you really have to have in production systems are metrics. That's what's going to tell you if shit is on fire and which ass is burning.

Has your app has hit a catastrophic failure and cannot continue? OOM? Sure, by all means log away. Did it have an error while trying to talk to a service? Are you retrying? Then _do not log_ until your retries (or whatever recovery mechanism) fail. Networks have become very reliable, but failures are expected and normal. Spit out metrics instead.

If you don't do this, some schmuck will have to spend their days trying to keep the logging infrastructure from falling over (Cthulhu helps you if you are on an ELK stack) while management will be complaining why noone seems to notice failures before customers do.

If you find yourself parsing logs to alert on issues, please revisit your architecture.

METRICS FIRST. Logs are a safety net and a tool for troubleshooters.

What you are describing is pretty clearly within a domain where logging doesn't bring much value. It's either superfluous debug messages from an earlier phase in development, or at the wrong granularity. If you are dealing with thousands of events per second, you can't log at every event. Your logging probably shouldn't produce more than a couple of lines/second on average unless something has gone very wrong.

Logging is something that needs to be maintained (like all code), it can't be just some vestigial by-product of the development that's left to its own devices, at least not if it's to be of any use. The same is true for metrics. If your metrics are just some arbitrary selection of times and counts that seemed good at the time, they're probably not very useful. Which data to collect is something that needs to be deliberately chosen, and re-evaluated over the lifetime of the application. When you are launching a new product, then lots of logging is probably a good idea. When that product is mature, you probably should have dialed it back several notches, as you most likely already know the general areas where things go wrong.

> If logs bring you some sort of value, then log away. If they don't, then don't.

The issue is -- some people find value in the logs but some other people don't.

I am in the strong camp that find logging provide negative value, mostly due to the extra complexity it brings and its obstruction on code reading. And I believe the reason the other camp finds value in logging is because they lack insight to pin-point issues and lack effort to gain that insight. But I can't really use that argument because it can easily be deemed offensive and solicit slash backs rather than constructive discussions and debates.

Logging is my most frustrating pet peeves.

> I don't get why people get so theological about logging.

IMO, it’s because the right logging solution for one situation is not the right solution for another situation. That means that one person is “right” and the other person is “wrong” (because the context required to identify why one person is right is complicated and hard to express, and is thus missing).

Observability (of which logging is a subset) has a performance impact, and software has performance requirements. This means that the right amount of observability is going to depend on the software, not a standard.

Most applications today run as replica sets (k8s pods/VM asgs) so log aggregators are another interesting part of the problem. Most shops are moving to saas logging for log collection which presents another failure mode: your logs are only as reliable as the saas provider.

It is true however that looking at the individual replica, application logs should be very reliable.

> log aggregators are another interesting part of the problem.

And then you hope you are not logging to a logging service that'll try to execute something that was logged.

Reply on news.ycombinator.com