Read Every Single Error | Hacker News Reader

Read Every Single Error

pulumi.com

84 pointsmooreds49 comments

One thing that I've observed is that the more errors there are, the less likely people will look at them – much less fix the underlying source – so the problem only grows. If you have 1 error/month it's "zomg! An error! PANIC!" but if you have 10,000 every day it seems an insurmountable task to fix it all, "so whatever". I don't think it's apathy as the post suggests, it's just that it can seem like an insurmountable task and whatever you do will just be a drop in the ocean.

An error is either:

1. A real bug someone ran in to.

2. Not a bug and just a spurious "error", which should be a log message, some statistic somewhere, or just disabled outright. Errors should only be errors – an uncaught exception, explicit error log, stuff like that. Everything else should be something else.

What I've found – and this certainly won't hold true for all cases, but it does seem to be the case often – is that half or more all in the second category: it's just stuff that shouldn't be logged as an error. This is usually trivial to fix.

This still leaves you with 5,000 errors/day, but typically ~10 bugs will cause ~4,950 of them. This takes more time to fix, depending on the bug, but usually also takes less time than "5,000 errors every day!" makes it sound.

The last ~50 errors are usually the hardest, especially if it's frontend JavaScript stuff. I "read every error", but I have a few that pop up regularly that I just can't reproduce and don't know how to fix. I think it may be extensions people use or something, but idk. Since this post talks about APIs I guess that's not really applicable to them.

> One thing that I've observed is that the more errors there are, the less likely people will look at them

It can be even worse than that. At one point I worked on an app that generated north of 500 lines in the PHP error logs per minute per server. In the course of implementing a feature I cleaned up a few of the low hanging warnings near the code I was modifying. This resulted in about a 3 percent reduction in the log file velocity.

When we deployed to prod, everyone panicked, because log file velocity staying about the same was one of the primary things they alerted on, as a proxy signal for server health. That was the one and only time in my career I was reprimanded for cleaning up warnings.

> log file velocity staying about the same was one of the primary things they alerted on, as a proxy signal for server health

I don't want to bash on PHP like it's 2002, but ... that's probably one of the most PHP things I ever heard.

I worked somewhere where people were against raising error_reporting because someone was in the habit of writing "flase" instead of "false" – which worked because PHP – and it wasn't deemed worth fixing :-/

Old PHP days were wild.

'log file velocity' ... that's one way to go about it!

A good logging tool will group similar errors together so one error occurring 10,000 times shows once in the list.

But what I’ve found is a whole lot of errors are things timing out which are kind of an issue, but not something you can really investigate or solve.

Grouping helps, but often errors are distinct enough that they will get duplicated regardless, especially after updates which changes the stack trace. To be honest it's been a few years since I used these tools, but last time it was helpful but very much less-than-perfect.

> a whole lot of errors are things timing out which are kind of an issue, but not something you can really investigate or solve.

Depends what is timing out; if it's SQL queries then you should probably do something somewhere, if it's HTTP connections then you probably shouldn't log them as an error but as info/debug, keep some aggregate statistic somewhere, or just not do anything with it at all (depending on your scale and what you need).

Or switch to gRPC :) if it is internal to the data center traffic.

You can still get network errors with that, no? I'm pretty sure I've seen some of those anyway.

The go API has two modes: expose network errors to the caller or “trust Google.”

Most logging tools if used correctly separates static messages from dynamic values. Which makes grouping trivial.

> One thing that I've observed is that the more errors there are, the less likely people will look at them

And then you have nodejs warnings

> One thing that I've observed is that the more errors there are, the less likely people will look at them – much less fix the underlying source – so the problem only grows. If you have 1 error/month it's "zomg! An error! PANIC!" but if you have 10,000 every day it seems an insurmountable task to fix it all, "so whatever". I don't think it's apathy as the post suggests, it's just that it can seem like an insurmountable task and whatever you do will just be a drop in the ocean.

If I may broaden this, I think the same principle applies for:

- poor performance

- tech debt/otherwise hard to manage extant functionality

- gaps in documentation

And probably a lot more. If it’s one or a few narrowly identifiable problems, it can be easy to escalate and rally and solve. If it feels pervasive, even trying to address one thing feels insurmountable. And speaking to this from a perspective of being highly motivated to tackle problems like those I mention, and having some fairly significant successes in that regard over recent months, it still feels daunting. But it feels a lot more reasonably possible and more importantly demonstrably valuable the more I chip away at it. But it has taken actual years to get even that far.

I guess my age is showing but why would you not read every error? Under what circumstances is it acceptable to ever use a metric like raw number of e.g 500 errors, if you don't know the breakdown of them, whether you're dealing with 50 different errors vs the same error happening 50 times?

At Google, we couldn't have read every single error that eg our Google Photos servers produce, even if we wanted to.

At that scale, you get some errors even purely from bits flipping randomly in memory due to cosmic rays. (The latter is speculation. I don't remember how many of those we got, or whether we used enough error correcting memory. It's been a while.)

Surely you would have wanted to know how many bit flip errors you were getting - surely you attempted to isolate them, count them?

Sure, you do statistics on them and you look at random samples in detail. But that's different from 'Reading Every Single Error' like the submitted article suggests.

Counting and grouping them is the opposite of what the article is arguing for. The whole point of the article is that you shouldn't be doing any aggregation and should be investigating each one individually.

But TA also argues that people reach for these aggregation tools way too early. I'd say the author is honest enough to refrain from statement about Google scale because they haven't experienced it. There is value in keeping error rates down even at that level.

Sure, but the comment I was responding to was missing the point, because the article wasn't trying to convince people that ignore errors to record and aggregate them, they are arguing that people who record and aggregate errors should instead look at each one individually.

Pointing out that errors should be counted is tangential to the argument at hand.

Ditto, my managed service at Google threw close to 3k 500s / second when I was still there. Anything from cosmic rays, faulty hardware bit flipping, hard drive failures.

We did, however, aggregate and group similar 500s and those did get looked at, but no way could we have looked at all errors.

The other thing, is that with resilient infrastructure, who cares about an occasional 500. Back off and retry. No harm done.

3k? What percentage of requests is that?

User experience might indeed not be influenced by these errors, but errors of a less stochastic nature will impact it. The former obscure visibility of the latter, and that's probably the point of TA.

It's likely some fraction of those errors would have had some tangible impact on user experience.

Curious whether there were any attempts to “read” them with AI. Is this something a LLM could do these days?

The article admits that it might not be feasible at Google scale, and that it makes sense to aggregate errors at that point. But surely that works better the fewer uninvestigated classes of errors there are, no?

It kind of depends on the types of errors. Do you need to do anything about every connection reset by peer error for web clients? Could you even if you tried?

OTOH, the correct number of connection reset by peer errors between your web servers and your database servers is likely zero, and there may be something to do.

There are different types of errors as you rightfully point out. Peer resets for web clients are (typically) caused by environmental factors outside the hosts control. It would make sense to group then and monitor then in the aggregate - if there's a surge of 200% then some investigation might well be warranted.

I will however guarantee that the author does not mean that they actually look at every peer reset individually.

Broken database connections - yes, a code red and they must be examined urgently.

The whole point is to classify your errors, and especially to know when a new class of error had been seen.

I work for a CDN, we are handling billions of requests a day. With 10s of thousands of servers around the world, we are always going to be having thousands of errors for all sorts of reasons outside our control. No way we can look at all those individually.

These are probably different kind of "errors" than what the author of this article is talking about. The kind of errors you'll see are dropped network connections, malformed data, and stuff like that which is basically impossible to prevent because it's an imperfect universe.

Even at a fairly small scale you will see "errors" of this kind regardless – you don't need to be a huge CDN for that.

But what the author is talking about – in my reading anyway – is programming errors: exceptions, out of bounds access, explicit report_error() calls, things like that which will result in 5xx errors for the client. You've got a lot more control over that. Sometimes random failures beyond your control will cause a 5xx error, and IMHO that's a bug and this usually should be a 4xx or something else (e.g. if the client sends wrong or junk data you shouldn't respond with a 5xx).

I suppose that's the corollary to this article: not all "errors" are the same, and you should distinguish between different types of errors.

Related advice for when there’s too many errors to read: remember that the original use of triage (in hospitals) was to quickly categorize cases so you could prioritize ones to work on first. It still rapidly terminates, and almost immediately results in individual doctors or teams of doctors working on individual cases. Your triage system will aggregate errors to determine what’s important, sure, but it should quickly tell you what to work on, and (most crucially) those tasks it returns should be disaggregated individual errors to work on.

The advantage of software engineering over medical treatment is that for the SRE, solving one case also solves all cases of that type, whereas the doctor has to administer the same treatment to each individual case. But don’t let that scaling trick you into tolerating a triage system that results in tasks like “re-architect this service”.

This is something that probably sounds painfully obvious to someone inexperienced, although they can’t appreciate why it isn’t so obvious. Perhaps this is simply an example of “do things that dont scale” while you can,

This is a great example of the wisdom of not overthinking it. I independently rediscovered this trick early in my career using grep, sort and uniq to get the best up front return. And then later a more sophisticated version with a system that logged to SQLServer. Implementing it as a dev policy is a novelty for me though. I heartily approve.

> Error budgets and the SRE model are haute couture.

The error budget does not mean that you don't read the errors. It is just a way to adjust the amount of time you spend fixing bugs versus pushing new features.

Also, as mentioned...

> Over the next six months, the team iterates, listens to customers, and delivers a ton of value. As a result, traffic levels grow to a rate of 10,000,000 requests per year.

The people who advocate looking at errors in aggregate may have much, much busier services.

I was about to write that some kinds of advice are very good in some scenarios and very bad in others. The arguments always seem to skirt that middle ground where good advice goes bad (or vice versa)

10MM requests per year is 20 requests/minute. Most people will probably be working on services at this scale, or less. So the advice is good, but not global, and the qualification is important.

I was confused when I saw them mocking error budgets at the end. It shows that they don't understand what error budgets are. What they describe themselves doing is basically an error budget by a different name.

We have an error budget at work and we still read essentially every error. There is a long tail of errors that might show up once or twice a week or month and I'll admit that I might have missed those, but within any practical sense, my team is still aware of any error that is contributing against our error budget. We resolve errors too, even while under budgetm because if we didn't then we would go over our error budget since new errors pop up all the time, so you need to resolve existing errors because new ones are introduced constantly.

Pulumi seems to think that companies with Error Budgets just wait until they are out of compliance to fix errors and then must only resolve enough to get back under budget.

I often find myself working on systems where there's a high volume of traffic/processing to developer ratio. Sometimes I'm the only developer. As such, when looking at errors coming in, it's very tempting to take a step back and come up with a methodical triage and error processing framework. However, I find that's a quick path to analysis paralysis. Instead, I'll just start with the most recent error, work through it and push a fix if applicable (whether that's five minutes or five hours), and then move on to the next one. This lets me very quickly add quality back into the system, and very often I find myself resolving other seemingly unrelated issues in the process.

If you're small then I agree.

If you've got 100,000 concurrent users then this isn't going to work.

1,000,000 actions per minute means that all of the actions need a 99.9999% success rate to generate 1 error per minute.

Lots of 9's of reliability is hard.

Also. At that kind of scale you start observing lots of non-deterministic errors such as rare race conditions. As you drill into these rare race conditions you realise that they're affecting 0.001% of customers and you'd have to re-architect your system to eliminate them. You hit a point where it's not worth fixing those errors, so you build an error aggregating system to help ignore them.

Of course any reasonable engineer is going to aggregate similar errors somehow?

You don't need to read every single connection reset by peer or similar network errors..

I agree with you, but the article specifically mocks error aggregators and says that there is more value in relishing each error.

This is counter-intuitive, but seems to be what the article is suggesting.

The authors essentially concur with this view. But comparatively few services have both such high traffic and reliability levels. For everything below, people reach for aggregation way too early.

A customer needed help with the size of their DataDog bill. A big part of that was log messages. They had poor logging discipline, so they started by filtering out all logs that were INFO or lower priority. But then the developers couldn't see what was happening in production, and they started logging messages at WARNING and ERROR in production. That made the output of unit tests in CI light up like a Christmas tree, and the output was unusable. Good times.

The solution was to carefully audit all log messages, reducing the number of messages, making sure they were at the right level, and replacing log messages with trace attributes. We also implemented trace sampling to avoid paying for redundant success messages. That reduced the bill by 80%, saving tens of thousands of dollars a month.

With compilers we do have -Werror. This is a godsend, because it forbids crawling up junk like this.

Runtime errors are even worse, for this we do have a CI with a 90% test coverage, or better a model checker like cbmc with 100% coverage. Every single error counts. Esp. on hard Realtime, where we reboot on a single error

IMO - log every 5xx (4xxs too), ensure your logs/error messages are groupable by message (e.g. keep variables like ids in a metadata object separate from the message), and keep a close eye on errors grouped by message and HTTP status code. Then fix any 5xx error that’s too common - IMO, for most cases, this means more than a handful of the same 5xx error most days. And of course, don’t just rely on ppl happening to look at dashboards, alert/page on significant 5xx volume.

But there’s no need to send every single error to Slack individually, just keep an eye on counts by error message.

Counterpoint: my team at Google did desire to read every single error, but still had SLOs and error budgets. In fact, there was a tool specifically for finding and aggregating unique errors so that they could be made into issues. Granted, in practice, that doesn't mean we fixed every single error, but I think that there's no reason to assume "read every single error" conflicts in any way with the typical SRE practices. I'd argue you actually would be doing yourself a disservice to throw away all of the SRE wisdom over this; measuring things quantitatively is definitely not a waste of time.

Reading every error is a great idea, but you don't need to read them all yourself - let the computer read them for you.

This is what error aggregators (e.g. Rollbar) do. Now you've got 10-1000x leverage on your time because you're reading unique errors instead of every occurrence, and you've got more data to debug with because you've got data about the aggregates ("when did it start? what request params are common? what servers is it on?") in addition to data about the individual occurrences. You can still commit to solving them all ("Rollbar Zero"). It's just a whole lot easier.

This sounds good, but I don’t understand how it meshes with building reliable systems on top of fallible components. I recently deployed a new service that has three clusters inside it; customer queries hit a proxy layer that queries all three and returns the first non-error reply to the customer. In a typical day, I will have tens of thousands of errors but customers see none. My first deployed code logged the internal errors as errors, but then the SRE started in on a high number of failures, so I switched to counting the internal errors but not reporting as errors, saving that for customer errors.

Cart before the horses? Why would one question error reading merits, if having a very good understanding of how they're produced, i.e. participate in their configuration and enablement, to begin with?

This article comes off as "holy than thou". While maintaining what seems like ignorance in the "haute culture" that they mock.

Their primary point of "read every error" is one that almost no one would truly debate (within reason). Most companies do try to read every error as is reasonable. Being aware of an error is different than immediately fixing it (since that can usually be impractical). Where companies tend to differ is in their ability to triage and correct errors. That is the secret sauce that people want to hear.

One of their secondaries points is mocking companies that use error system aggregators. Why is this so bad? They claim it prevents you from reading every error, but I'd beg to differ.

Let's say your system gets 5,000 errors today. It is highly likely that many of those errors are replicate instances of the same error. Out of 5,000 errors you probably have 5 errors that are 800-1,000 instances each, which consumes 4,800 or so of the error instances. Then you probably have a long tail of errors that only show up once or twice a day. So why is an error aggregator bad? Is there any point in reading the same error 1,000 times? Wouldn't it make more sense to read the error once, have the system show you that it shows up 1,000 which is the highest in your system. Then you can spend less time reading more duplicate errors and spend that time fixing them. You can look at a graph and see that the error increases over time, which tells you to prioritize resolution of this error.

If you have 2 hours a day to fix bugs, would you rather fix the bug that causes 2 errors per day or the one that fixes 1,000?

They seem proud of their error burndown chart that shows errors decreasing over time, but this is likely not the result of "reading every error" as much as actually putting time and attention into error resolution. My company for example put a team together at the end of last year who is a cross-functional team who's sole job is fixing prominent errors. We use an error aggregator and our error burndown chart looks the same as Pulumi's. Why? Because we are dedicating resources and attention to fixing it.

Where most companies go wrong is not that they don't read errors. It is that they are so busy pumping out features that they don't dedicate time and resources to fixing things or they don't provide enough time to build less error-prone code before it hits production in the first place.

Lastly, the "haute culture" of error budgets are funny to me. Yes I get that this is the SRE mantra. But what Pulumi describes here is basically an error budget by a different name. You have just set a lower threshold (ie 'budget') for acceptable errors. Just because a team has an error budget doesn't mean that they don't read the errors until they go over. They are reading and fixing errors to stay within the budget.

ok, but don't start looking at runtime errors till you've fixed/eliminated every warning during your build

Reply on news.ycombinator.com