Hacker News Reader

The most dangerous kinds of mistakes in that area are bugs or oversights that cause massive resource usage or endless loops. Those can easily bring down parts of the application that are otherwise robustly separated in the way you mentioned. Consuming too much memory will kill the app at some point, just using lots of CPU or IO will make everything else grind to a halt. Holding onto resources that are pooled like DB connections is a good way to break everything as well.

There are some languages and frameworks that are designed around even handling and isolating issues like these to some extent, Erlang/Elixir for example.

But in general I would expect a web application backend to isolate services the way you mentioned. But I don't think this is really a preconfigured default in every case, and it's not safe do catch everything and keep running. Isolating web requests and only failing the individual request is of course reasonable and should be the default. But for background services your framework can't know which ones are required and which ones are optional and when to crash the process. Especially as most languages allow shared state there, so the framework can't know how independent they are.

Indeed, I remember a significant degradation of service that happened when some data files were corrupted.

Each request would need to work with one or more of them. There was a cache, and the code was written to avoid a thundering herd. But in the case where the data was corrupt, an exception was thrown, and nothing was put in the cache. So the application sat dedicating multiple cores to a cycle of loading and parsing these data files then errorring out.

Have you worked with Go codebases before?

Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics.

Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entire process.

We were aware of this but ended up including one by accident anyway. It’s very sad to me that you can’t apply a global process handler to ensure this type of thing doesn’t happen, but to my knowledge that isn’t possible.

Worth mentioning Go doesn’t really encourage ‘frameworks’, and most Go apps compose together various libraries to make their app, rather than using something that packages everything together. Failures like this are an obvious downside to not having the well reviewed nuts-and-bolts-included framework to handle this for you.

I probably misunderstand what you wrote. Because I think a (wrapped) panic will only result in crashing the one request that caused it?

For example Gin provides a middleware wrapper for handling panics: CustomRecoveryWithWriter returns a middleware for a given writer that recovers from any panics and calls the provided handle func to handle it.

(https://github.com/gin-gonic/gin/blob/master/recovery.go)

The post-mortem we published about this outage explains the details of how this crashed the binary.

It gives some code examples and explanations that should clear this up: https://incident.io/blog/intermittent-downtime#mitigation-1-...

^ link should go to the relevant section

Will have a look, thanks!

To me, it's unclear what the best solution is here. Other languages solve this differently with tradeoffs, e.g. Java defaults to threads silently dying when an exception isn't caught. Your program will continue to run, but it's probably in some undefined state at that point. There are mechanisms for propagating exceptions elsewhere, but they have to be explicitly set up (like in Go). You can set a default uncaught exception handler, but that's effectively a global variable with all the subsequent "fun", and the uncaught exception handler has to know how to clean up and restore state if the exception was thrown from anywhere, which seems generally difficult to do correctly.

Erlang/Elixir have a great story here: “let it crash”. Because each slice of activity in an application is wrapped in its own process (think single threaded loop but you can run a million at a time, almost free to create and destroy), if it crashes it only takes down that web request/process. Recovery mechanisms are built in to get back to a know good state.

Pm2 for nodejs is the same.

I haven't used Erlang extensively, what happens if you crash in the middle of e.g. holding a lock, or during a coordinated dance with other processes?

My concern isn't really "does the program keep running?", it's "does the program keep running correctly?".

The key to Erlang error handling is that crashes should bubble up to a high level which then restarts everything below it in a known good state.

If you're in a coordinated dance with another process you link to that process. If a process you're linked to crashes then you crash too. There's no way to block yourself in Erlang such that you can't be told to crash.

After you crash your supervisor might restart you, if that's what you configured. Or you might give up on your specific task.

That sort of problem is beyond the scope of the runtime in any case, isn't it? In either of the examples you offered (holding a lock, coordinating with other processes), there must be timeouts enforced by the lock or the other processes so that, if something goes wrong, the system isn't waiting for a crashed process to continue the work.

Erlang/Elixir do make this pretty easy to manage, including the scenario where the process does recover by reverting back to a known good state. It won't do it for you automatically, but it exposes enough surface area to make problems like that solvable without reaching for a lot of extra tools - it's built into the runtime.

> That sort of problem is beyond the scope of the runtime in any case, isn't it?

Yes, which is why Go's outright crashing also makes sense to me...both Go and Erlang's behavior seem conceptually the same, with some architectural tradeoffs. It's not really that different for a process to die and restart. If some shared resource reaches an undefined state, then you have to kill everything and reset your state anyway. I suppose Go's behavior lends itself better to "microservices", whereas Erlang's behavior is better suited for "monolith" processes that do a lot of different things.

IMO either of these are better than Java's default behavior of silently swallowing the exception and allowing the thread to quietly die.

> when an exception isn't caught

Not catching all exceptions is a glaring P0 bug.

You should almost never catch all exceptions, i.e. Throwable on the JVM. That is one of the few things that Scala really got right. The `catch NonFatal(e) =>` idiom is doing that nicely. It will catch all throwables with a selected set of special cases, e.g. OutOfMemoryException and all the other VirtulMachineErrors. Catching those in a framework lead to extending the time until a crash follows on a serious issue. Crashing early is often beneficial in such a situation. Together with a process watchdog, like systemd, kubernetes, dockerd, whatever crashing early increases the uptime.

Node.js changed behaviors over time.

> Have you worked with Go codebases before?

Several but ...

> Standard practice is to wrap all your entrypoints [...] in a defer recover()

... I've never seen that. Is there some literature pointing to this as best practice?

You’ll find some libraries do this for you, such as HTTP servers.

They do this because if your server code makes a mistake such as accessing a nil pointer, a segfault or panic would bring down the entire process. That’s why you want to recover(), to avoid a your process dying.

> They do this because if your server code makes a mistake

This is neither good or best practice.

My take:

- Know that there are simple-mistake panics, and internal-state-just-went-bonkers panics. For the latter, you can guess the boundary of impact (one request, one connection, one job, one userID, one process, ...) but exit(1) is much more reliable than guessing.

- Tests can easily catch simple mistakes like accessing a nil pointer.

- Know that the kind of programmer who does not care about simple tests, will also not care about concurrency bugs which introduce the more dangerous types of state corruption. This corruption would likely be not limited to a single request.

Good luck to frameworks who assume that nothing bad would happen if they ignore a panic and continue serving more requests.

> - Tests can easily catch simple mistakes like accessing a nil pointer.

no they can't . this is exactly what is hard to test for, complex state that can occur by some combination of many variables on many values

Respectfully, this is a really bad take. And one that flies in the face of the Go stdlib, given the default HTTP server will catch panics for you by default.

I mean, net/http does it. That's the standard library.

A go convention that I've made up, or that is perhaps a real one, is to always look at the standard library for guidance on how to write Go. net/http suffers a little bit from being a very early library and thus you might not want to emulate its API surface. But in general, the Go team thought "you know what every HTTP handler in Go needs? recovery from panics" and that is worth some weight when considering your own design.

I would personally recommend fuzz testing your code, including HTTP handlers. The more panics you find in development, the fewer customers you lose from panics in production. Remember that recovering from a panic in an HTTP handler still means your user's request didn't get processed. They are not happy about that, even if your program can still service other users.

We had a single supervisor for both a Elixir/Phoenix web app and all the tasks it spawned to do long running jobs and scheduled tasks. That was very naive. A job too quickly too many times would bring down the supervisor and the web app. We moved the web app to its own supervisor tree and isolated tasks to their own supervisor too. I don't remember the details but at the end everything could crash and everything else would keep running.

That's why you use a separate `Task.Supervisor`.

> I've never seen a production ready web framework or library that would let your entire process crash because of a bad request.

It's rare, but I've seen linked C libraries trigger a segfault that takes the whole thing down; no global try/catch strategy can help you when that happens. There should generally be something that supervises and restarts the whole server process, but it's definitely painful and can affect threads other than the one handling the bad request.

Depends on the underlying framework on how it isolates processes... I could see this happening in a monolithic JVM application where it pretends there are separate containers, but a fatal error in the JVM will crash the world on that server.

A related example I lived through with a Perl application, someone decided to use this library 'Storable' that would serialize the memory in a binary format. We upgraded the library and started seeing "slow" performance across the server farm.

We recognized the processes were intermittently crashing and after decoding core dumps... figured out it was this upgrade to the Storable library. Apache httpd server chugged along just fine restarting processes. So different run-time, different type of crash resiliency.

Long-term lesson... be extra cautious with memory-serialized objects. Newer libraries have better protection on this to parse a header to detect compatible issues before loading the raw object into memory, but the potential is there especially with distributed systems today.

> There's always some error handling logic wrapping around the request handling

Yeah, some languages make this easier than others.

The way you described it made me think it was Rust, a language where those handlers are not trivial, but not incredibly hard either. But it seems that Go developers have it worse (no big surprise here).

Of course, the champions on unhandled failures are always C and C++, where almost any issue is impossible to recover from. It's no coincidence that those are low level languages (and Go).

When you move from the more controlled languages into those, you tend to also move from in-language error recovery to system-wide error recovery.

I actually found that in practice my go applications were much more resilient. First one in production was handling well over 100k req/s and it was extremely reliable. As a beginner it was very satisfying.

My team has experienced _exactly_ the same cyclical panicking in production before because of the same library.

The Google Pub/Sub Go library does not handle panics by default, so if any message payload plays badly with your code and panics, you cyclically panic the service.

That's because the message keeps getting retried because you don't `Ack` it. Non acknowledgements get retried automatically, you get the picture.

> I've never seen a production ready web framework or library that would let your entire process crash because of a bad request.

Node does this (regardless of framework/server) if you have some unhandled promise rejection/error in an asynchronous function.

You can catch these with Node but I wouldn’t do it - if it’s not explicitly handled then it’s either a bug or something really bad happened and the app needs to get alarm bells ringing.

Error recovery in long-running dynamic-y processes is nontrivial, especially for “programming errors”. While it can be done, much easier to stick to crashonly practices. Also helps in production where you can safely crash it and restart without worrying about stale something.

which can be handled easily with unhandledrejection.

Though ideally, web frameworks such as NestJs is already handled the rejection on request level and can be override easily with Filters. Yeah I know express isn't handled, you'll need to handle it yourself.

Erlang and any Actors models I think

Yes and no. Erlang does apply the "fail fast" idea and recommend to let actor crash instead of trying to handle the error (with some pragmatism ofc). But we are not talking about crashing the process of the whole app. In most case in Erlang (but you find the same idea with over massively parallel system or with micro-services), your actor will be handled by a supervisor, which job it is to keep it alive, and so restart it when it crash. Also, most of the time in Erlang, having a actor crash and spawning a new one is really fast, so you can afford to crash even for something as basic as a malformed json request or similar. Not something you can really afford when spawning a new system is more in the order of several hundred ms.