Hacker News Reader

Comment by sb8244 | Hacker News Reader

The argument here is that the system can get itself into a state that is irreparable (and perhaps very difficult to track / debug).

By crashing and having it start into a good state, the system will at least be operational again.

View on HN

With all due respect, that does nothing to prevent bugs. In fact, it almost seems to encourage them: "Your process crashed? Don't worry about finding and fixing the bug. We'll just restart it again."

Depends how the scale you're working on, I suppose.

If you running a thousands of programs on hundreds of thousands of machines 24-7, you're bound to run into weird edge cases at the system level.

Instead of worrying about and optimizing all of these edge cases, SOMETIMES it is better to just have a system that is tolerant of these edge cases by design.

Perhaps, but that's an entirely different claim from the one made in the article (that Erlang can prevent such bugs).

The claim was it prevents Heisenbugs.

It does not prevent them. The reasoning is, that those bugs are so obscure, only occuring in vanishingly rare constellations, that finding and preventing them would be prohibitively hard/expensive.

Say you have an application that takes user input, validates it and then try to add it to the current state. If there is a bug in the validation so that 0 is allowed as input, but later when you try to add it to the current state you divide by 0.

In Java you would explicitly handle this case with a catch and take action on a "division by zero exception". In Erlang you would just let the process crash (and log the reason so bug can be fixed) and restart the state to what it was before the erroneous input, no matter what bug/case the wrong input triggered. By having this generic handling you will make a really resilient system since you don't have to handle every possible bug on a case-by-case basis.

What would be the advantage of handling this case in the Java defensive programming approach? Maybe someone will catch the error and return/introduce null to the state? Or BigDecimal.ZERO? Then you might end up in an unexpected state for all subsequent requests.

Well, division by 0 could be handled with pattern matching without crashing. Excuse my rusty pseudoerlang.

(edit) formatting

  handle_zero(data, 0) -> {:error, "division by zero"},
  handle_zero(data, _) -> {:ok, transform(data)}.

  handle_input(data) -> handle_zero(data, data.divider).

This doesn’t prevent bugs (if we’re talking about logical ones), it prevents extended outages because of them. In other words, if the application crashes you can log such fact and fix it later while your end users won’t notice a problem on your side in the right circumstances (unless it’s a crash loop which can’t be temporarily solved by restarting).

Or you can log it and never fix it. I noticed a flurry of crashes that came about on restarting the vm. Almost certainly transient startup race conditions. Causes effectively zero service downtime and no degradation of QOS for end user (other nodes in the cluster could service requests). But having a disorderly startup is less of a pain in the butt to program and may even result in a quicker time to start. So who cares. Never fixed it.

The point is the app doesn't crash. One process (object) crashes and immediately restarts. The entire rest of the app keeps chugging away. Compare to a language with one or a few main threads, where one bug can crash the whole system. It's a brillant scheme designed for systems required run for a decade or more with only minutes of downtime (telephone switches).

Detecting a bug, crashing, and then restarting, can be much easier to write than code which is bug-free. In fact, some times, you need so much code that it is likely you are introducing more bugs by trying to handle an edge-case.

A key point of Erlang systems, however, is that they are really good at reporting the state of the system when it crashes (due to functional programming, you have the state from before the crash happened, and what event lead to the crash).

The restart is a stop-gap measure that gives you service for the system as a whole. You can then look at the logged bug report and fix the problem. But you are in control of how quickly you want to fix the problem. There is a cost to fixing a bug as well.

It's more the subtle difference between robustness and correctness.

Yes, and isn't that amazing!

Bugs in production will happen in every language, and crashing one of many running processes and restarting it to get it back into a known good state sure beats having to scramble to find a fix, code it, build and deploy a new version of the code.

It enables systems with an uptime of 99.nine9s% that means less than 4 seconds <edit>downtime per</edit> year.

It’s all about resilience, not about preventing bugs completely. The beauty of it all is that you can skip on a lot of error handling code whose only purpose is to keep the system running. You only need to handle business logic errors. Let’s take a web server as an example.

The way the “let it crash” philosophy is done in Erlang is that you crash individual processes first (e.g. the current HTTP request from one browser). If that keeps happening there’s a counter that crashes the subsystem (e.g. the web file listing component). If that too keeps crashing the whole web frontend might crash, but the node might still be up (e.g. serving FTP or whatnot). And lastly the node will shut down completely if the web frontend keeps crashing.

This way, the rest of the system keeps performing and serving requests even if some parts are not working intermittently.

On top of this you of course have (built-in) logging of all these errors so you can investigate them.

As long as you don’t cause a crash loop you’re golden in my experience.

A crash loop would indicate something to be fundamentally broken in ones code. That's OK too, because then we know, we have to fix it.

Reply on news.ycombinator.com