Hacker News Reader

> Turning something off and then on again may be one of the worst ways to fix problems with certain large systems.

From an overall system-wide perspective, sure. But from an individual component perspective, it seems to work pretty darn reliably. There's a reason why Erlang/OTP (as one example) has such a reputation for fault-tolerance and robustness.

And on that note, Erlang's model of supervised preemptive processes seems like it'd be a good fit for unified-memory computing, especially if taken to an "everything is a process" extreme.

Not only that, but in order to build a fault tolerant system you __WANT__ your components to fail fast and hard. Anything else runs the risk of your system doing the wrong thing while you have no idea it's happening.

failing is noisy, it's a very clear signal to the system to pay attention.

Yeah I think the whole “need to reboot to get to a clean state” is maybe a bug not a feature.

It is certainly possible to build systems that can run for years or decades.

I’m guessing that persistent memory will require us to think about changes to state in memory are more like ACID in databases. We should think about rollback instead of reboot.

The problem isn't so much about the correctness of a record, or being able to recover from changes in state, it's more about ensuring the consistency of the states of multiple different parts of the overall system. How do you recover consistency?

Even our brains have not entirely solved this problem. Sometimes when you hear a stimulus such as a phone ringing different parts of your brain will receive and process the stimulus at different times, so it can seem as though you become conscious that the phone has rung before you become consciously aware of hearing the sound of it ringing.

In a computer system you might have some parts working based on derived values from a record in the source of truth. This can lead to other parts working based on other derived values based on a different value when it has changed, so they disagree.

It is possible to build such reliable systems, but it is not economical for the vast majority of software that gets written — at least with current tools.

If getting something production-ready takes X time, making sure that thing doesn’t ever need to be restarted takes many times X, not to mention the huge investment in testing to ferret out those rare edge cases that cause hangs or inconsistent state.

Someday when software engineering comes into its own as an actual profession, maybe our tools will also achieve a higher level of consciousness, as it were. But nothing mainstream today comes close to enabling this.

> is maybe a bug not a feature.

Once you start thinking of safety as a feature you've ruined your ability to think responsibly about anything else. A reset is a safety mechanism. You shouldn't need it, but only a sociopath would try to get rid of it.

Almost all of the software we write is now affecting people's lives. That your bug may eat someone's term paper has been a problem for decades, and nobody ever internalizes how that affects that person's life. The avalanche of cause and effect may never settle down. Now we can get people fired. Ostracized. We are trying to take the power but not the responsibility, and it's bullshit. No, hypocritical bullshit, because we spend a lot of time here complaining about others doing the same thing.

> You shouldn't need it, but only a sociopath would try to get rid of it.

> Almost all of the software we write is now affecting people's lives.

Don’t twist a technical disagreement into some kind moral outrage. We both have the same goal of providing reliable software to people.

The problem is that for the past decades, resetting the machine has deleted countless term papers. Nowadays, people generally put their term papers in cloud services that are always on. This is a huge step forward. We can bring this kind of reliability to offline usage. I think this requires taking lessons learned from cloud/server farm/mainframe world (always-on) and figuring out ways to apply them to personal computers.

The main lesson we can take is to design our software with the assumption that the machine can be turned off (or crash) at any time. That is obvious when you’re in a browser or running in a VM, but it was very far from engineers’ minds 20-30 years ago.

Most note taking software I’ve used doesn’t even have a save command. That’s a huge step forward.

I think that there are non-sociopathic reasons to explore interesting areas of designspace. It might turn out to be safer in the long run, and until it is, nobody's forced to use experimental software.

Sometimes the solution is different because the scale is different.

Sometimes, the solution is different because we think we understand a problem that we really don't. I'm not ready to allow that the current wisdom on system design is not what Tony Hoare was thinking of when he said, "a system so complex that there are no obvious problems".

Arguing such things with people only seems to stick when we are collectively in the Trough of Disillusionment, which may be coming up pretty soon. So ask me again in 18 months.

> the “turn it off and on again” source of truth is usually a sequence of imperative commands rather than a description of the desired system state

Not quite sure if it's related, but this is why I've been hesitant to go down the Docker rabbit hole that so many devs are pursuing these days.

On the one hand, I absolutely see the value of 'disposable' containers: discarding flaky instances and rebuilding a pristine one from a version-controlled config (i.e. "turn it off and on again").

However, the mechanisms to do that (re)building seem to consist of an unreproducible opaque binary blob (a "base image"), which gets augmented by a sequence of highly imperative commands; usually doing completely unpredictable things like 'apt-get update && apt-get install -y foo'.

I want the former, but the latter seems like a massive step backwards. I've seen some work on building Docker "images" using Nix, so that's the path I ultimately want to try. Colleagues may take some convincing though!

I fear you are suffering from misconceptions about how containerization works. It's as if stating that a compiler is invoked every time when a process starts.

Most Docker images consist of multiple superimposed filesystems. When a container is started, a new empty layer is stacked on top which captures all filesystem writes during the container's execution.

This allows the following features:

- After the container stops running, these layers can be combined into a new image. This is what happens when you build images.

- The original image is never changed and can be used at any time to start a new container.

- When a new version of the application is released, it is not necessary to rebuild the whole image from scratch. You just start from the layers below.

Building Docker images is now in nixpkgs, I think the function is named "buildDocker".

If you're deploying to a system under systemd, you can run anything that looks like a sysroot using nspawn. I find this much more enjoyable than going through the huge sidecar that is the Docker runtime.