Hacker News Reader

I was stuck in the planning phase for a pet project up until a week ago, for a reason why Lisp and Smalltalk always made me nervous.

Bugs can put your persistent state into uncharted territory, and there may be no clear path back. There's a reason we still have 'turn it off and back on again' in our bag of tools. Often it's the only thing that reliably works. This makes systems designed never to be shut down and turned back on again deeply unimpressive to the more practically minded members of the audience.

One of the places persistent state crashes and burns is when the system of record and the source of truth combine. Once the source of truth breaks we can't know what's true anymore.

My breakthrough was figuring out that I'm trying to get data from something that is already a pretty good system of record, I can just keep letting it do that job indefinitely. My source of authority needs to transform that data, not own it. As long as I can detect out-of-band changes to the system of record (which I can), I can always rebuild my models from scratch. That allows me some excellent test fixtures. That also gives me the option to do manual 'surgery' on the system of record rather than sinking my roadmap to implement a full feature to do something I might not need to do again for another year, or that no customer will ever see.

I am just on the same continuum as everybody else. My code needs to make a lot of decisions. I can't afford to make them all from first principles every time. I need to store intermediate values. I also need to identify intermediate values and question them. If I don't get help with this from my architecture, myself and coworkers will blur the lines every time a problem seems a little beyond our abilities, and eventually nobody will know what's true anymore except the delusional ones. How do I know this? Because I have never seen any other outcome. The only differences are in how much surprise the team exhibits.

> There's a reason we still have 'turn it off and back on again' in our bag of tools. Often it's the only thing that reliably works.

An alternative perspective—at the very large scale, the opposite may be true. Turning something off and then on again may be one of the worst ways to fix problems with certain large systems.

At a large scale, you would document and manage the state of the system, make changes which put that system into a new state, and build tools that monitor and enforce system state (declaratively where possible, but it’s not always possible to be declarative).

Sometimes there will be duplicated effort between making sure a system runs smoothly and making changes while it’s running, and making it so the system can turn off and on again. Think about the duplicated effort between setting up a new database and making schema changes to a live production database.

YES, you want to have a source of truth. But the “turn it off and on again” source of truth is usually a sequence of imperative commands rather than a description of the desired system state, and comes with its own reliability problems. A description of system state as source of truth has a different set of reliability problems but I think it’s absolutely necessary to explore the solution space between these two extremes—shooting for a completely declarative system is futile, but an imperative system can be very hard to reason about.

Yeah I think the whole “need to reboot to get to a clean state” is maybe a bug not a feature.

It is certainly possible to build systems that can run for years or decades.

I’m guessing that persistent memory will require us to think about changes to state in memory are more like ACID in databases. We should think about rollback instead of reboot.

> is maybe a bug not a feature.

Once you start thinking of safety as a feature you've ruined your ability to think responsibly about anything else. A reset is a safety mechanism. You shouldn't need it, but only a sociopath would try to get rid of it.

Almost all of the software we write is now affecting people's lives. That your bug may eat someone's term paper has been a problem for decades, and nobody ever internalizes how that affects that person's life. The avalanche of cause and effect may never settle down. Now we can get people fired. Ostracized. We are trying to take the power but not the responsibility, and it's bullshit. No, hypocritical bullshit, because we spend a lot of time here complaining about others doing the same thing.

I think that there are non-sociopathic reasons to explore interesting areas of designspace. It might turn out to be safer in the long run, and until it is, nobody's forced to use experimental software.

> You shouldn't need it, but only a sociopath would try to get rid of it.

> Almost all of the software we write is now affecting people's lives.

Don’t twist a technical disagreement into some kind moral outrage. We both have the same goal of providing reliable software to people.

The problem is that for the past decades, resetting the machine has deleted countless term papers. Nowadays, people generally put their term papers in cloud services that are always on. This is a huge step forward. We can bring this kind of reliability to offline usage. I think this requires taking lessons learned from cloud/server farm/mainframe world (always-on) and figuring out ways to apply them to personal computers.

The main lesson we can take is to design our software with the assumption that the machine can be turned off (or crash) at any time. That is obvious when you’re in a browser or running in a VM, but it was very far from engineers’ minds 20-30 years ago.

Most note taking software I’ve used doesn’t even have a save command. That’s a huge step forward.

The problem isn't so much about the correctness of a record, or being able to recover from changes in state, it's more about ensuring the consistency of the states of multiple different parts of the overall system. How do you recover consistency?

Even our brains have not entirely solved this problem. Sometimes when you hear a stimulus such as a phone ringing different parts of your brain will receive and process the stimulus at different times, so it can seem as though you become conscious that the phone has rung before you become consciously aware of hearing the sound of it ringing.

In a computer system you might have some parts working based on derived values from a record in the source of truth. This can lead to other parts working based on other derived values based on a different value when it has changed, so they disagree.

It is possible to build such reliable systems, but it is not economical for the vast majority of software that gets written — at least with current tools.

If getting something production-ready takes X time, making sure that thing doesn’t ever need to be restarted takes many times X, not to mention the huge investment in testing to ferret out those rare edge cases that cause hangs or inconsistent state.

Someday when software engineering comes into its own as an actual profession, maybe our tools will also achieve a higher level of consciousness, as it were. But nothing mainstream today comes close to enabling this.

Sometimes the solution is different because the scale is different.

Sometimes, the solution is different because we think we understand a problem that we really don't. I'm not ready to allow that the current wisdom on system design is not what Tony Hoare was thinking of when he said, "a system so complex that there are no obvious problems".

Arguing such things with people only seems to stick when we are collectively in the Trough of Disillusionment, which may be coming up pretty soon. So ask me again in 18 months.

> Turning something off and then on again may be one of the worst ways to fix problems with certain large systems.

From an overall system-wide perspective, sure. But from an individual component perspective, it seems to work pretty darn reliably. There's a reason why Erlang/OTP (as one example) has such a reputation for fault-tolerance and robustness.

And on that note, Erlang's model of supervised preemptive processes seems like it'd be a good fit for unified-memory computing, especially if taken to an "everything is a process" extreme.

Not only that, but in order to build a fault tolerant system you __WANT__ your components to fail fast and hard. Anything else runs the risk of your system doing the wrong thing while you have no idea it's happening.

failing is noisy, it's a very clear signal to the system to pay attention.

> the “turn it off and on again” source of truth is usually a sequence of imperative commands rather than a description of the desired system state

Not quite sure if it's related, but this is why I've been hesitant to go down the Docker rabbit hole that so many devs are pursuing these days.

On the one hand, I absolutely see the value of 'disposable' containers: discarding flaky instances and rebuilding a pristine one from a version-controlled config (i.e. "turn it off and on again").

However, the mechanisms to do that (re)building seem to consist of an unreproducible opaque binary blob (a "base image"), which gets augmented by a sequence of highly imperative commands; usually doing completely unpredictable things like 'apt-get update && apt-get install -y foo'.

I want the former, but the latter seems like a massive step backwards. I've seen some work on building Docker "images" using Nix, so that's the path I ultimately want to try. Colleagues may take some convincing though!

Building Docker images is now in nixpkgs, I think the function is named "buildDocker".

If you're deploying to a system under systemd, you can run anything that looks like a sysroot using nspawn. I find this much more enjoyable than going through the huge sidecar that is the Docker runtime.

I fear you are suffering from misconceptions about how containerization works. It's as if stating that a compiler is invoked every time when a process starts.

Most Docker images consist of multiple superimposed filesystems. When a container is started, a new empty layer is stacked on top which captures all filesystem writes during the container's execution.

This allows the following features:

- After the container stops running, these layers can be combined into a new image. This is what happens when you build images.

- The original image is never changed and can be used at any time to start a new container.

- When a new version of the application is released, it is not necessary to rebuild the whole image from scratch. You just start from the layers below.

Wifi routers have this notorious problem with the nvram. Sometimes some state in the nvram is corrupted and the routers would stop working. Turning off and on again won't work. A factory-reset to wipe everything out is the only way to get out of the corrupted state.

Around the time of Randall Monroe's "shibboleet" comic, I had an ISP connection that would flake out. If nobody kicked the cord on my end, the same series of events played out every time:

Check the cables.

Turn it off for 30 seconds.

Check the connection to the router.

Check the lights.

Try random other shit.

ISP reboots their modem instead of mine.

Networking restored.

Now, in theory, when a commercial modem loses carrier it should reset the connection completely. This they kept telling me, every four or five months for two years. And every time I got to sit patiently while their support person learned that this is not always the case, over the course of the next 20 minutes. If I tried to tell them about last time, then they would verbally pat me on the head and keep wasting my time. Whatever cooperative game or The Killer Is About to Reveal Himself was now a distant memory.

> This makes systems designed never to be shut down and turned back on again

Not necessarily; you just need a way to bootstrap the memory. Plan9 is pretty easy to run a fully functional terminal from memory/network access. Live CDs work pretty well, too.

As systems become very large, ‘turn it off and back on again’ is often counter-productive to the point of being the most destructive thing you can do. Repairing state is slow and expensive, and meanwhile the world is happening. This is the reason why all ultra-scale systems use some kind of “k-safety” durability model. The system would effectively be unrecoverable if you actually rebooted it because it would struggle to get back online. This is increasingly the rule rather than the exception for data infrastructure.

Even in the handful of recent systems I worked on where there was an explicit external system of record and rebooting was an option operationally and technically achievable, we still had to do semi-exotic custom engineering to make sure it was practically achievable.

When I started in the industry, ‘turn it off and back on again’ was the default strategy. These days it is rightly reserved for the dwindling number of cases, outside of small systems, where it still works. The computational cost of restarting is prohibitive.

Right the problem is really about maintaining consistency, how do you checkpoint the entire system and all your different components with their derived values.

I used to work on a system with a distributed hierarchical object store. We had copies of the data in various geographic regions, with copies of the same or different applications running in each region connected to a local copy of the store. When you write to your local copy, the storage system would propagate changes out to the other copies.

Maintaining consistency was a constant problem. One way to handle it was to give each region running a copy of a given application it's own regionally named folders in the object store.

The problem is similar to when you have application or database clusters with nodes that lose contact with each other, so called split brain syndrome, and strategies for determining which node should become authoritative. This is where the term STONITH comes from - Shoot The Other Node In The Head.

You might not want to restart the whole system, but you do need to be able to tell subsystems that they need to refresh their state. A reboot is simply a manual implementation of that, but in a continuously running system you need to build algorithms and strategies for deciding when to do such refreshes in the system itself.

> OS/400 had a 20 year run; released in 2008, end of life in 2018

Err, setting aside the math inconsistency there, AS/400 was released in 1988, and the 2 most recent versions are still supported..

[0] https://www.ibm.com/support/pages/release-life-cycle

Sorry, I meant 1988. My bad. Wikipedia has old info about versions and end of life.

They are not only still supporting it, it is a current product under a new name (IBM i). We just bought at S924 last year running IBM i.

OS/400 and AS/400 are some of most underrated and unknown machines.

AS/400 (I only used it back in those days, I don't know enough about newer series). Would run for years with no attention given to them. I saw people stuff it in a closet and forgotten about it.

Some did have the smarts to rotate backup tapes then they knew where it was.

If the machine needed help it would "call" IBM and report it, and there would be a technician there to help it before the customer knew there was a problem.

As with many things, I wish IBM would make it GPL, but the changes of that happening is very low. Also the programming languages is not familiar to many

The current machines we have (S924) calls IBM by itself. It had a bad RAID card so an IBM technician showed up and replaced it. It is a bit strange as a System Admin being the second one called.

We do a daily tape backup. LTO-7 is big enough to hold all the financial data. There is also a couple of companies that will do automatic offsite backup for a very reasonable price. They will even stage the data to a hosted environment until you can get a replacement machine (and I suppose a replacement building).

I do wish IBM had a simulator or small machine that programmers could use. The cost of entry to write software for the things is way too high.

I would like a BSD / MIT license as I am a BSD fan. Honestly, I wish I had time to work on a non-UNIX non-Windows Clone Operating System. Ideas like the AS/400 / IBM i architecture are amazing.

Real applications don't hit CPU walls, they're bottlenecked by I/O more often, and the biggest CPU wall is memory accesses. There is a lot of room for innovation there if you shirk off POSIX compatibility.

Also, state is not a problem for functional semantics. Remember "functional" is a semantic distinction, it doesn't imply a ton about optimized implementations of those semantics. For a stateful system the only thing different about a functional approach is the programmer cannot express a program where yielding next state invalidates current state. Most of the time a sound, functional program can be compiled into in place (mutable) operations on the state, the difference is they're free of logical errors.

You are making the author's point for them :-). Now add in that some of the stuff that was "custom hardware" that supported the concept of 'always resident' OS state is now off the shelf (specifically Optane/Crosspoint stuff) Of course Intel killed of consumer Optane for SSDs so one wonders what is going on there.

The second point, about how to think about GPUs in the context of "modern" computing is, for me, the more interesting one. As mobile processors have for years been a GPU with a processor accessory laptops and desktop computers seem to be headed that way as well. To be honest I'm surprised there isn't a GPU "motherboard that you can plug one or two CPU daughter cards into.

I wasn't aware that intel killed Optane-Only SSDs for consumers so I looked up and here's a recent article about it: https://www.tomshardware.com/news/intel-kills-off-all-optane...

I think the answer as to why Optane consumer SSDs were killed off is fairly clear.

They were 5 or 6 times the price per gigabyte as NAND based SSDs and didn't offer more performance in most consumer applications. The SOHO community didn't like them all that much either because the consumer Optane SSDs lacked most of the low queue depth performance that the enterprise disks are known for. Most consumers are also replacing their SSDs long before they actually wear out, so the extreme durability of Optane is kind of moot there.

It's a fantastic enterprise product, but I don't think most consumers are really getting any extra value out of them over NAND at this point. The only guy I knew who bought one wanted to put their Gentoo build directory (/var/tmp/portage) on it.

[Author of the piece here]

Thanks for the comment.

You do make me wonder if you listened to the whole talk, or read the script (https://docs.google.com/document/d/1wM1-c7euvQaRaCL4hKCaE8VR...)

I specifically addressed and discussed Lisp, Symbolics, & OS/400 (which is still alive and well).

IBM i (as it's now called) seems to more or less treat everything as a single giant disk than as a single memory space, though. However, it's been hard to find solid info for non-IBM-specialists.

It is not at all focussed on GUIs, no. However, I needed something pretty to put on the slides.

The focus is explicitly, as discussed, on end-user general-purpose computers, not servers. Servers are very well-served (pun intended) already.

Yes, I _know_ that this is not all-new. Indeed, that was one of my core points. The reasons are:

[1] I think there is more significance to something obscure or niche that has survived for decades and is still actively maintained and used than almost any flashy new offering which could fizzle out again just as fast.

[2] Using existing tech means that there are people out there with experience in it. It is not all-new for everyone. That's helpful.

[3] I personally am not very interested in GPU tech, and I think the new Apple M1 ARM chips bear out why: the standalone super-powerful GPU with its own RAM is probably a temporary artifact of PC designs and the gaming market.

I believe there are only two problems in computing: how data is divided up, and how data is bound together. They function nearly the same because you can use one to get the other, but the choice of which one to apply is critical and influences the shape of the media built on top. Where division is easy, things get shoved into countless categories. Where binding is easy, everything gets glued together.

With respect to the issues of operating systems, "data" maps to "hardware resources". With execution logic, "data" maps to "software architecture". And so on.

[Author of the piece here]

Those are not things of central concern to me currently, but another project that has some similar central axioms is looking at just that.

This talk is long but goes into quite a lot of depth. You might enjoy it.

https://www.youtube.com/watch?v=bSNda9EzNOI

>If you want to think about a new generation of operating systems, think about what to do in the GPU space. That's the future of hardware. Single CPUs have hit the wall on speed, but there's plenty of room for GPUs with more functional units.

Will it be more like the web with HTTP?

As long as the problem you are trying to solve is shaped like rendering pixels on a screen, GPUs are great! The future is multicore processors not bottle necked by the bus to a GPU. And coroutines.

OS/400 is very much alive, its called System i now.

Wait, OS/400 is going to be EOL-ed? How, why? :-/

It's not, bad source apparently misled commenter.