A Brief History of High Availability | Hacker News Reader

A Brief History of High Availability

cockroachlabs.com

59 pointsEntICOnc21 comments

I was very unsurprised to see that this history starts a bit late. In 1992 I was working on HACMP/6000, which was a high-availability clustering product for IBM's AIX (though I worked at a company that was on contract to them). Databases were a primary use case, relying on shared storage instead of replication. We were hardly the first into that space BTW. DEC in particular had developed a lot of the concepts (e.g. locking models) that we and others at the time emulated, and the history in this space goes back even further than them. Were those systems as scalable as what we have now, or even as robust? Nowhere even close. But the erasure of that history still irks me. Modern distributed databases (and distributed systems in general) in their turn relied heavily on experience and concepts from those earlier systems, but rarely acknowledge the fact.

High-availability database systems go back to 1974 or so, with Tandem Computers.

https://en.wikipedia.org/wiki/Tandem_Computers

"Tandem's NonStop systems use a number of independent identical processors and redundant storage devices and controllers to provide automatic high-speed "failover" in the case of a hardware or software failure. To contain the scope of failures and of corrupted data, these multi-computer systems have no shared central components, not even main memory. Conventional multi-computer systems all use shared memories and work directly on shared data objects. Instead, NonStop processors cooperate by exchanging messages across a reliable fabric, and software takes periodic snapshots for possible rollback of program memory state."

The general ideas of failover for high-availbility go back much further.

https://en.wikipedia.org/wiki/Failover

"The term "failover", although probably in use by engineers much earlier, can be found in a 1962 declassified NASA report. The term "switchover" can be found in the 1950s when describing '"Hot" and "Cold" Standby Systems', with the current meaning of immediate switchover to a running system (hot) and delayed switchover to a system that needs starting (cold). A conference proceedings from 1957 describes computer systems with both Emergency Switchover (i.e. failover) and Scheduled Failover (for maintenance)."

Also worth a mention that the well known Paxos distributed consensus algorithm was described in a 1989 paper by Leslie Lamport, and the almost identical (but less well known) predecessor Viewstamped Replication in a 1988 thesis by Brian Oki.

I had the recent privilege of interviewing Brian Oki [1] of Viewstamped Replication in September last year, since VSR is the consensus protocol that we use in TigerBeetle [2] to process a million financial transactions per second.

One of the first things that Brian Oki mentioned was Tandem NonStop, and I understand that Brian's VSR was then the first consensus protocol to deal with network partitions, where the network is not reliable, for high availability.

What was also special was that James Cowling joined me afterwards—we covered Viewstamped Replication Revisited, the 2012 revision, with a focus in both interviews on Barbara Liskov's influence on both James and Brian.

[1] https://www.youtube.com/watch?v=ps106zjmjhw

[2] https://www.tigerbeetle.com

Tandem started selling their NonStop systems in the mid-70s, so clearly high availability was important.

At Basho we worked hard to acknowledge our debt to distributed systems pioneers. Not everyone is ignorant of that history.

Yup, the Basho folks were/are very good about that. BTW hi, we've quite likely met, though I don't know which Basho alum you are. ;) And yes, even I was a pretty late entrant to the HA game. I never worked on Tandem systems, but I worked with Tandem people at the job I mentioned, after having worked with both Stratus and Sequoia systems previously. Those were all FT rather than HA, but the point remains that work on making systems survive failures through redundancy goes back much further than the OP mentions.

Oh, definitely, was trying to add to rather than detract from your point.

I was mainly on the periphery at Basho, started as tech evangelist, migrated to engineering to help get 2.0 out the door. John Daily.

I always thought Basho was a great company with a great product. I felt like I learned a lot through Basho documentation. I never really understood what happened to the company though as it seemed highly regarded amongst the dev community as well and Riak was always mentioned in the same sentence as the other distributed data stores at that time.

Tough market, limited potential customer base for such a specialized tool, some product/release planning issues with Riak 2.0, and ownership structure that didn’t make anything easy at the end (but I’m far from an expert on this topic, especially the last one, just parroting distant memories).

Engineering was very talented, and the customer support team was truly top notch. Such a shame, best job I’ve ever had.

Here's a link to a PDF of the September 1987 issue of the Digital Technical Journal, which was all about clustering on the VAX/VMS machines.

https://vmssoftware.com/docs/dtj-v01-05-sep1987.pdf

Not only late, but strangely Google-centric as though we lived in caves before the Google monolith appeared.

Yes, there wasn't much "history" there. I worked on an AIX system that used HACMP. Before that, I used Veritas clustering and file system on Suns. Prior to that, I was at a VMS shop that made use of DEC's built in clustering.

Fun facts about availability:

1) the more nodes, the higher probability of failure, 2) the more load, the higher probability of failure, 3) complex logic and/or nodes that aren't tested frequently have a higher probability of failure

Basically, every person who has claimed "X is easy, I do it for my personal use and it's fine with one giant node, don't need HA" is true (for them) because it's literally less likely to fail. But if they had more load they'd see more failures, and probability doesn't mean they won't encounter the random failures that are much harder to recover when there's no backup system.

HA means you're going to have more failure, which means more work to keep it running. But it's necessary work if total system failure is unacceptable. Two engines in an airplane is twice the maintenance, but it's better than the only engine conking out mid-flight.

HA has a very sweet spot at the place where "the system is down at 3AM and must be fixed RIGHT NOW!" do not happen, but the complexity isn't increased enough for it to generate much ongoing work.

Anybody bellow that spot is just begging for being taken out by random at the worst possible times.

> 1) the more nodes, the higher probability of failure

I may be misreading your comment, but how is this true?

This is one of the classic problems of talking about HA: making it clear which "failure" or "availability" you mean at any given moment. In the particular quoted sentence, "failure" refers to component failure even though the "availability" in HA means system availability. The goal of HA design is to make a system that becomes more available as components are added instead of less (as you seem to expect). Redundancy, internal monitoring, and failover are the tools of the trade. What you should end up with is a system more like two chains holding up an object instead of one chain twice as long.

If you have one node with the probability of x% that it fails within a year and you add a second one with the same probability to fail the total pobability that one of the two nodes fail within a year goes up.

Not OP, but two things I’ve found to be true:

* the more complex the configuration required to provide high availability, the more likely something will go wrong and reduce availability

* the more nodes you have, the more likely one of them will fail. Whether that reduces availability typically depends on the failure mode and control layer.

For an example of the latter, with the distributed database Riak (and I imagine others as well) if a server is starting to fail (corrupt disk, bad network card, etc) you’re in a much better place if it stops working entirely. A gradual node failure is much much worse.

> second, by placing computers under constant demand, they were more likely to fail

Really? In what sense? Hardware fails or what? The complexity usually brings everything down.

Anyway, looking forward to cockroachdb, it seems it solves some high-availability / multi-site problems.

I think what the authors are getting at is that many failure conditions are more likely to manifest under load. This includes various kinds of resource exhaustion, including those induced by leaks. It also includes things like timeouts due to long queues, and even race conditions. The higher the load, or the longer it's sustained, the more likely it becomes that such bugs will manifest. In every such system I've seen, from very early days of the internet or old-school HA to modern multi-thousand-machine clusters, there have been bugs that only showed up under heavy and/or sustained load. Both Erlang's supervisor trees and Recovery Oriented Computing (Berkeley/Stanford) are based on the observation that such failures will always be with us, and have influenced many subsequent systems - arguably including CockroachDB.

Heat caused failures also increase with load, as does electromigration, disk stress, power failures...

The GP has a very digital model of computers. It's not really useful for predicting failure.

One concrete example involving DBs is the PostgreSQL TXID wraparound. It's basically 32 bit counter where if things go wrong or were built wrong it just keeps going up until it becomes a problem. If you're only doing <=1 request per second this won't ever be a problem since even if you ran it your whole life without freezing any old records it wouldn't approach max value. On the other hand if it's under a constant 1000 request per second load it can be the source of problems in about a month.

You could argue to just call the TXID "complexity" but that'd come with a wide enough definition of what complexity is that it just becomes a replacement for saying "buggy code" which isn't very useful. Most leave complexity as being more like "grossly overengineered" to leave room to avoid that.

Reply on news.ycombinator.com