Hacker News Reader

Comment by superuser2 | Hacker News Reader

I understand that this is probably wrong somehow, but what about a system wherein there is sufficient capacity to take a "partitioned" datacenter out of the loop entirely and serve web traffic from your other locations, then bring it back up to speed while still offline after the disruption ends?

Again, I accept that that's probably bad, but why?

View on HN

Your strategy as I understand it is CP. In the case that you have a partition and it's not a majority, it stops accepting traffic or goes read-only.

Say an outage leaves your three DCs (or more generally, N partitions, of which none are a majority) unable to talk to one-another. They are all a minority and so all go offline or read-only. No partition can accept writes without risking that another partition is too, and that those writes will be inconsistent.

You're partition tolerant, and have consistency guarantees, but you're not available.

Can I "cheat" by having satellite, PSTN, cell phones, semaphore, smoke signals, sneakernet, etc. to allow the DCs to negotiate (perhaps randomly) which will consider itself the master? Then the non-masters could direct users to the master.

I guess this isn't a true "partition" and if we're talking about an internet backbone cable cut, it might go unavailable for some users who could not be redirected, but for the rest it is intact. While networking failures are to be expected, could I minimize my risk of a complete partition by sharing this status information? I suppose if we're in a cartoon end-of-the-world scenario where my 3 DCs drift out to sea on separate continents, I go unavailable, but at that point everyone has bigger problems to worry about.

If the PSTN, cell phone network, Internet, dedicated fiber link, shortwave radio, roads, and navigable airspace (drone, helicopter, carrier pigeon) between all 3 of my locations are severed, then the system goes completely unavailable, but at this point we're talking about a mass casualty incident, civilization collapse, or other situation where no one would be trying to use my system anyway. Medical records for first responders would have to fail to "eventually consistent" but patients can only be in one place at a time. Unless my system is the nuclear deterrent, does it matter?

The partition part is about how tolerant your system is to a partition situation. Having massively reliable communications between systems will reduce your MTBF, but it doesn't address how your system will actually handle a partition. You do need to account for it one way or the other. If you get a partition, you can't keep operating with consistency across the whole system.

Now when you look at patient data .. sure, eventually consistent might be just fine.. but that's not consistency as referred to in CAP. So while it's perfectly acceptable to your app, it's not about making CAP invalid.

>If you get a partition, you can't keep operating with consistency across the whole system.

But why can't I shut down some nodes and direct all my users to the other(s)? As long as I have enough of a channel to communicate which node is now authoritative (and if all systems including driving between DCs fail simeltaneously, nobody needs my app anyway) can't I keep serving my users out of one DC?

Thank you for engaging this thought experiment, by the way. This is fascinating stuff.

As the linked site notes, clients are part of the distributed system, which means they can be partitioned as well, both from each other (more important in a P2P app) and from your servers. So you may not be able to "direct all your users to the other(s)."

Reply on news.ycombinator.com