Hacker News Reader

Comment by kstrauser | Hacker News Reader

Quorum doesn't let you ignore partitions. It can't. You still have to decide what happens to the nodes on the wrong side of the partition, or how to handle multiple partitions (you have 5 nodes. You have 2 netsplits. Now you have clusters of 2 nodes, 2 nodes, and 1 node. Quorum say what?).

I don't think TFA is nuts. I do think the premise that engineers using cloud can ignore CAP theorem is wrong, though. It's a decision you must consider when you're designing a system. As long as everything is running well, it doesn't matter. You've got to make sure that the behavior when something breaks is what you want it to be. And you can't outsource that decision. It can't be worked around.

View on HN

It's poor timing for this article.

The CrowdStrike incident has demonstrated that no matter what you do it is possible for any distributed system to be partitioned in such a way that you have to choose between consistency and availability and if the system is important then that choice is important.

You're ignoring that a system is temporarily partitioned for the duration of the time it takes to retrieve information from all nodes.

Partition intolerance means you can only proceed when all nodes have been reached. This means partition intolerance not only deals with availability in the yes or no sense, but also in terms of latency.

That's true, but I think the catch is that most customers likely didn't view it as an investment that could decrease availability.

> You still have to decide what happens to the nodes on the wrong side of the partition

You don't because their clients won't also be isolated from the other servers. That's TFA's point, that in a cloud environment network partitions that affect clients generally deny them access to all servers (something you can't do anything about), and network partitions between servers only isolate some servers and none of the clients.

That’s fine if you value availability over consistency. Which is fine if that’s the appropriate choice for your use case! Even then you have to decide whether to accept updates in that situation, know that they may never be reconciled if the partition is permanent.

No, you can have availability and consistency as long as you have enough replicas. Network partitions that leave you w/o a quorum are like the whole cloud being down. Network partitions don't leave some clients only able to reach non-quorums -- this is the big point of TFA.

> No, you can have availability and consistency as long as you have enough replicas.

You've misunderstood the problem.

You cannot be consistent if you cannot resolve conflicts, taking writes after a network partition means you are operating on stale data.

Honestly the closest we've ever gotten to solving CAP theorum is spanner[0], which trades off availability by being hugely less performant for writes. I'm aware spanner is used a lot in google cloud (under the hood), but you won't solve CAP by having hundreds of PGSQL replicas, because those aren't using Spanner.

In fact, you can test it out, ElasticSearch orchestrates a huge number of lucene databases, a small install will have a dozen or so replicas and partitions, but you're welcome to crank it to as many nodes as you want then split off a zone, and see what happens.

I'm becoming annoyed at the level of ignorance on display in this thread, so I'm sorry for the curt tone, you can't abstract your way to solving it, it's a fundamental limitation of data.

[0]: https://static.googleusercontent.com/media/research.google.c...

Did you read TFA? Please, the point of TFA is NOT that you don't need distributed consensus algorithms, but that once you have them you don't need to worry about partitions, therefore you can have consistency, and you get "availability" in that the cloud "is always available", and if that sounds like cheating you have to realize that partitions in the cloud don't isolate clients outside the cloud, only servers inside the cloud, and if there's no quorum anywhere then you just treat that as "the cloud is down".

I suspect you never needed CAP in the first place, because even outside of cloud, you don't seem to pay a lot of attention about whether your service is up or not or how to increase that uptime.

In the same spirit, some businesses are fine with a single db, making backups every night and losing some data when an issue happens. It doesn't mean that people maintaining these systems get to tell others that distributed systems are essentially a non-problem.

> you get "availability" in that the cloud "is always available",

The cloud isn't what's being described as available. The service is available or not. The service is hosted on multiple computers that talk to each other. If they stop talking to each other, the service needs to either become unavailable (choosing consistency) or risk having not up to date data (choosing availability).

I did, and it doesn't provide insight that didn't exist 15 years ago.

Cloud is always available? I have a bridge to sell you.

That presumes you can only have 1 partition at a time. That is not the case. You can optimize for good behavior when you end up with 2 disconnected graphs. It’s absolutely the case that a graph with N nodes can end up with N separate groups. Conceptually, imagine a single switch that connects all of them together and that switch dies.

You can engineer cloud services to be resilient. AWS has done a great job of that in my personal experience. But worst cases can still happen.

If you have an algorithm that runs in n*log(n) most of the time but 2^n sometimes, it can still be super useful. You have to prepare for the bad case though. If you say it’s been running great so far and we don’t worry about it, fine, but that doesn’t mean it can’t still blow up when it’s most inconvenient. It only means it hasn’t yet.

Again, a partition that leaves you with no quorum anywhere is like the cloud being down.

> Again, a partition that leaves you with no quorum anywhere is like the cloud being down.

What does this mean? What is the cloud being down? The cloud isn't a single thing; it's lots of computers in lots of data centres.

It means there will be an article on the front page declaring the outage

A partition that leaves you with no quorum anywhere is like the cloud being down if you value consistency over availability.

Yes, but again, TFA says that you will trust the cloud will be up long enough that you would indeed prefer consistency.

My goodness. So many commenters here today are completely ignoring TFA and giving lessons on CAP -- lessons that are correct, but which completely ignore that TFA is really arguing that clouds move the needle hard in one direction when considering the CAP trade-offs, to the point that "the CAP theorem is irrelevant for cloud systems" (which was this post's original title, though not TFA's).

The issue people are pointing is that the author conflates "Cloud is good enough for my needs" with "CAP is no longer relevant".The point is especially annoying considering that the author works at a cloud vendor and not a cloud user.

Huh? You can create partitions in ways other than issues with the underlying network. You can mess up firewall rules for example and find yourself with no quorum.

Those aren't the only failure modes - you can have two sets of servers partitioned from one another (in two different data centers), both of which are reachable by clients. Do you allow those partitions to remain available, or do you stop accepting client connections until the partition heals? The "right" choice depends entirely on the application.

In TFA's world-view clients will not talk to isolated servers.

So that would be choosing consistency over availability.

Yes. That's the point, that cloud changes the trade-off such that consistency becomes worthwhile.

Indeed and this is a very subtle point that I found very hard to explain when talking about availability in the event of network partitions.

You have to consider the system as a whole including external clients since the definition of the availability of the entire system ultimately depends on the point of view of the end users of the system.

Quorum just means you are a CP system rather than an AP system.

Reply on news.ycombinator.com