Comment by sradman | Hacker News Reader

Comment by sradman | Hacker News Reader

Martin Kleppmann applies critical thinking to the claims of NoSQL marketing; I wish I was aware of his work at the time.

I always found it frustrating that the semantics of the word "Consistent" in ACID has nothing to do with the semantics of CAP "Consistency". ACID Consistency refers to integrity constraints while CAP Consistency seems to refer to Cache Coherence across nodes in a cluster. I was a little surprised that Kleppmann uses the idea of "Linearizability" to explain the discrepancy but it does clearly identify the same mistaken assumptions.

I was also frustrated with the NoSQL movement's focus on Partitioned clusters when my intuition was that it was a very rare failure mode. Perhaps we can now return to the metrics of data loss (RPO - Recovery Point Objective) and Recovery Time Objective (RTO) since we seem no closer to fully transparent failure recovery.

Regardless, the goal was always to have available and scalable databases that operate on clusters of commodity cloud servers and all of the SQL, NoSQL, and NewSQL solutions have converged around that goal. The transition to cloud computing seems to have been more about NoRAID than NoSQL. Some technologies like LSM-Trees address important use-cases in commodity clusters so maybe we/I shouldn't focus on the truthiness of the original rhetoric.

>I was also frustrated with the NoSQL movement's focus on Partitioned clusters when my intuition was that it was a very rare failure mode.

Network errors are very common among errors database has to deal with, and particularly common when we are talking about networks between datacenters.

Agreed, network errors are very common but the partial failure scenario, with a fully functional client-server connection but failed server-server replication connection, seems rare. Maybe I don't read the network failure post-mortems carefully enough. When routers fail or are misconfigured, all hell breaks loose rather than clean partitions occurring.

One place I’ve run into these kinds of errors frequently was in a heavily locked down corporate LAN. All kinds of messed up firewall rules doing wild things. As an example, had a node with a misconfigured egress rule. Trying to connect to it, the node would receive the SYN, but the SYN/ACK would get eaten on egress... so for 60 seconds both sides believed a connection existed, but you couldn’t do anything. Then it times out and starts all over again.

The classic would be a geographically distributed datastore with intra-datacenter link being fine, but inter-datacenter being down/degraded. Clients within the datacenter would still be able to talk to the server fine, but server to server inter-datacenter is not going to be OK.

Another part of it is that all hell breaking loose can sometimes look very similar to clean partitions, depending on the network. When network QoS classes are available, what appears to be increased latency or packet loss for one class may be 100% packet loss for the other. Though, you're totally right that all hell breaking loose can be very different - the set of systems that handle flapping & other ugly connection failures well is smaller than the set of systems that handle clean disconnections well :)

Clean network partitions do happen, although it's certainly not a weekly occurrence.

For instance, a metro might be isolated from the rest of the network due to simultaneous fiber cuts.

https://www.bbc.com/news/technology-50851420

Or we could see heavy, prolonged network congestion on the backbone, impacting comms to/between backend services, while frontend serving is unaffected.

https://status.cloud.google.com/incident/cloud-networking/19...

(Incidentally, IIRC, Youtube's video serving infra was unaffected, but the backend jobs for youtube.com were largely inaccessible.)

Or improperly-executed maintenance that isolates a metro by draining routers in the wrong order. Insufficiently-provisioned network capacity (or network augments lagging far behind). Backbone traffic engineering gone awry. Underwater fiber cut due to shark attacks. QoS inversion (i.e. user-facing traffic as high priority, but replication as low priority) causing a partition during periods of heavy network load.

While yes, a lot of these scenarios might have (and did!) impact Google's serving capability, Google servers were still IP-reachable. 8.8.8.8 should have still been able to serve DNS (albeit with potentially stale data).

(In more recent news, if you use a transit provider to provide connectivity between your datacenters, and said transit provider suffers a multi-hour outage, that's a clean partition.)

>partial failure scenario, with a fully functional client-server connection but failed server-server replication connection, seems rare

You're more likely to see it when you are running services across multiple datacenters/availability zones/regions/whatever. It still happens frequently enough that it's a scenario worth keeping in mind. You probably have your application servers local to your database servers in that other datacenter, so if your public facing network falls over in a datacenter, there's probably still some amount of work happening that modifies things in the database - what happens when that connectivity is restored?

Being able to gracefully handle network partitions is hugely important in distributed computing.

Reply on news.ycombinator.com