* You should never trust, and always verify, the claims made by database manufacturers.
* Especially when those claims relate to data integrity.
* Super-especially when every safety level provided by the manufacturer that includes the word "SAFE" is actually unsafe.
> Even though the Postgres server is always consistent, the distributed system composed of the server and client together may not be consistent. It’s possible for the client and server to disagree about whether or not a transaction took place.
If you want to avoid that kind of problem, use 2PC.
Do you (or the author) see this as a bug, or just something that might surprise people who haven't thought through the guarantees?
Quoting the article:
> The 2PC protocol says that we must wait for the acknowledgement message to arrive in order to decide the outcome. If it doesn’t arrive, 2PC deadlocks. It’s not a partition-tolerant protocol. Waiting forever isn’t realistic for real systems, so at some point the client will time out and declare an error occurred. The commit protocol is now in an indeterminate state.
It would be foolish for a client to issue a COMMIT and then assume the transaction aborted because of a connection drop. The client should wait until the connection can be reestablished and determine the real transaction state before making a decision based on it.
It's the same issue as with a power failure during fsync. The durability of that transaction is indeterminate, but it doesn't matter because the system is down. Before the system comes back, it will go through recovery, and either find the commit record or not, thus getting back to a determinate state.
http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-j...
Sadly, fdb has been bought by Apple[1], and you can't download it anymore. I sincerely hope foundationdb gets opensourced or something.
[1] http://techcrunch.com/2015/03/24/apple-acquires-durable-data...
Jepsen tests cannot be used to prove the system is safe, but to prove it isn't.
It looks like very often he looks into source code in order to figure out how the system operates, that way he can find weaknesses and write test to his testing framework to demonstrate the issue.
I wouldn't trust any company that uses Jepsen to show that their product is safe.
"During the network partition, no requests are successful" is not the best result for a CP system, IMO."
HBase should provide partial availability in the face partitions.
Also, if you follow the rest of the twitter conversation you may realize, as they did, that only requests to the minority partition are unsuccessful - which is exactly what you want from CP.
If you look at his articles: Redis, PostgreSQL, Cassandra, ElasticSearch etc all had data consistency errors. And none of those have vendors making any claims.
It's pretty sobering to say the least.
https://aphyr.com/posts/282-call-me-maybe-postgres
There were no acknowledged writes lost. The only unacked-but-successful writes resulted from a connection while a commit ack was in-flight. That doesn't qualify as a data-consistency error, it means the client has to check if the data is present after reconnecting.
But in no cases would the client reconnect to find that there were acknowledged-as-committed records that were missing or stale. In no cases would the client find that responded-as-rolled-back data was actually committed. This is very, very different than what is seen with MongoDB.
Furthermore, the postgres experiment only checked that no writes were lost. As Aphyr acknowledges, MongoDB did not lose any writes with "majority" write concern. The postgres experiment did not include the verification of a linearizable history of reads, which is what the bulk of the OP is about.
I'd like to see a similar experiment run against a replicated postgres configuration with auto-failover.
My broader point was that you need to assume the worst from your database in the application layer. Which I think you missed.
That article has various issues, for example, calling Postgres commit protocol as a special case of two phase commit is not really correct. Postgres has 2pc: http://www.postgresql.org/docs/9.2/static/sql-prepare-transa... but that was not tested.
The described behavior is "expected" and "understood". Saying that "you should assume worst from your database" is not something I would ever use for describing DB with ACID semantics.
And ACID does NOT guarentee that you will not lose data. It is a theory not an implementation. I have lost data with both Oracle and Teradata due to bugs.
Aphyr article about Postgres could be renamed to call-me-maybe-acid-db-over-the-network and could remain the same.
https://aphyr.com/posts/285-call-me-maybe-riak
Now I hear they support consistency as well.
[0] - http://www.kdnuggets.com/2015/03/interview-dave-mccrory-bash...
Basho needs to sell damn fast or just call it a day and open source the enterprise version.
Well in that case have you heard about LeoFS? Wonder if it overlaps with any of the Riak CS features for you.
Contact info in profile, this username at all the usual suspects.
Disclaimer: I work for Basho.
Here's what I mean. Think of all the nice things you expect to come out of Riak's theoretical basis -- bulletproof distributed writes, for example.
Well, the default last-write-wins writes aren't bulletproof. They clearly fail Jepsen [1]. And if you turned off last-write-wins, then you'd have to handle siblings -- and how you were supposed to handle them without introducing inconsistency was quite unspecified. Riak clients gave you no help there.
Or, as the Jepsen article says, you could use CRDTs. Before 2.0, Riak had exactly one CRDT, the counter.
Something that might be appealing to some is built-in MapReduce. On the forums, people would warn you not to actually use it unless you didn't actually want availability after all. I don't know if they ever sorted that out.
Another supposedly nice thing was Riak Search. A distributed DB with full-text search out of the box -- that sounds great, right? But there were two things called Riak Search, and the first one just plain didn't work. They deprecated it before it had a replacement, but the replacement was supposed to come in 2.0.
So, Osborne effect. When people gradually discovered that Riak 1.x was bad, and the response was "but Riak 2.0 will be great!", that's a great reason not to use 1.x. 2.0 took a very, very long time, enough time for customers and potential customers, including us, to find other solutions.
Not that it matters to me anymore, but it does sound like having all data in CRDTs is the only way to pass Jepsen. (Or to have your data be immutable, in which case Jepsen doesn't apply.)
The original Riak Search in vs 1.x was replaced with integrated Solr in Riak 2.x, and it works.
I'd love to hear more about your use case. Contact info in profile. I'm this username in all the usual suspects.
Disclaimer: I work at Basho.
If you wanna talk about it I'm this username at all the usual online places.
Disclaimer: I work for Basho
I'm this username in the usual places online if you wanna talk about it.
Disclaimer: I work for Basho
I understand you lost data with Oracle and Teradata.
1. Most big corporate vendors do not have systems which are very well designed. (1) The sale is made at a business level. (2) Most Oracle customers are not very tech companies, and have mixed quality employees. As a result, there is little pressure on building a robust, correct product, rather than one which meets a feature checklist. In addition, Oracle doesn't really recruit smart people (I know people who work there). It's just not very robust compared to something like PostgreSQL, which was written by Stonebraker, a legendary computer science professor and entrepreneur.
2. Still, more likely, the reason you lost data is because you didn't know what you were doing. Words like Eventual Consistency, ACID, Two-General's Problem, etc. are not just abstract. They have strict, formal meanings, and you need to understand what they do and do not guarantee. Otherwise, you will lose data again.
Missed this reply and thought it was funny. I work for one of the world's largest retailers and we are one of both Oracle's and Teradata's most loved customers. We have 4 Teradata DBAs provided BY Teradata amongst a team of 20 SQL Developers. We aren't messing around.
What YOU don't seem to understand is that bugs in your database can cause data loss. ACID or Strong Consistency will not save you.
Regardless, your comment was about PostgreSQL, not Oracle. Oracle is a giant piece of software written by a corporation with over 100,000 employees. Something like that is bound to have bugs, and it has bugs indeed. Data corruption with Oracle certainly happens. PostgreSQL is written by a small, ultra-elite team. It's a much smaller codebase, so an expert developer can understand how the whole system works. There's a big difference in robustness between the two.
Of course database bugs can cause corruption. I've certainly had MonogDB eat my data. But the level of robustness of different databases is very different. There are many databases which are essentially bug-free. If you're losing data with PostgreSQL, odds are you're the one losing the data, not PostgreSQL.