We got bitten _hard_ in production when we accidentally allowed some of the nodes to get above 90% of the storage used. The whole database collapsed into a state where it could only do a few transactions a second. Then the ops team, thinking they were clever, doubled the size of the cluster in order to give it the resources it needed to get the average utilization down to 45%; this was an unforced error as that pushed the size of the cluster outside the fdb comfort zone (120 nodes) which is itself a problem. The deed was done though and pulling nodes was not possible in this state, so slowly, slooooowly... things got fixed.
We ended up spending an entire weekend slowly, slowly getting things back into a good place. We did not lose data, but basically prod was down for the duration, and we found it necessary to _manually_ evict the full nodes one at a time over the period.
Now, this was a few years ago, and fdb has performed wickedly fast, with utter, total reliability before that and since, and to this day the ops team is butthurt about fdb.
From an engineering perspective, if you aren't using java fdb is pretty not great, since the very limited number of abstraction layers that exist are all java-centric. There are many, many issues with the maximum transaction time thing, the maximum key size and value size and total transaction size issue, the lack of pushdown predicates (e.g., filtered scans can't be done in-place which means that in AWS, they cost a lot in inter-az network charge terms and also are gated by the network performance of your instances), and so on.
What ALL of these have issues have in common is that they bite you late in the game. The storage issue bites you when you're hitting the DB hard in production and have a big data set, the lack of abstractions means that even something as finding leaked junked keys turns out to be impossible unless you were diligent to manually frame all your values so you could identify things as more than just bytes, the transaction time thing is very weird to deal with as you tend to have creeping crud aspects and the lack of libraries that instrument the transactions to give you early warning is an issue, likewise for certain kinds of key-value pairs, there's a creeping size problem - hey, this value is an index of other values; if you're not very careful up front, you _will_ eventually hit either the txn size limit or the key limit. The usual workarounds for those is to do separate transactions - a staging transaction, then essentially a swap operation and then a garbage collection transaction - but that has lots of issues overtime when coupled with application failure.
There are answers to ALL of these, manual ones. For the popular languages other than java - Go, python, maybe Ruby - there _should_ be answers for them, but there aren't. These are very sharp edges. Those java layers are _also_ _not_ _bug_ _free_. So yeah, one has a reliable storage storage layer (a topic that has come up over and over again in the last few years) but it's the layer on top of that where all the bugs are, but now with constraints and factors that are harder to reason about than the usual storage layer.
One might say, hey, SQL has all of these problems too, except no. You can bump into transaction limits, but the limits are vastly higher than fdb and the transaction time sluggishness will identify it long before you run into the "your transaction is rejected, spin retrying something that will _never_ recover" sort of issue that your average developer will eventually encounter in fdb.
That said, I love fdb as a software achievement. I just wish they had finished it. For my current project, I have designed it out. I might be able to avoid all of the sharp edges above at this point, but since we are not a java shop, I also can't rely on all the engineers to even know they exist.
(see also https://aphyr.com/posts/283-jepsen-redis)
I care much less about how easy it is to use or deploy, but "good" is a subjective term, so other people might see things differently.
Jepsen has never tested FoundationDB, not sure why you claim this and link to Jepsen's site.
Aphyr / Jepsen never tested FDB because, as he tweeted "their testing appears to be waaaay more rigorous than mine." We actually put a screen cap of that tweet in the blog post linked here.
[1]: https://twitter.com/obfuscurity/status/405016890306985984
[2]: https://web.archive.org/web/20220805112242/https://twitter.c...
[3]: https://web.archive.org/web/20150325003526/http://blog.found...
They link to the website for a definition of the term they are using.