I like the part about three replicas of redis destroying all the data when the primary went down. If anyone out there is using Redis as more than a cache then you're doing it wrong.
I'm happy to see there will be features added to support the use cases that Stripe and others have in mind.
In a distributed system, some nodes can act like a cache and others like a persistent store.
Their setup would have been fine _if_ the master never came back up. If the master didn't restart, then something in their infrastructure would have probably started using the replicas for data (and/or they would have promoted one of the replicas to be the new master, then the other replica would replicate from the new master).
But—having persistence on the replicas did allow them to copy the dataset off to backup storage. Then they were able to restore the old data when it was needed. So, we can prove their setup was persistent since they lost all their data and recovered it. :)
The replicas will try to reconnect to their original master forever unless something else (like Sentinel) redirects them in an actual failover/promotion setup.
So, the master had data, it died, it restart with no data, then the replicas immediately reconnect. If the master had persistence enabled, it would have reloaded the old dataset on startup and the replicas would have re-downloaded everything—since they are replicas of the master, they will always prefer the master data over their own, even if the master is empty.
If you were in a strange case where the disk failed and you replaced it with an empty disk (is that what you mean by "fresh disk?") then it's the same as starting an empty dataset. Not entirely relevant since the server would be intentionally started empty after a maintenance action instead of just restarting the already-populated process that restarts as empty because there's no saved dataset to load on startup.
The "all replicas resync an empty dataset" is a logical consequence of the configuration they enabled, but one without obvious repercussions without either directly experiencing it or a longer multi-chain thought experiment. (but, fixes for such things are already on the way—soon!)
Funny enough what triggers this problem when you have master persistence turned down is, the lack of failover, if the reboot happens fast enough, in case you are using Sentinel, for it to failover to a replica. So no failure was sensed at all, just the master magically wiped its data set.
So from the point of view of distributed systems, if you want to analyze the sum of Redis replicated nodes + Sentinel, the problem is that the system is not designed to cope with nodes losing state on restarts.
However it is possible to improve it, and I'm doing it, but before diskless replication it was IMHO pretty useless to have support for persistence-less operations in conjunction with replication, since the slaves to synchronize required anyway the master to save an RDB file on disk.