Game day exercises at Stripe

We only started doing this about 3 years ago after I heard John Allspaw speak at Velocity. It has cut down dramatically the "Oh, that system that we thought was redundant was only redundant in the sense of an appendix, not in the sense of kidneys..." It was initially a hard sell to my ops team to implement it. (I promised them air cover for any losses we incurred as a result of game-day testing, of course.)

I agree with you that it should be common, but I'd suspect an honest survey of the field would show that it's far from standard practice. I'd guess fewer than 25% of companies do this in any meaningful way.

The main point of the post is that if there's a configuration you expect to work, you need to test it, independent of whether or not it's supported by the author. I can think of a dozen other distributed system failures I've seen that happened despite using stock configuration — this particular failure was simply a recent example, and helps illustrate why game days are so valuable.

On the plus side, Antirez has said that this configuration will soon be supported: https://news.ycombinator.com/item?id=8522630 (thanks Antirez!), so future users will be able to run in this configuration safely.

This is not what happened, a replica with persistence turned off restarted with an empty data set, and the slaves replicated that empty data set. Basically Redis replication currently must be used with some form of on-disk persistence turned on. However after the introduction of diskless replication (http://antirez.com/news/81) we now have a good reason to support replication with persistence turned down in a proper way, so there is already a feature work in progress in order to support the Stripe-alike use case: https://github.com/antirez/redis/issues/2087

Ever seen anyone drop tables from a MySQL master to have that statement replicated across the cluster? (I don't think this problem set is correlated with cache vs. primary store, but with the fact you have some kind of replication going on.)

No, their tests don't imply that they are getting better write times with redis across the board, they are measuring the 99%-tile latency.

What they have found, in a worst case scenario Redis is worse than Postgres.

So for example, their Redis response times might look like ,{1ms, 1ms, 1ms, 9ms}, while their Postgres response times might look like {3ms, 4ms, 3ms, 4ms}. Looking at it this way you can expect redis to perform better, however that outlier is worrying. For stripe they value having consistent and predictable performance over varying performance.

That's kinda an obtuse version of a middlebrow dismissal. You're positioning yourself as "of course—I knew better all along! Bow before my foresight!"

In other anecdotal evidence, I've been using Redis as a persistent store since 2011 and haven't lost any data.

There's always the giant caveat: just because software says it does X doesn't mean it does X until you have seen it actually happen. In this case, restarting a zero-persistence master was bad because it goes run->die->restart[empty]. Then, the replicas immediately reconnect, resync[empty], and now they are completely up to date with the master (which is the only Redis contract asked for here: always be an exact copy of your master; since persistence was not requested, the replicas re-sync an empty dataset).

In better news: there are already fixes for each of these issues showing up in Redis very quite Real Soon Now.

It also literally states it's a key-value store on their site as the first sentence. Redis is not just a cache anymore.

Looks like graphite:

https://www.google.com/search?q=graphite+graphs&tbm=isch