Hacker News Reader

I was agreeing with you until

> And this most of the times does not happen too.

This rubs me very very wrong. There is no maybe in data integrity. Redis wasn’t designed to be ACID and shouldn’t be treated as such just because “usually” it doesn’t lose data “under the right circumstances.”

If a sysadmin uses an RDBMS with relaxed fsync, they know what they are getting (and durable writes is not one of them, no usually about it). Same for Redis.

> sysadmin

we live in the age of devOps. there's no such thing as sysadmin, or god forbid, DBAs.

/sarcasm

For real! Where did all those DBAs go??

They retired and since no millennial wants to work for an Oracle degree no new ones came in to the field /s

They're charging $400 an hour to come in and unfuck years of devs thinking they can design databases because they read a MySQL How-to Article

We gladly paid one 800 € for three hours of work.. fixing a botched stored procedure on IBM DB2. Gray haired DBAs with the right blend of dev, ops and most importantly business domain knowledge (something many people in IT seem to overlook) are worth gold

infrastructure as code == infrastructure by coders

What I meant is that beat effort consistency can be super naive or can try hard, even without guaranteeing it, to avoid losing data in certain failure modes.

> If a sysadmin uses an RDBMS with relaxed fsync, they know what they are getting

Removing fsync is generally referred to as "running with scissors" mode, although postgres now supports it better with unlogged tables.

Building computer systems / architectures is always about compromises. If your requirements are that "everything must be perfect always", you will never deliver anything.

I came up with a "mental framework" that allowed me to get beyond these details early when I started architecting systems for companies.

1) Identify what are the possible malfunctions

2) What mechanisms will I use to detect these malfunctions as early as possible

3) Do I have a clear and concise recovery path for said malfunctions.

This frame of thought has taken me far (in terms of being able to accomplish a lot in my career so far) and allowed me to move forward without (a) extensive, time-consuming, research into the specific tools I'm using into the specific situations which may never happen, (b) extensive costs hiring experts in these niche areas.

ACID guarantees feel like they can be distilled to "everything must be perfect always" - and you can ensure ACID and still ship, just by using an ACID database. It's a hard problem, but it's also a solved problem.

Sometimes you don't need ACID guarantees, which is part of the reason databases like Redis exist, and is why the compromises you're describing are important. But in this context it sounds like the framework you're describing would lead to rolling your own ACID for Redis instead of just using an ACID database to begin with.

I guess it's almost like speaking two different languages when talking about what a DB guarantees and what your application REQUIRES in the form of guarantees. Typically, in my experience, they're not the same thing. Or not ALWAYS the same thing. Within the problem space you're trying to come up with a solution for, I'd say it's highly unlikely "ACID" guarantees are sufficient to say categorically (with 100% accuracy) that "my application is perfect because: ACID".

agree with you (computerGuru) at large scale any rare or corner case happen multiple time per day :-)

Redis was never designed for strong Synchronous replication of transaction or Strong leader election.

So each time a master crash you have a high chance of some recent acknowledged write being lost.

I'm one of those people with moderate throughput production applications using Redis as a primary store (500-2000 requests per second); in a multi-region active-active setup without redis enterprise no less. Its a very real use case at a LOT of companies.

Favorite datastore to date. The only thing I wish for frequently is native global change-data-capture without doing something manually with streams (I've thought about writing a replication protocol consumer, but the way resync's are handled is a little problematic)

How are you doing active-active?

piggybacked off Dynamodb in AWS for now unfortunately; so its more "the entire system is active-active, not just the redis bit". Reads always go to Redis in every region (no read-through), writes go to dynamodb (in global table config). Then there's a worker in each region consuming the dynamo stream and keeping it's region's redis in sync

Worker is a DIY stream consumer implementation. The stream -> lambda integration works, but its super limited (and ~300ms slower to get new events on average)

* writes are multiplexed to redis and dynamo in the origin region.

Why is the origin region Redis not also a reader of the replication stream? Just for latency?

yep. in this specific use case there can be pretty fast reads after a write, and dynamo stream latency is not great even intra-region.

Via the lambda integration you should probably expect ~1000ms p99 latency intra-region, and ~2100ms p99 from, e.g, east-1 to west-2. Shave maybe 200-300ms off of those numbers if you do a DIY stream consumer without the aws provided lambda connector.

Former DBA here ... you don't disable fsync.

No but often you set it to fsync every second and not at every write, if you are really concerned with performances in certain kind of applications. Or in practical terms, check the synchronous_commit config option of PostgreSQL. Basically sometimes you have fsync enabled because you don't want to corrupt the log, but the writes will not wait for the WAL, so the last transactions can get lost on crash.

I would just throw more money at the problem not weaken the strength of my confidence in the underlying storage subsystem or the DB by playing with the fsync behaivor willy-nilly nor even in a calculated way. I would instead just throw more money at the problem by investing in faster and faster disks -- you can do a lot with enterprise grade NVMe now.

I think what the author means is:

> "Remember redis (in AWS) holds ephemeral data in memory"

(because EC2 machines don't persist their disk data between start/stops and a lot of people don't realize that until it is too late - hey remember that Bitcoin exchange that got bitten by this?)

I suspect that’s what you meant but just for clarity, Instance storage does persist for reboots (which is effectively an OS reboot), but not if you stop/start (or terminate) the instance.

Correct, I've updated the comment.

Yes if you do an ordinary OS reboot the disk remains.

This was mostly applicable to older generations of EC2 instances (C3, M3, etc.)

EBS is now the default when creating an instance, and instance types of newer generations generally don't have instance store at all (except disk optimized instance types, and those with "d" in the name.)

Yeah, I think Amazon gave up on trying to communicate the meaning of ephemeral storage, and I think that's for the best tbh. It's too easy to have important data vanish that way. Something as simple as someone issuing a shutdown from the console is translated into an "instance stop" in AWS, which would purge the storage -- unfortunately often a lesson learned the hard way.

Will this announcement be Day 1 or Day 2? Looking forward to it as may solve some decisions we will be doing going forward

Today starting at 6:30 Pacific time or alike. Not sure about the time of this specific announcement.

Use a redis cluster to store session data, and then wonder why so many users get double login forms at times.

Bad idea. Bad tool for the job.

The querying capabilities of SQL shouldn't be ignored though, another issue is that your data must fit in memory.

thats not really relevant, I don't think anyone assumed it was always a drop in replacement for sql

Anyway if you want to see a version of Redis where you can store also bank accounts, transactions or any other of the most critical stuff you can store in a DB make sure to check the news at Redis Conf 2020 (the conf is in streaming and free).

He sort of implies here, not for you, but it might create that impression on others, was clarifying for those.

Regardless of whether Redis is meant to hold ephemeral data in memory, it is well-known to anybody who has had to maintain a Redis instance as part of their deployment that Redis is only effective when used for ephemeral in-memory work.

I understand your defensiveness, but please understand that because of your massive experience with Redis, either you have never lost data with Redis, which means that nobody besides you understands how to run your software in production, or you have lost data with Redis, which means that it's somewhat hypocritical of you to insist that Redis is durable.

> that Redis is only effective when used for ephemeral in-memory work

Hogwash and there are plenty of examples out there that prove otherwise.

> somewhat hypocritical of you to insist that Redis is durable

If Jepsen has demonstrated anything, it's that no database is as durable as it claims to be.

There are lots of dbs that do very well under Jepsen tests.

I can't think of a single Jepsen test that hasn't demonstrated issues with data loss.

Just going from the last one I read for DGraph, it did extremely well. Pretty sure etcd did well.

They always have bugs somewhere, but there are huge differences between bugs that show up for very specific, niche cases, and normal "I wrote to the db and it dropped it".

From the latest test:

"We found five safety issues in version 1.1.1—some known to Dgraph already—including reads observing transient null values, logical state corruption, and the loss of large windows of acknowledged inserts."

Loss of large windows of acknowledged inserts. Durability is hard.

I don't really feel like playing the quotes game... but, sure.

"All of the issues we found had to do with tablet migrations"

"ndeed, the work Dgraph has undertaken in the last 18 months has dramatically improved safety. In 1.0.2, Jepsen tests routinely observed safety issues even in healthy clusters. In 1.1.1, tests with healthy clusters, clock skew, process kills, and network partitions all passed. Only tablet moves appeared susceptible to safety problems."

No one is here to claim that anyone is getting through any kind of rigorous testing without bugs found. But there is a huge difference between "My extremely common write path + a partition = dropped transactional writes" and "Under very specific circumstances, with worst case testing, multiple partitions, and the db in a specific state, we drop writes".

There is an ocean between, say, mongodb's test results, and Dgraph's.

Read Redis's evaluation, for example: https://aphyr.com/posts/283-call-me-maybe-redis

"If you use Redis as a queue, it can drop enqueued items. However, it can also re-enqueue items which were removed. "

"f you use Redis as a database, be prepared for clients to disagree about the state of the system. Batch operations will still be atomic (I think), but you’ll have no inter-write linearizability, which almost all applications implicitly rely on."

"Because Redis does not have a consensus protocol for writes, it can’t be CP. Because it relies on quorums to promote secondaries, it can’t be AP. What it can be is fast, and that’s an excellent property for a weakly consistent best-effort service, like a cache."

Again, Redis is a very different type of database, so expectations should be aligned. Further, this test is quite old.

But that's a huge difference from DGraph's results.

Basically, saying "Well no one does well on Jepsen" isn't really true. Lots of databases do well, but you have to adjust your definition of "do well".

(author of Dgraph here)

As staticassertion is mentioning, some of the violations that were found were only around tablet moves, which happen only in certain cluster sizes and quite infrequently. Of course, Jepsen triggers those moves left-right-and-center to evoke some of those failure conditions; but that's not how tablet moves are supposed to work in real world conditions. This is different from other edge cases like process crashes, or machine failures, network partitions, clock skews, etc., which can and do happen. In those cases, Jepsen didn't find any violations.

We were planning to look into those tablet move issues and get them fixed up (shouldn't be that hard), but honestly, the chances of our users encountering them is so low that we de-prioritized that work over some of the other launches that we are doing.

But, we'll fix those up in the next few months, once we have more bandwidth.

Note that Redis only provides best effort consistency: it means that sometimes it can lose acknowledged data in special conditions (during failovers, or during restarts with a relaxed fsync policy). So it will never pass Jepsen tests in the default setup, but may pass it only when Redis is used with a linerizable algorithm on top of it that uses Redis as a state machine for a consensus protocol. But this does not mean that people haven't applied Redis for data storage with success. For instance many *SQL failover solutions, also can't pass Jepsen tests, yet people use them to store real world data. There are a lot of applications where to be able to really scale well and with a low cost (spinning sometimes 1/10 or 1/100 of nodes) makes perfectly viable to pick a system that is designed for that, and as a price, will lose a window of acknowledged writes during failures, while trying hard to avoid it in common failure scenarios.

Sure, I was speaking more broadly. Jepsen isn't a meaningful tests for a lot of databases, or use cases/configurations of databases.

> restarts with a relaxed fsync policy

I can't think of a single database that's solved the fsync/O_DIRECT issue on Linux completely. Postgres had to patch it _again_ last year.

As with all things it totally depends on the scale you're at.

Redis is an extremely reliable service. I've never "lost" data with Redis.

Your comment seems to suggest that everyone that's ever used Redis except antirez has lost data. I don't think that's true. I've used Redis since 2012 across dozens of production services and never lost data, and I don't have any particular skill that makes me special.