The bigger they are, the harder they f(etch)all | Hacker News Reader

The bigger they are, the harder they f(etch)all

redislabs.com

16 pointsitamarhaber17 comments

This is a first: a title change that is dramatically less informative than the original title. Please do not change the title of submissions. The guidelines are fairly clear on the policy and all you end up doing is making more work for "the ever vigilant deity of community goodness and truth in titling," aka dang.

  > please use the original title, unless it is misleading or linkbait.

http://ycombinator.com/newsguidelines.html

Sorry - no intent to mislead here... we were debating internally between the two titles and although we finally opted for the 500M one, I guess I liked this one better and it stuck in my subconscious, hence the submission.

The original title: I Have 500 Million Keys But What’s In My Redis DB?!?

Doesn't this point to a design issue with redis? The single threaded, blocking operation of it works, but it seems to present more problems than it solves for a large quantity of use cases. While it works, and works quickly for a narrow range of use cases, it comes at a huge loss in flexibility when compared to an SQL DB.

While I think it is possible to use redis well, I'm at a loss for what redis does for the average user, that a decent use of SQL doesn't do. My gut feeling is that if people would spend 30 minutes reading up on how to read a postgres query plan maybe they wouldn't reach for redis so quickly.

I don't think single- vs. multi-threaded is not relevant in this context. Any database, regardless its architectural design choices, will probably have a hard time if you try process this much information.

As for when to use SQL and when not to, well, that really depends on the problems that you're trying to solve, your preferences and a whole myriad of other considerations. I fully agree there are tasks that can be handled well by more than one solution, while others are best tackled with specialized tools.

It is very relevant because with a multithreaded DB you can survey your pkeys while only minimally interfering with the day to day operations of the database.

I understand that it's a judgement call, I'm merely asserting that most people make the wrong judgement because they want to play with new tech.

You can do the same with Redis. Regardless of the fact you can simply hang a Redis slave and work on that; Redis has a *SCAN commands that can incrementally iterate over your entire keyspace or keys of any Redis collection (hashes, sets, etc).

For a simple "fetch one row by pkey" lookup a traditional SQL DB spends something like 2/3 of the execution time parsing the SQL. So using a binary protocol's always going to win on performance.

(You may be right, maybe the main value of redis is as a way to work around stupid corporate/DBA policies? I know in theory I could have a single table key / json value in Postgres, but I've never found a company willing to actually do that. Whereas in Redis it's "no, I can't define a schema for the data, because the tool doesn't have one.")

You are 100% right about the perf there, but most people don't really need to optimize those lookups, they just like to because engineers like shiny things.

I think it's more that engineers are (rightly) lazy. Redis feels very simple - install it, run it, do your key/value operations.

No database system is perfect for all use cases, that's why we have options.

I agree, I just think redis is mostly used as a premature optimization

Redis offers several data structures that make it far simpler a solution than SQL for certain uses.

One example is its lists: http://redis.io/commands#list

I'm actually quite familiar with Redis, but most of the commands aren't meaningfully more simple. Redis' greatest advantage is performance.

From the example:

  Size range
  0-49
  600-649
  ...

Why does the range skip from 0-49 to 600-649? I'm not sure I get this.

I'd assume, without further information, that the random samples the script grabbed didn't have any sizes in range 50-599 inclusive. While that seems a bit odd, it doesn't seem impossible to me.

An alternate explanation (and in this case also the real one :)) is that the database doesn't have any keys at that specific size range.

Ah! heh, thanks for the clarification - I somehow expected it to have a row, with a count "0" :)

Reply on news.ycombinator.com