Thredis is Redis + SQL + Threads | Hacker News Reader

Thredis is Redis + SQL + Threads

thredis.org

94 pointsjanerik20 comments

Interesting! But not where the original Redis project is headed to (other than Cluster I do not want to add any major feature for a while), so it made a lot of sense to fork and hack on it as a separated project.

Agree completely, this is not where I want Redis should go, I was just tinkeing with the idea.

Antirez: are you considering a Sorted Map of some kind with a way to iterate over it? (ZSET is almost it, but semantics are a little different and Redis does not have iterators, at least not exposed to the protocol). With SQLite4 a Redis Cluster with such a type could serve as the underlying storage engine, basically creating an infinitely scalable database.

BTW - kudos on Redis, I LOVE the clean C code. I must note that in the process of hacking on Thredis I couldn't help noticing there are many "cultural" similarities between SQLite and Redis, e.g. fondness of Tcl, integers as strings, etc.

you can make your own iterator over sorted sets pretty easily with zrange

True, but not quite. The first thing zrange does is look for the starting point, which is an approximately O(log(N)) for a skiplist (which is how it's implemented). An iterator's next() operation should be O(1). If I need to iterate over a gazillion elements, it would make a huge difference. Or course you could zrange chunks, and it would be more efficient, but what's wrong with a simple START and NEXT semantic?

BUT this is all a moot point - a Sorted Set is not the type of structure that something like SQLite4 needs anyway. The keys should be sorted lexicographically, not by score. I think a Skip List fits the bill perfectly, and Redis already contains an implementation of it (as part of Sorted Sets), why not expose it as a standalone type?

Someone created something simmilar (Alchemydb) and later got acquired by Aerospike.

I think this approach manages to combine all the disadvantages and almost none of the benefits of SQL and NoSQL (and threads?). It's cute but seriously, where would this ever be practical? Consider the alternatives:

If you need concurrent client transactions over sets wouldn't you be better off just using a SQL database?

If you need fast KV, why not just use NoSQL over SQL ala memcache or something newer like memcache/innodb[1] ?

If you want complex updates over redis why not just use redis 's Lua interface[2]?

I think sqlite4's approach[3] (SQL above NoSQL) makes a lot more sense than this since it will presumably let me write a SQL application and deploy it with any KV store.

1- https://blogs.oracle.com/MySQL/entry/nosql_to_mysql_with_mem...

2- http://oldblog.antirez.com/post/redis-and-scripting.html

3- http://www.sqlite.org/src4/doc/trunk/www/design.wiki

Oh wow, this is awesome. I wish to put the bitcoin block chain into thredis tonight.

Probably a good idea to have antirez feedback on it, and see if this is something that will make sense in Redis overall. I think the dump as a SQLite DB file is a great idea since it can be potentially used by some other system by simply copying it and have plain old SQL applied to it.

What are the threads used for? It's mentioned in the page but not explained. Looks awesome.

They are explained here: https://github.com/grisha/thredis/blob/master/README-THREDIS

Initially Thredis was just threaded Redis, and I had a very specific use case for that. The idea of adding SQLite came later.

It's duct-taped together but cute as all hell, like a lil' Frankenstein's Monster

Redis used event loop model to handle requests. So basically, at any time, there is only 1 thread change data. This model make redis fast and still guarantee data is atomic. I wonder if thredis use thread how thredis solve that issue?

A single-threaded event loop gives you low latency... when you're not under load and servicing multiple concurrent connections. As the amount of load on Redis (or for that matter, any single-threaded program) increases, the amount of time Redis spends servicing other requests increases, and so the average latency increases proportional to load.

If Redis could actually process requests in parallel instead of one-at-a-time, this would not be an issue and Redis could actually have a reliable latency profile under load.

About the only solution in this case is to turn Redis into a distributed system via sharding/slaves, and as soon as you do that, you lose Redis's guarantees around atomicity. Furthermore, Redis provides few tools that are really essential in a distributed system, like read repair/quorums or a failover system that isn't a joke (i.e. sentinel, although redis-failover could fit the bill)

Threads are used because SQLite calls block.

I like it. I don't know that I would ever use it, but I like it.

Really cool little project. Kudos.

Someone posted about one of their projects here the other day, Akiban [0]. While not exactly the same it was also bridging the SQL/NoSQL worlds. Not 100% sure on how it works but it has table-groups to let you pre join collections together (I think).

[0] http://www.akiban.com/

No source code?

I don't know if I need this, but bravo for being crazy

That's insane. In an awesomely insane kinda way.

http://www.quickmeme.com/meme/3spmcc/

Reply on news.ycombinator.com