Rediscala: Non-blocking Redis driver for Scala | Hacker News Reader

Rediscala: Non-blocking Redis driver for Scala

github.com

67 pointsetaty16 comments

Interesting. It feels like it bleeds a bit too much of the underlying "Akka-ness" out into the actual everyday implementation of the library, which makes it feel slightly more boilerplatey than perhaps is needed.

I noted that Debasish Ghosh recently updated his redis client to have Akka as the underlying handler as well.

Bleeding out the akka-ness is a feature, at least if you live in akkaland. For example:

    path("get_a_list" / PathElement) { externalKey =>
      get {
        complete {
          for {
            internalKey <- redis.get("internal_keys_from_external":externalKey)
            finalResult <- redis.smembers(internalKey)
          } yield (finalResult.toList)
        }
      }
    }

Spray already expects an akka future, as does everything else, so this library fits in perfectly.

And it appears to use futures that are part of Scala core (which Akka now uses as well) so I'm not sure how much this can be said to be Akka-ness anyway.

You are welcome to create a issue or a pull request.

Rediscala needs an akka actor System and sometime replies are of type ByteString.

What the diff with https://github.com/debasishg/scala-redis ? Almost every scala redis driver supports futures and is easy to wrap it into futures. Also by making all redis call async we have noticed performance issues as redis serves only 1 req every time there was too much time spending in thread-join. We found that the best practice is to have non blocking requests (methods) and all the redis reads in a blocking way. The only way to gain speed it to have N redis CPU's with N threads but that's almost impossible.

It looks like the asynchronous interface to debasishg/scala-redis is synchronous under the hood, using a pool of blocking connections. This means that there is no pipelining. Each of these blocking connections must wait for a request to travel over the wire, be handled by redis, and travel back before it can be reused. An asynchronous client allows full pipelining automatically, which is a much more efficient use of a single connection.

> we have noticed performance issues as redis serves only 1 req every time there was too much time spending in thread-join

Performance should be better with an asynchronous client. When I've benchmarked e.g. jedis vs various async clients, that's been the case.

The main difference are the performance and a better support of commands (almost all of them are available right now). Rediscala matches redis type answer to scala types.

Redis serves 1 req at a time, but you can write to Redis socket and read from Redis socket during that time.

Rediscala (the RedisClient) use 2 actors: The first one handle I/O (I/O bounds). That actor also merge redis requests in a buffer, so there is less messages (but bigger) going to the TcpWorker (akka worker handling a tcp socket). The second decode replies (CPU bounds)

Really interesting. I'll try the Rediscala driver and let you know about performance then.

A few suggestions: you could remove this kind of idiom: .map( result = { ... }) In the blocking commands example the snippet resembles the Javascript callback hell when it could be simpler to read: https://gist.github.com/felipehummel/6187248

Also, even simplifying it, it still seems overcomplicated. The returning type: Future[Try[Option[(String, ByteString)]]] is too much. Can't you remove the Try or the Option? Futures can already hold either a Sucess or Failure, so isn't the Try redundant?

I removed `Try` in the version 1.1

I'm inclined to think that finagle-redis is a much less bulky (akka) library to choose.

https://github.com/twitter/finagle/tree/master/finagle-redis

@etaty Thanks much for this effort! Looking forward to a user guide. Also did your redis-benchmark gets to 200k+, if not this will be one hell of performance mark. And even if it does, what the heck, its very hard to find use case which saturates this rate of inserts(except may be data generation frequency in cern reactor)

Yes, it does 200 000 requests/second

Benchmark : http://bit.ly/12QZsRs

Readme : https://github.com/etaty/rediscala#performance

Cool. There is actually another similar project from mDialog here: https://github.com/chrisdinn/brando

May be helpful to compare the underworkings to see different approaches.

Thanks. Rediscala doesn't have a shard manager yet. Brando version looks interesting.

I'd love to see this setup in the web framework benchmark (maybe even with spray) to see just how fast it can be. I may give it a try if I find some free time.

Reply on news.ycombinator.com