willbmoss

Turns out escaping json suck, so I decided to lean on Python. v2: https://gist.github.com/wmoss/4774406

At high concurrency, I'd argue you will probably end up being more cpu efficient as well. The cost of context switching effectively larger frames and getting into and out of privileged mode can get expensive.

We had two replica sets of three nodes each. After the first migration, we took it down to one replica set.

I agree with you in theory, in practice, my experience has been a bit different. Specifically,

1. Since Mongo has a database (or maybe collection now) level lock, doing rebalancing under a heavy write load is impossible.

3. Mongos creates one thread per connection. This means that if you've got to be very careful about the number of clients you start up at any given time (or in total).

I'll agree they are not operational nightmares, but now that we're set up with Riak we can do things that I'm pretty sure your Postgres/MySQL/etc. setup cannot.

1. Add individual nodes to the cluster and have it automatically rebalance over the new nodes.

2. Have it function without intervention in the face of node failure and network partitions.

3. Query any node in the cluster for any data point (even if it doesn't have that data locally).

I'm sure there's other things I'm missing, but the point made by timdoug is the key one. We're at a scale now where it's worth trading up-front application code for reliability and stability down the line.

In a nutshell, gevent monkey patches the socket library, whereas diesel doesn't. This means that you can use any (previously) blocking libraries with gevent, whereas, in diesel you have to write them again. The upside of the rewriting is that it creates a more coherent (and opinionated) ecosystem.

I added an implementation [1] in diesel [2][3], which uses select.epoll (or libev, on non-Linux systems) and got a around 150x speedup [4]. I only repeated the tests a few times (but they were all close) and didn't install the Go compiler so I could test against Go (I'd be interested to see how this stacks up on your machine). Like you say in your post, it's nice to have something wrap up the bother of epoll for you.

[1] https://github.com/wmoss/Key-Value-Polyglot

[2] diesel.io

[3] https://github.com/jamwt/diesel

[4] The first run is against the diesel one

wmoss@wmoss-mba:~/etc/Key-Value-Polyglot$ time python test.py

real 0m0.134s

user 0m0.040s

sys 0m0.020s

wmoss@wmoss-mba:~/etc/Key-Value-Polyglot$ time python test.py

real 0m20.164s

user 0m0.096s

sys 0m0.072s

Agreed, I should have clarified. We don't just use IRC, but IRC along with other tools to handle pasting text and file uploads. I think what has always bothered me about Campfire (which I'm sure can be solved) is that it's not easy to do thing from the command line. I want to be able to pipe the output of git diff to my pastebin, or upload a log file from a remote machine.

At Bump we tried Campfire, but ended up using IRC (and use an open source IRC bot, https://bitbucket.org/yougov/pmxbot/src). I'm curious why you decided to go with Campfire instead of IRC.

We (Bump) have found two bugs in Redis and both times we've worked closely with Salvatore and had them fixed in less than 48 hours. He's a great programmer who really cares about the product.

One thing you don't note in your post (but made a big difference for us) is that 2.4 uses jemalloc. This reduced memory fragmentation for us by around 25%.

Just as a word of warning, Redis 2.2 cannot read rdb or aof files created by 2.4 (meaning I'm pretty sure 2.2 can't be a slave of 2.4. So, if you have a fail over scenario, you might be forced to upgrade your master to 2.4.

Bitcask can guarantee one disk seek, whereas LevelDB will do one disk seek per level, so at least from that perspective, it can't be better.

Level also has to look down the entire tree if a key is missing. This means inserts end up being more expensive than reads or updates (which are all just a hash lookup in Bitcask).

Mathias, what Riak (or other distributed databases provide) that Mongo doesn't is "just-works" scaling characteristics beyond one machine.

* If I want to ensure that my data is written to three machines, all my writes will stop working in MongoDB if one machine goes down. With Riak, it will start issuing the writes to another node in the cluster and rebalance when the missing node comes back online.

* If I'm running MongoDB in a sharded configuration, if one of the shards cannot be reached, all writes will stop. With Riak, any node will accept the writes and, once the network issues are resolved, move them to the appropriate node.

That said, conflict resolution is hard and there's no real way to get around it when you're using a distributed database like Riak. With Riak, as Chad says you get "increased development complexity for massively decreased deployment complexity." There's no silver bullet, it's important to look at the trade-offs of each option.

It's worth noting that not only is the list keys operation expensive, but since it uses Bloom filters, it's not guaranteed to returns all keys.

My sources at Basho tell me that this is fixed in 1.0, but until that's officially released, basically don't try to list keys.

I wrote the post and to clarify, I didn't mean to suggest that interfacing with Android was impossible with Clojure, simply that is was easier with Scala. As rickmode points out, there are some things that make Clojure more difficult.

Is this what you are looking for? http://goloco.org/

Agreed, a very cool idea. I believe you are right on the parallelism front, cloud.call just offloads the processing on their servers, but you'd have to call it multiple times to get any parallelism.