Hacker News Reader

Comment by timhaines | Hacker News Reader

If you're thinking about using Riak, make sure you benchmark the write (put) throughput for a sustained period before you start coding. I got burnt with this.

I was using the LevelDB backend with Riak 1.1.2, as my keys are too big to fit in RAM.

I ran tests on a 5 node dedicated server cluster (fast CPU, 8GB ram, 15k RPM spinning drives), and after 10 hours Riak was only able to write 250 new objects per second.

Here's a graph showing the drop from 400/s to 300/s: http://twitpic.com/9jtjmu/full

The tests were done using Basho's own benchmarking tool, with the partitioned sequential integer key generator, and 250 byte values. I tried adjusting the ring_size (1024 and 128), and tried adjusting the LevelDB cache_size etc and it didn't help.

Be aware of the poor write throughput if you are going to use it.

View on HN

That's strange, that doesn't look like a normal graph to me, it looks like a cache or queue of some sort is backed up. Did you try to use dtrace / iosnoop / iostat etc to see what might be the bottleneck?

For average commodity hardware I found something like 400 reqs/s/node was normalish, even sustained. Yours looks like about 2 minutes in it dies. Come to think of it, could you have your open file descriptors limited in the OS settings? That looks just like pattern I'd expect to see from that.

Might be unrelated but common pitfalls I had were: - Using the HTTP proto. Protobuf is way faster. - You can tweak the r and w values to get less read and write consensus when you can afford to, depending on the task and data. - ulimit open file descriptors might be too low.

In any case, if you were to do a short writeup, I'm sure the basho guys at the mailing list would be interested.

Hey - the Basho guys were aware and reproduced it pretty quickly. They saw the same response from their new bloom filter branch they're introducing soon too.

I was monitoring with iostat and a couple of other tools. It was certainly very heavy on io, with 80% util, 20% iowait, and that increased as the currency went up.

I was using protobuf, and a w value of 1, so I was out of things to optimize.

When I was inserting objects already in Riak's cache, it ran about 3 times faster, but of course that's not possible with new objects.

How long after they reproduced did you give them to fix the issue? I looked up the thread on their mailing list and you seemingly jumped the gun a bit on your conclusions.

Feel free to investigate further. I had to move on.

So what you are saying is I was right. Thank you. People who report a bug and give less than half of a day for someone to investigate has never dealt with a vendor like oracle or IBM. This tells me you haven't had a data problem before and based on your willingness to give up so quickly leads me to believe you won't end up with data problems that this article is talking about anyway.

Ha. I've had and have plenty of data problems. After 2 days of making adjustments as per Basho's suggestions to try and improve the write throughput, I moved on. You seem to be making a lot of judgments and assumptions about that decision based on very little information. I guess this is troll food.

Meanwhile, back in Postgres-and-MySQL land we're wondering why we should have to entertain this kind of ridiculousness.

Riak loves random read/writes, spinny discs do not, try things out with a SSD sometime and watch things go from a shoddy XXX ops/sec to XXXX(X) ops/sec.

As a simple remark on this, I've gotten 1000+ ops/sec on a single machine operating as 3 nodes (equating to about 3000 ops/sec per node) when using an SSD and a measly 150 ops/sec with a spinny disc in the same setup (equating to about 450 ops/sec per node)

While SSDs will undoubtedly be faster that spinning disk, LevelDB is designed to address slow random writes by batching and writing sequentially.

That would be true except each vnode (by default 64!) has its own backend database. That means with 4 physical nodes each one gets 16 leveldb/bitcask/whatever database backends.

LevelDB's write batching and caching is completely void when thats considered in many circumstances.

Its something that I think Basho should consider changing. Its a trade off of fault tolerance or performance, and I would personally love to see riak go a lot faster.

Bitcask is specifically designed around not doing random I/O, particularly for writes. A bitcask back end is essentially a gigantic sequential transaction log.

Thanks for mentioning Basho Bench. Looks slick. For anyone else interested, it's at: http://wiki.basho.com/Benchmarking.html

The benchmarking tool is very slick. Easy to configure for a variety of scenarios, and once you figure out how to install R it produces those pretty graphs.

Major weaknesses in it I've found:

- The compare script is fragile. Often times it doesn't want to compare two tests I did with the same exact config, just flipping code I'm testing against.

- It doesn't have a good mechanism for storing auxiliary information. We end up faking errors for it but it just looks ugly and hard to distinguish a correct run from a bad one.

I had the same experience about throughput being a bit sub-par. For me it was a test on a single macbook pro with a regular 2.5" hdd. Which client did you use to write to riak? protobuf or http? Also: which language? did you use threading? Did you enable search?

Well, for the benchmark, I was using Basho's benchmarking tool which is erlang, and I was testing with protobuf. I had 5 concurrent clients running for the benchmark, but also tried with more and less, and got about the same results.

Search wasn't in use on the test bucket.

For my app, I'd integrated Riak using ruby.

Reply on news.ycombinator.com