Cassandra Hits One Million Writes Per Second on Google Compute Engine

That is 8 terabytes (n1-standard-8) of RAM for 18.6 * RF gigabytes of data or am I wrong? The entire thing should fit in the page cache assuming compaction keeps up. There are no deletes so no tombstones. On an overwrite workload I don't know what the space amplification will be and whether you will actually run out of RAM for caching.

90% of people who read a benchmark aren't going to look at it the way I do. They don't have a cost model that says how fast things should be and how much they should cost. I do and when things are fitting in memory I have a different a different set of expectations. If I am looking at the wrong instance type please let me know.

For the workload you described Cassandra shouldn't be doing random IO. I would expect there to be three + N streams of sequential IO. The write ahead log, memtable flushing, compaction output, and N streams reading tables for compaction.

All the write IO can be deferred and rescheduled heavily because fsyncs are infrequent. Reads for compaction are done by background tasks and shouldn't effect foreground latency.

If read ahead is not working for compaction (and killing disk throughput) that may be something that needs to be addressed. Compaction should be requesting IOs large enough to amortize the cost of seeking. Page faults in memory mapped files don't stop read ahead and I think that the kernel will even detect sequential access and double read ahead. For a workload like this with no random reads you could configure the kernel to read ahead 2-4 megabytes and the page cache would probably absorb it fine.

For the workload you described the only fsyncs would be the log (every 10 seconds) and memtable flushes/compaction finishing and that would normally be so infrequent as to not move the needle on overall IO capacity (although obviously it consumes sequential IO). You set trickle fsync so we are still talking about 10 megabyte writes.

Granted there are many things I don't know about Cassandra. I've plumbed a lot of it, but it isn't my day job. Using a 24 gigabyte heap and a 600 megabyte young generation is questionable to me. I think Cassandra can do better, and I also think there are several tools that would do the same job with 10-20x less nodes by exploiting the fact that they never have to do random reads from disk.

Can any person really bring up a cluster of this size? On AWS, you need special permission to launch more than ~20 instances of one type, and it's granted only after you make a business case for it, which they won't grant to regular peons. With my regular Google account that doesn't have any kind of special approval, can I really launch 330 instances within a few minutes?

According to Netflix article the AWS experiment did run at a cost $561 per 2h, that is ~$280 per hour. Perhaps they were not utilized the cluster fully in those 2h in which case we should multiply the 1h test that performed 500k inserts per second, in that case the cost would be $182*2 = ~$365 per 1h.

GCE test did run at the cost of $330 per hour. Give or take few dollars difference if anything it's surprising GCE can do at roughly the same cost what AWS was capable of 2+ years ago.

Saying all that GCE guys did a great effort. I wonder though how much speed you can squeeze from AWS and at what cost now when AWS is sporting SSD disks.

If the people running GCE see that people are not using it because it sucks at X, they will probably do their best to address X. So complaining is a good thing, especially if it contains enough information in order to understand the problem (e.g. latency from/to where? internal network or external network?)

If Compute Engine is Google exposing internal infrastructure as an additional product, its quite likely that lots of people are using it (and that it is strategically important for it to be more usable) even if few people are paying for it.