Hacker News Reader

Comment by pquerna | Hacker News Reader

in memory vs not, will always have these kinds of results.

most people, most systems, aren't so hot about in memory datastores -- MySQL had MySQL-Cluster/NDB, and for lots of reasons it had trouble ever taking off.

For certain use cases, and definitely for benchmarking purposes, in memory datastores will always crush the competitors, but in the end, most people like reliable data storage.

View on HN

Most use cases also don't have the entire data set equally active, and it's silly to pay to keep everything in RAM instead of just the hot parts.

That's part of the proposal: real world data requirements will soon completely fit into RAM available on a rather standard system.

In-memory doesn't imply "unreliable". If you need high-availability you'll want to use replication anyway.

In-memory with periodic snapshots sure implies "not durable," though. (That is, you will lose whatever data has not yet been snapshotted on a power failure or crash.) Replication doesn't fix that, unless you're willing to impose the latency of replication to different datacenters for each update. (I don't think Volt even supports this. Most systems don't.)

Why? If a single node fails, you still have the data on 2 other nodes. Nothing is lost. Of course, assuming you have a reliable UPS.

Because the data may not have replicated to the other nodes yet. As the parent comment noted, you could synchronously write to all replica nodes, but thats a big performance penalty and not partition tolerant.

They pay the penalty in latency, not in throughput. That's like no performance penalty at all if you work asynchronously. They make sure that the data cannot become inconsistent between nodes by ordering the transactions. The penalty that comes from this is that latency can become high, i.e. the answer time for individual transactions. While maintaining a maximized throughput of millions of transactions a second. It's like an internal 'pumping rhythm' with tightly synched timestamps.

VoltDB replicates synchronously.

So if your entire data center loses power... If you have any notice (local UPS, etc), you snapshot to disk. If you have no notice, you lose some number of transactions depending on your data size, snapshot frequency and disk speed. Seconds or minutes, but probably not hours.

Future versions of VoltDB will do more address this single-data-center catastrophe scenario.

Can anyone suggest problem domains where it is useful to have the ability to do 100K writes per second, but not have durable transactions?

I can see the in memory story for read only queries, or transient data.

Transactions ARE durable if you use replication.

can you elaborate more on those reasons? (just asking)

Reply on news.ycombinator.com