Hacker News Reader

Comment by a1369209993 | Hacker News Reader

a1369209993

If you're willing to accept that durability is a lie, and also to compromise (possibly a lot) on performance and trust that the CPU and RAM aren't actively helping the disk to sabatoge you, you can make a fairly full-featured ACI database in a weekend using Merkle trees and a single-threaded rendevous serializer. The hard part is getting those guarantees while also having latency and throughput vaguely comparable to naive file writes. (I'd add something about a better API than assembly language, but database people decided to imitate COBOL, so I can't really criticise assembly with a straight face.)

View on HN

Log structured merge is the last piece of your architecture. It solves the latency part. It can also solve your durability issues, if you fsync when necessary and you understand your hardware.

Yeah, sorry, I assumed log/journal plus reapply-on-recovery went without saying. You can't have guaranteed durability (short of blockchain-style distributed stores) because it's always possible for your database server to suffer a unrecoverable hardware failure between transmitting "transaction success" and taping out its weekly backup. If all else success, your datacenter catching fire and burning down would suffice. So it's only ever a matter of reducing (not eliminating) the probability of having to roll back commited transactions. (Tldr: fsync can't help if your hard drive melted.)

Oh, I assumed we were talking distributed. Fsync to more than one disk. My biases :).

Fsync to more than one disk is great way to reduce the probability of forced rollback, but simutaneous failure is a still possible.

Then again, ACI does assume you aren't having CPU/memory/etc failures, so to the extent that you count anything as guaranteed, it is probably possible to 'guarantee' durablity. My original point was more that "minimally adequate" isn't that hard compared to going from there to a high performance, general purpose system (and also a offhand snipe at SQL).

Reply on news.ycombinator.com