> Could the reason your DB is so fast because it's missing something in terms of safety/reliability that other databases do? That would be one thing I'd be concerned about hearing about a 100x performance increase.
That's a great question! First off, the database does already implement "ACI" from "ACID" (currently uses snapshot isolation but I have a design for serializable isolation, which shouldn't impact transaction latency by much). But the main reason it's so fast is that it eliminates IPC (this is a single-node system, so network latency isn't a factor). With 2 IPC round-trips for begin/commit transaction, it's impossible to get transaction latency below 10-20us. But after eliminating IPC, it's pretty easy to get latency under 1us (I currently have update transaction latency <200ns and I'm not done optimizing, perhaps even <100ns is possible).
The elephant in the room here is durability, which is perhaps a more nuanced question in the domains I'm targeting than many HN readers would expect. There are a few points I'd like to make about durability:
1. Full transactional durability cannot be achieved on generally available hardware (even NVMe SSDs) without fatally compromising latency. Durable transaction latency can never be lower than round-trip storage latency, which for an NVMe SSD is at least 10-30us, 1-2 orders of magnitude higher than the sub-microsecond latencies I'm targeting.
2. Full transactional durability isn't necessarily a requirement for many applications that use an in-memory database primarily as a concurrency control mechanism rather than as the system of record for data. Many of these applications require no durability at all (think Software Transactional Memory), and many others only need on-demand checkpoints.
3. Even traditional databases like Postgres are rarely run in fully durable mode (much like they're rarely run in fully serializable mode), because of the inevitable compromise to latency (and throughput, in poorly designed systems).
4. What seems to be to be an acceptable compromise to me is to offer on-demand checkpoints (with the option to run automatically on system exit), along with asynchronous durable logging that can keep up with throughput and doesn't compromise latency. I'm also considering an API that would designate an individual transaction as a "durability synchronization point", so that once commit_transaction() has returned, the caller knows that transaction and all previously committed transactions are durable.
I'm fairly confident that with careful design (io_uring and O_DIRECT) I can make SSD I/O throughput keep up with in-memory transaction throughput, but I can't be sure until I prototype it.