Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

I love this.

This is another in the "Now that we have computers with gazillions of bits of RAM and many cores and gazillions of cycles, what conventional wisdom is no longer wise?" series :-).

I suggested to a client that they use Sqlite3 for their company database rather than a disk based database. And they looked at me shocked, as if I had spoken heresy. I pointed out how many records they could store on a 64GB machine and that is a small data center class machine and it sort of boggled their mind. Add read-only copies, shared from a NAS device gave them perfect scalability in terms of queries per second (their sales folks had their own database that they were updating). It was fun to watch their expression go from horror to wonder to excitement when they finally implemented it.

View on HN

Yup! I'm currently running a b2c solution for a client that's grossing ~$3 million/year using Python + sqlite3 behind nginx.

I work for another shop that scolds me for even suggesting using something else than k8s. How can a persistent file system possibly scale? Lol. Ironically they're grossing -$3 million/year.

I am curious about your case is the traffic mostly read heavy? Ha wish you could share more. Interesting stuff.

> I suggested to a client that they use Sqlite3 for their company database rather than a disk based database.

Sqlite is a disk based database...

Well, maybe not "disk based" per se. It can be stored on disk, yeah. But it can also run in memory. I think in this case it's clear that ChuckMcM meant a machine with 64GB of RAM (not disk space), therefore they are talking about having SQLite running in memory, not on disk.

Even if a DB is disk-based, you can throw it in a ramdisk to get the same speedups, often. I would be curious to know if there are any drawbacks or dangers from doing that, apart from the obvious “if your box powers off, the ramdisk is gone” and “each time the box restarts, you have to copy the db to the ramdisk.”

The main problem I was uneasy about was, I would simply use the equivalent of rsync -Pa /ramdisk /disk in a loop, which seemed to ensure safety in the event of a poweroff. It never broke, but I wouldn’t say it was a good idea, necessarily.

Sure speeds up most things though. The benefits are pretty crazy for I/O bound programs.

Does anyone know the risk of radiation bit flips in this case? I don't think it's an issue with ecc ram, but if it's not that could be a problem for certain applications.

Exactly! I was wondering about that. It seems like it's so rare as not to matter, but that's no guarantee it won't matter.

A friend of mine wrote a 'chaos engineering' tester for this specific issue: http://github.com/enferex/cme

Randomly inject bit flips into a target process memory space, so you can see/test how your software handles it (if at all).

Bit flip is always a risk even if you don't use ram disk.

>I would simply use the equivalent of rsync -Pa /ramdisk /disk in a loop

That's probably a bad idea. rsync isn't going to copy over writes in the same order they were written by the DB. Most DBs assume that when there's some crash that at least the disk would have been in a logically consistent state with respect to the order of certain writes separated by e.g. fsync and barriers.

'The most common way to force an SQLite database to exist purely in memory is to open the database using the special filename ":memory:".'

https://yourdatafitsinram.net/

The IBM 53-bit quantum compute result from a ton of hard drives + compute power was also intriguing (https://www.ibm.com/blogs/research/2019/10/on-quantum-suprem...).

128PiB of disk space needed for a 54-qubit computation on their system (hard drives). Fun to see where the supercomputers are still needed, but smaller qubits can be emulated on normal computers. 40 qubit calculations probably can be emulated on a 16TB team of hard drives in RAID0.

The trade-off isn’t really size in a lot of cases though. When stuff is stored in RAM you lose the persistence story.

Disk storage for the WAL and periodic checkpoints can get you sub-second recovery targets easily.

Ninja edit: or a UPS and a shutdown procedure triggered by it that writes the database to disk. You can write 100GB in a couple minutes over a 10Gbps network. Faster to a local SSD.

Sub-second recovery and UPS triggered shutdowns are still nowhere near the reliability of a database ensuring fsync has been called when you call “commit”.

There are just certain classes of data (e.g. any kind of business transaction) where you can’t afford this type of best effort durability.

Reply on news.ycombinator.com