I would like it to raise an error and then provide an option to continue or stop. Since continuing is the default, we need a way to opt in to stopping on checksum failure.
Not all checksum errors are impossible to recover from. Also, as the post mentions, only some non important pages could be corrupt too.
My main complaint is that it doesn't give developers an option.
If what we're really interested in is the log part of a write ahead log - where we could safely recover data after a corruption, then a better tool might be just a log file, instead of SQLite.
Attempt to recover! Again, not all checksum errors are impossible to recover. I hold the view that even if there is a 1% chance of recovery, we should attempt it. This may be done by SQLite, an external tool, or even manually. Since WAL corruption issues are silent, we cannot do that now.
There is a smoll demo in the post. In it, I corrupt an old frame that is not needed by the database at all. Now, one approach would be to continue the recovery and then present both states: one where the WAL is dropped, and another showing whatever we have recovered. If I had such an option, I would almost always pick the latter.
You do have backups, right?
Instead of that, I'd prefer for it to fail fast
It seems like you're focusing on a very specific failure mode here.
Also, what if the data corruption error happens during the write to the actual database file (i.e. at WAL checkpointing time)? That's still 50% of all your writes, and there's no checksum there!
I do see your point of wanting an option to refuse to delete the wal so a developer can investigate the wal and manually recover... But the the typical user probably wants the database to come back up with a consistent, valid state if power is lost. They do not want have the database refuse to operate because it found uncommitted transactions in a scratchpad file...
As a SQL-first developer, I don't pick apart write-ahead logs trying to save a few bytes from the great hard drive in the sky, I just want the database to give me the current state of my data and never be in an invalid state.
Yes, that is a very valid choice. Hence, I want databases to give me an option, so that you can choose to ignore the checksum errors and I can choose to stop the app and try to recover.
If you attempt to do the former in a system that by design uses checksums only for the latter, you'll actually introduce corrupted data from some non-corrupted WAL files.
1. How does the database know that.
2. In your example Alice gets the money from nowhere. What if another user had sent the money to Alice and that frame get corrupted. Then you just created 10,000,000 from nowhere.
At the very least, rolling back to a good point gives you an exact moment of time where transactions can be applied from. Your example is very contrived and in a database where several transactions can be happening, doing a partial recovery will destroy the consistency of the database.
I've written more about this here: https://news.ycombinator.com/item?id=44673991
Yes! But I am happy to accept that overhead with the corruption detection.
As I see it, either you have a lower layer you can trust, and then this would just be extra overhead, or you don't, in which case you'll also need error correction (not just detection!) for the database file itself.
The checksums are not going to fail unless there was disk corruption or a partial write.
In the former, thank your lucky stars it was in the WAL file and you just lose some data but have a functioning database still.
In the latter, you didn't fsync, so it couldn't have been that important. If you care about not losing data, you need to fsync on every transaction commit. If you don't care enough to do that, why do you care about checksums, it's missing the point.