Comment by supriyo-biswas | Hacker News Reader

Comment by supriyo-biswas | Hacker News Reader

Parent Full thread

I wouldn't have jumped to a conspiracy angle immediately, but there are some signs which are difficult to overlook:

- Said person was apparently employed due to his good understanding of databases and distributed systems concepts (there's a HN thread about how he found an issue in the paper describing an algorithm); yet makes fundamental mistakes in understanding what the WAL does and how it's possible not to "partly" apply a WAL.

- Said person expects a SQL database to expose WAL level errors to the user breaking transactional semantics (if you want that level of control, consider simpler file-based key-value stores that expose such semantics?)

- Said person maligns SQLite as being impossible to contribute; whereas the actual project only mentions that they may rewrite the proposed patch to avoid copyright implications.

- Said person again maligns SQLite as "limping along" in the face of disk errors (while making the opposite claim a few paragraphs ago); while ignoring that the checksum VFS exists when on-disk data corruption is a concern.

I think it's kind of possible to partially apply the WAL manually. Imagine your frames are:

1) Insert new subscription for "foobar @ 123 Fake St." 2) Insert new subscription for "�#�#�xD�{.��t��3Axu:!" 3) Insert new subscription for "barbaz @ 742 Evergreen Terrace"

A human could probably grab two subscriptions out of that data loss incident. I think that's what they're saying. If you're very lucky and want to do a lot of manual work, you could maybe restore some of the data. Obviously both of the "obviously correct" records could just be random bitflips that happen to look right to humans. There's no way of knowing.

And if the “obviously correct” last entry is actually an old entry that just hadn't been overwritten yet? Or if it was only permitted because of something in the corrupted section?

The database should absolutely not be performing guesswork about the meaning of its contents during recovery. If you want mongodb, go use mongodb.

I'm aware that nothing should automatically attempt this recovery, but asking the database to throw an error when it happens does not seem crazy to me. I mean, I get why it doesn't. I have worked on systems with "mount /dev/sda1 || mkfs.ext4 /dev/sda1" because there isn't a UI to recover the filesystem, so nothing is lost by erasing it. But I don't think it makes you bad at databases to want this condition to be optionally fatal. (I can also see why "welp it's corrupted <unlink>" is also annoying. If I'm the author of this software, I would be interested in checking out the corrupted file before deleting it. I can live with the slight hit to my free disk space if it means I can fix a bug!)

Yes, in this particular example you could. But in general the database cannot make assumptions that changes are independent of each other.

I think SQLite assumes that a failing checksum occurs due to a crash during a write which never finished. A corrupt WAL frame before a valid frame can only occur if the underlying storage is corrupt, but it makes no sense for SQLite to start handling that during replay as it has no way to recover. You could maybe argue that it should emit a warning

This could work for a simple key-value store; but SQLite also does referential integrity which means we might just end up with extra entries with no entities on the other side of the table. IMO, best avoided in a transactional database.

of all places, I did not expect to get personal attacks on HN :)

> yet makes fundamental mistakes in understanding what the WAL does and how it's possible not to "partly" apply a WAL.

Please provide citation on where I said that. You can't partly apply WAL always, but there are very valid cases where you can do that to recover. Recovery doesn't have to automatic. It can be done by SQLite, or some recovery tool or with manual intervention.

> - Said person maligns SQLite as being impossible to contribute; whereas the actual project only mentions that they may rewrite the proposed patch to avoid copyright implications.

Please provide citation on where I said that. Someone asked me to send a patch to SQLite, I linked them to the SQLite's page.

> You can't partly apply WAL always, but there are very valid cases where you can do that to recover.

Without mentioning the exact set of cases where recovery is possible and it isn't, going "PSA: SQLite is unreliable!!1one" is highly irresponsible. I think there's quite a bit of criticism going around though, you could add them to your blog article :)

Please also consider the fact that SQLite being a transactional database, it is usually not possible to expose a WAL level error to the user. The correct way to address it is to probably come up with a list of cases where it is possible, and then send in a patch, or at least a proposal, of how to address it.

> Please provide citation on where I said that [SQLite is impossible to contribute].

https://news.ycombinator.com/item?id=44672563

I don't know if you have some personal vendetta against me, because you are citing things I did not say. I did not say SQLite is unreliable. I said SQLite stops at checksum errors found in WAL and stops recovery, which may lead to data loss. Which part of this is incorrect?

On SQLite contribution, I did not say it's "impossible." I said it's not open to contribution. This is the exact phrase from the linked page.

To me it isn't incorrect, but misleading. A checksum error means corruption, corruption in the main database just randomly can cause any damage. Why specially consider checksum error in the WAL?

> of all places, I did not expect to get personal attacks on HN

You must be new to the site.

Reply on news.ycombinator.com