Comment by TacticalCoder | Hacker News Reader

Comment by TacticalCoder | Hacker News Reader

Parent Full thread

> deduplication at the file level: every file is now stored under its BLAKE3 hash

Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).

If I've got a file named:

    DSC98731-b3-7b39197a22.JPG

then:

    - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)

    - if any other file has the same Blake3 7b39197a22 hash, it's a duplicate

    - if that 7b39197a22 checksum is in my database, "things can happen".

For example my DB can say "any file with a Blake3 hash of 7b39197a22 can always be deleted" or "any file with a Blake3 hash of 887463c09e, if it's got a generic filename like "dscXXXXX" can always be renamed to "20260722jackJohnAtTheBeach-b3-778463c09e.jpg" (or whatever suits you).

It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.

For those wondering like me: Blake3 generates hashes of at least 224 bits, not, as a literal reading of that comment indicates, 40 bits (which would be bad for file deduplication, giving you a 50% hash collision after around a million files)

Yup sorry if I mislead people. I usually use 10 hexdigits (40 bits) but no matter how many there are, it's the x bits of the beginning of the checksum that are verified against the hash (in the examples I gave 40 bits).

I wasn't very clear.

So at 3 million different files you have a 98.3% chance of a hash collision. Wouldnt that cause problems in real datasets?

Another cool thing about BLAKE3 is that it is a merkle tree. Not only it allows for good parallelism, but it also has a lot of cool property for data transfer. For example, you can check for partial validity, which allow for streaming error-detection and resend during the transfer. You only need to have the data and checksum in a way that you can start to reconstruct one or more subtree.

> I use it for my own "deduplication / integrity / berzerker" utility

Do you have it in a public repo you could share?

No sadly I don't have any public repo: it's mostly really a collection of shell scripts and then the database one (for the berzerker/renamer) is in Clojure and I run it from my always-on REPL.

But I know others did similar thing so maybe there are public repos out there. But in any case: it should now be the kind of thing relatively easy to vibe-code if it's for your own use.

It's annoying that most file formats don't checksum their own content.

Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.

PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.

This is one great benefit of using .zip files as file formats, you get this for free.

I think it's reasonable for a DB like SQLite to delegate that to the filesystem. There is an overhead for doing it on the DB level, and since SQLite is just a file on the filesystem which, presumably, is serving many other files as well, why would you trust anything else on the filesystem if you don't trust SQLite? Like, your PHP script (or nginx server executable, or whatever) that is calling SQLite, that's not going to be check-summed either. Either you trust your filesystem or you don't, and if you don't, checksum and error correct on the filesystem level.

Though fair enough, it could offer it as an opt-in thing.

Because SQLite is a program, that will refuse to start or crash, and which you can trivially replace, if corrupted.

Whereas your sqlite data are your data, and if they're corrupted they can be lost forever or propagate the issue to backups.

Hopefully it crashes if corrupt... But it could instead jump straight to some code designed to clean up files and delete your database

Checksums use CPU cycles. SQLite will do checksums with an extension (https://sqlite.org/cksumvfs.html) but that is off by default since an overwhelming majority of developers are more interested in day-to-day performance than detecting (very rare) storage malfunctions.

Thank you for creating it!

I wonder if something changed, with todays SSDs, memory bottlenecked CPUs, and hardware accelerated CRCs.

FWIW, RAM bit-flips are much more common today, on consumer devices where SQLite is used a lot, since today's memory operates at the limit (see RowHammer).

If you use a filesystem that does checksums, like ZFS, then you'd end up doing it twice for no reason. It makes more sense for archival formats like FLAC, though.

> SQLite is a Recommended Storage Format for datasets according to the US Library of Congress. Recommended storage formats are formats which, in the opinion of the preservationists at the Library of Congress, maximizes the chance of survival and continued accessibility of digital content.

https://www.sqlite.org/locrsf.html

It also lists XML, JSON and CSV, so they obviously expect you to provide an external checksum. The source even mentions checksums: https://www.loc.gov/preservation/resources/rfs/data.html

> It's annoying that most file formats don't checksum their own content.

I agree.

> Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.

I mostly run ext4 (desktop, laptops, etc.) but for my main server at home, it's a ZFS (mirrored) tank on an old server with ECC RAM.

And backups. So much backups.

Why BLAKE3? I hadn't thought too deeply about it but at some point I settled on https://github.com/Cyan4973/xxHash for similar things.

It's probably the fastest strong* cryptographic hash function to date.

Reply on news.ycombinator.com