Hacker News Reader

Comment by dylan604 | Hacker News Reader

who is attacking you in this situation though? you're scanning the files on your local system and storing their hashes. you then look for duplicate hashes, and compare the files that created them. if the files are truly duplicates, you can now decide what to do about that. if they are not truly the same, then you claim to have found another case of collisions, write your blog post/twitthread and move on, but does that constitute being attacked?

sometimes, i really feel like people in crypto just can't detach themselves enough to see that just because they have a hammer, not everything in the world is a nail.

View on HN

Why would you pick a function that is known to have issues when there are other functions that do the same thing but don't have known issues?

Your comparison is flawed. It's more like if you have a nail and next to it a workbench with two hammers - a good hammer and a not as good hammer. This isn't a hard choice. But for reasons that are unclear to me, people in this thread are insisting on picking the less good hammer and rationalizing why for this specific nail it isn't all that much worse. Just pick the better hammer!

thanks for the reply that did not answer the question asked.

Your question is irrelevant. If you don't care about security, SHA1 is a bad choice because there are faster hash functions out there. If you do care about security, SHA1 is a bad choice because it has known flaws and there exist other algorithms that don't. The only valid reason to use SHA1 is if there is a historical requirement to use it that you can't reasonably change.

Any analysis about how hard it is for an attacker to get a file on your local file system via a cloned got repo, cached file, email attachment, image download, shared drive, etc is just a distraction.

You would be right, except that there are no faster hash functions (on all modern Intel, AMD and ARM CPUs, which have hardware SHA-1).

BLAKE 3 is faster only in wall clock time, on an otherwise idle computer, because it fully uses all CPU cores, but it does not do less work.

BLAKE 3 is preferable only when the computer does nothing else but hashing.

Uh no.

On a modern intel CPU, one core of SHA1 does about 500MB/s worth of hashing. Blake3 on the same core is 1.5GB/s or faster.

On the ThinkPad I'm typing this on, the single-threaded BLAKE3 benchmarks hit 8.8 GB/s :)

Because people already have two decades of SHA-1 hashes in their database and a rewrite + rescan is completely pointless? Hell, I have such a system using md5. So you produced a hash collision, cool, now fool my follow-on byte-by-byte comparison.

Edit: Before anyone lecture me on SHA-1 being slow, yes, I use BLAKE2 for new projects.

because it's shorter, dealing with long hashes is annoying (I use md5, not sha1)

You could just discard half the sha256 hash. Using the first 16 bytes of sha256 is a lot more secure than using just md5, in which case you might as well just use crc32.

Its worship of the new. Nothing worth anything existed last week, so reimplementing an entire infrastructure is zero cost.

I dont think this is what most people think of when they say deduplication. There are quite a few systems which will just scan for duplicates and then automatically delete one of the duplicates. In such a system sha1 would be inappropriate.

If you are just using sha1 as a heuristic you dont fully trust, i suppose sha1 is fine. It seems a bit of an odd choice though as something like MurmurHash would be much faster for such a use case.

SHA-1 cannot be trusted only when there is a possibility that both files whose hashes are compared have been created by an attacker.

While such a scenario may be plausible for a public file repository, so SHA-1 is a bad choice for a version control system like GIT, there are a lot of applications where this is impossible, so it is fine to use SHA-1.

I'm not sure what scenarios there are where you have a possibility of the attacker creating 1 file but not both. Especially because the attacker doesn't need to fully control both files but could control only a prefix of one of them and still do the attack.

I also think working out all the possibilities is really hard, and using sha256 is really easy.

Most people. I haven't been part of that group, for like, ever maybe?

If we're a group of devs with a not insignificant percentage of those devs being frontend/UI/UX types, then having the same image in multiple sizes, formats, etc is going to be pretty common. Looking for multiples of the exact file is only going to reduce so much. Knowing you have a library of images with a source and then all of the derivatives is going to get you a lot less files as long as you know you have the source, then running image based sameness is much more beneficial. Sure, this is niche territory, but yeah, and, so?

Maybe there's someone new(-ish) that hasn't really had to deal with cleaning up thousands of images to this extent. One would hope the same image in its various forms within a dev's env would be similarly named, but that's not guaranteed. If we could depend on filenames, we wouldn't need hashing, right?

In some cases deduplication happens at the file system layer transparently without you even realizing it. E.g. there are tools like https://github.com/lakshmipathi/dduper

I agree that image editing workflows are a different use case more suited to perceptual hashes than cryptographic hashes.

If you are going to follow up with a full comparison you can use something much cheaper then sha-1

Reply on news.ycombinator.com