Hashlookup.circl.lu API – hash lookup of known files | Hacker News Reader

Hashlookup.circl.lu API – hash lookup of known files

gist.github.com

31 pointsadulau14 comments

isn't that just VirusTotal only without the convenience of having the files virus checked?

lol, if-then checks for maliciousness

Which files will be in the database?

Why use hashes which aren't collision resistant for the lookup, instead of say SHA-256?

A simple practical reason is that the volume of known files that has MD5 or SHA1 hashes is huge compared to SHA256. It's preferable to have something collision-resistant, but also if your goal is to simply be able to identify files, and MD5 or SHA1 checksum can be fine. That will change over time, but realistically tons of AV scanners and tools like this need to support MD5 or SHA1.

Of course, if you're generating new files that you know about or checking existing known good files, one would include the SHA256 and SHA512 with it.

Isn’t that something the `Accept` header would be perfect for?

Sorry, I don't follow what you mean. The accept header can specify mimetypes and negotiate what the response can look like. So maybe you could stretch it to say "I'm only accepting SHA256" somehow. That might improve the API, but then you're indicating to the API to return 404's more often since it won't be able to find files with that hash.

Maybe there are other things you could do instead though? I'm not sure.

Well, I think you could definitely stretch it that way, because the Accept header is absolutely intended for situations like this: It lets you specify precisely which content formats you're willing to accept, and in which particular order.

  Accept: text/html, application/xhtml+xml, application/xml;q=0.9, image/webp, */*;q=0.8

No reason why that couldn't read like this instead:

  Accept: application/vnd.hash+sha256, application/vnd.hash+sha1;q=0.9, application/vnd.hash+md5;q=0.8

That would indicate to the server that you'd like to receive (preferably) SHA256, with a fallback to SHA1, and another fallback to MD5, if none of the other two is available. If the server isn't able to provide any of those, it returns a "406 Not Acceptable" response. HTTP covers all of this out of the box, people just tend to overlook the more advanced stuff.

SHA256 is the current standard for this use case. Please use and encourage the use of that.

Fair enough! I over-spoke in my previous comment (I don't seem to able to edit it anymore). Thanks for pointing that out.

Using SHA256 or greater is what should be used for generating new hashes, especially if you can update old entries and files with SHA256's. Many databases still have way more MD5/SHA1 hashes which won't be updated to include the SHA256, which is why you'll see them be used often.

> the API is also described as an OpenAPI.

The response formats are not described so not very useful.

Curious if IPFS file hash is compatible with this list?

Most probably no. Instead of calculating the hash of whole file, IPFS chunks a file, calculates the hash of each chunk and turns them into a DAG of hashes (Merkle DAG). Also because there are multiple ways to construct the Merkle DAG, it's possible that same file can have different hashes.

Thanks appreciate the info!

Reply on news.ycombinator.com