Since he asks for feedback at the end...
Using sha256 just to compute changes is probably overkill. Using md5 instead is almost certainly adequate and will be a good deal faster.
Using sha256 just to compute changes is probably overkill. Using md5 instead is almost certainly adequate and will be a good deal faster.
You could always try it and see :)
Example test: openssl speed md5 sha256
As for why it was chosen, I think it's because there are known examples of MD5 hash collisions (though the likelihood of it on a filesystem is remote) and likely SHA-1 was skipped because it's considered 'likely' a collision could be created (though so far only with weakened versions of SHA-1).
But - all this to say: The chances of having two files with the same MD5 hash that are identical in size is vanishingly small. As such, for the known MD5 collision mechanisms, the different file size would be enough evidence something has changed.
... Why he didn't include file size in the metadata check, I can't tell you. Timestamps can be faked - but generating a hash collision with a file of equal size is a Hard problem.
I haven't thought about the filesize/hash to reduce collision, but I chose to stick with the last-modified time in the article, because it can takes hours computing hashes for a big directory tree.
Tools like rsync relies on last-modified time by default, and since I want to use this to track my own files, I won't fake it, so I think it's not a big deal?
So I guess the point is that also including the file size will be one more (fast) data point to help ensure 'accurate' change tracking, without adding the overhead of computing content hashes.
Thanks!
>> You could always try it and see :)
>> Example test: openssl speed md5 sha256
When I was looking for a fast hash easily called from Python, I settled on adler32 (as the fastest) after some trial-and-error on my files. I don't now recall all the utilities/functions that I tried, but they certainly included md5, sha1, and crc32. I only needed to test for accidental corruption, and only computed the hashes if meta data matched.