The long tail of MD5
tedunangst.com
tedunangst.com
* This is not an endorsement of MD5 for such use cases.
If the application was just looking for a longer CRC/Adler32/etc. and not depending on the hash to be strong against attack, then MD5 wasn't an appropriate choice in the first place, as its much slower than need be. But the security problems are irrelevant.
Maybe that's the sweet spot for your project, but I think most non-security uses of MD5 are because it already exists in most languages and is pretty universal
The worst and most persistent security problems emerge when someone defends half-assery as acceptable because, take your pick, it's a quick hack / it's a personal project / it's a small project / it's not used for critical infrastructure.
Until eventually is one or more of the above are violated.
The primary advantage of MD5 is that the hashes are (slightly) shorter than those of other checksum methods. This makes it slightly more convenient to manually compare or transmit hashes.
My problem is that I happen to have used md5sum so often and for so long that it's wired into my own wetware and muscle-memory, and I'm not sure which of the alternative SHA sums I should use, and which of those are widely available. I honestly don't know the answer to that off the top of my head, and "what to use instead of md5sum" as a DDG or Google search doesn't turn up a clearly useful answer. "sha1 sha256 sha384 sha512 which to use" does better.
And locally I've got utilities for SHA 1, 224, 256, and 512 installed, that I can tell.
Looks as if SHA-2 and SHA-3, which include keylengths of 224-512, are considered secure:
http://en.wikipedia.org/wiki/Secure_Hash_Algorithm
Then there are some openssl utilities. But let's not go there.
* note that even if it isn't immediately obvious, in most situations you really don't want collisions to be an easy thing to cause. For example, many algorithms and structures involving hashes (even a simple in-memory hash map) suffer massively degraded performance when there's a large number of collisions. If your user can craft an input that causes operations to take orders of magnitude longer, that's a denial of service attack.
There is certainly risk of hash collisions in files, but most attacks require you to generate two different files with the same hash? Do people think a pre-image attack is feasible? If feasible, at what scale? What type of actor can give me a file that has different content, but the same hash as winword.exe. If they can, what is the likelihood that said file will also be executable or contain a linkable library
*similarly, and hypocritically, not an endorsement of md5
This is why I get fairly upset when people design something new I n 2014 that uses md5. Yes even if your application does not use md5 for anything security related, the mere fact that you use a bad, slow algorithm should be considered wrong. And adding new dependencies will make it harder for us to migrate away and extend that tail even further.
If you need a good hash function that is fast, use SipHas. If you need a really secure one, use SHA-512, SHA-3, BLAKE or just SHA-256.
However, as a hash function (e.g., with a fixed key), it is entirely too weak: 2^32 work for a collision, and 2^64 work for arbitrary (second-)preimages.
A PRF is secure as long as attackers don't get to know exactly how it works, which, with SipHash, they don't if they don't know the key used for it.
A cryptographic hash function is secure --- it's Hard to find two inputs that produce the same output, it's (even) Hard(er) to find a specific input that generates a given output --- even when it's not keyed.
SipHash is a cryptographic building block that is performant in part because it takes advantage of relaxing the constraint of being a good cryptographic hash function, and only tries to be a good cryptographic PRF.
Don't forget VBA digital signatures BTW, for which MD5 is the only choice. I wonder how feasible a collision attack would be.
Also the author is wrong that there isn't a proposed replacement -- it's the "Digest:" header (RFC 3230) although I don't think anybody uses that either.
Provide public-domain easy-to-compile/use versions for all languages, and furthermore, get their google page ranks high.
Do not underestimate laziness. If Joe Random can find a suitable MD5 algorithm in 10 seconds but it takes 30 seconds to find a suitable SHA algorithm, guess which one gets used?
The problem is exactly the one elucidated in the blog post above: the long tail of baked in brokenness. These systems were never designed to be extensible, they are cooked into code that hasn't been serviced in years, perhaps even decades in some cases. They're bolted into specifications in ways that either obsolete the technology completely, or make it so incredibly complicated to update the technology that doing so outweighs the apparent cost. And that's without considering problems like deployment and phase-out.
These types of problems make it very likely we will be stuck with the stupidity of DES and MD5 in strange places until it becomes a fire drill and then all the sudden people will be baking in SHA-1/SHA-2 or BLAKE2 and we'll be going through these very same motions again in 5-15 years, wondering why we didn't learn from the mistakes we made last time.
Huh?
There are serious flaws with MD5, and I would never recommend its use today. But there is no known attack that can generate a malicious binary with an arbitrary hash. The attacker needs to control both files to generate collisions, which is a huge limitation.
Maybe those attacks will exist some day. But I wouldn't be setting fire to all your legacy software without first understanding exactly how they use MD5, and if that is vulnerable to any known attack.
At that point even a sloppy brute-force approach would have a high probability of a collision on at least one key of a sufficiently large key-set. Against a specific target it may or may not be viable.
And the point is fair, there's not really anything today where MD5 is an obvious choice.
The trouble is, they have to pass the name of the form field to the server on the other side, which is some variation of "password". So anyone doing a simple string search will eventually find the perl scripts, and have access to my passwords (supposing they have root).
If I rot13 the form field name, and then have the name be rot13('cnffjbeq') in the script, this makes is unsearchable.
A master key. Which would have to be included in the perl script... so that when they find the script, they have all my passwords instead of just a few. You've solved it. Why didn't I think of that?
> You can make the script prompt you for the master key when you start it up,
Why the fuck would I want to sit around being a meat robot inputting 9 passwords at the beginning of each month just so I can have a copy of my bank statements and electric bill?
The whole point was that this was in a crontab.
If you want this done in a fashion that's secure, deploy it on a small device like a Raspberry Pi and firewall that thing so aggressively you can barely get in. It's hard to hack what you can't connect to.
What's meant by that is if you can find a source string that hashes to the same thing. Using ROT13 is pointless since it's trivial to generate those.
MD5 is only slightly harder.