> Curious to know if there are downsides to this (performance is an obvious one) or whether you're weakening the stronger hash by hashing a weaker one?

You're weakening the stronger hash by hashing the weaker one. The two main problems of MD5 are entropy (128-bit maximum) and collisions (because of lack of entropy).

By always hashing to MD5, you are losing the entropy of the stronger hashing algorithm.