Man, reading this just makes me wonder again why -c (checksum) isn't the default in rsync by now, especially with SSDs everywhere. Is it really just about that tiny bit of speed?
Degradation won't get worse except when a file changed without metadata having been modified, but then that's exactly what you want
And the speed difference is only tiny for tiny files.