IME, on basically all filesystems, just walking a directory tree of lots of files is expensive. Half a million files on modern systems should not be a terribly huge issue but once you get into the millions, just figuring out how to back them all up correctly and in a reasonable time frame starts to become a major admin headache.

Since git is essentially a filesystem with extensive version control features, it doesn't surprise me that it would have problems handing large amounts of files.