Exactly, if you're hand soldering through hole components, it's not a big deal. However, blue/black/white solder mask pigments were noticeably (>2mil) smaller in tests than the standard green for a high volume manufacturer. Most of the time a partially mangled pad reflow wouldn't be a problem, but when you're automatically stuffing the boards, you won't know and could have latent failures (temperature/time) after production test. This is especially the case for aQFNs and BGA ICs.
Fiducials are automatically placed in top copper, but we also noticed larger misalignments of paste and smaller 01005 passive components (pre-reflow) on non-standard colors. Again, unlikely to cause large yield problems unless you've got a lot of components, but why risk it.
We did still use red for test PCBs so that they wouldn't be shipped to customers. If we were making gaming video cards colors would have been more of a priority.