Because the answer is "yes" for colorblindness. It's just a wildly overstated comparison. I mean, even tone-deaf people don't experience the kind of difficulties that people with colorblindness experience. You could just say that I could never entirely understand the benefits of having perfect pitch since I don't have it, and maybe that would be true, but I have literally never experienced a professional musical situation where I felt or thought, "Oh man, if only I had perfect pitch - I would have succeeded where I just failed." And this includes directing gigs, accompanying gigs, performance gigs, etc.
I'm occasionally wistful I don't have synesthesia just because that sounds actually expansive, but I don't pine for perfect pitch at all.