Well, we don't actually know this do we? We know that it's very hard, and that it's never been done before, but so was running a sub-4 mile for most of human history.
If we knew the mechanism through which people learn perfect pitch and could prove that it vanishes with age, then you'd have an argument. But we're not there yet.
I have a pet parrot who seems to always sing in the same key. Now I feel bad for whistling off-key to him all these years!
Assuming that perfect pitch is an immutable skill after a certain point, I wonder whether the age of acquisition still matters. Do people who learn the skill earlier have more fine-grained perception?
From personal experience, long ago I spent some time trying to acquire perfect pitch using a synesthetic method from somewhere, and eventually convinced myself that I was making some progress, but that the effort required to take it to the point of being at all useful wasn't going to be worth my practice time.