I used to think this, too, but Aaronson changed my mind with the observation in [0] that, if P != NP is independent, then it would necessarily imply a lot of what we would demand from a constructive P = NP proof. In particular, we could get some pretty hilarious improvements on those exponents in NP-complete algorithms, to the point where they become tractable for day-to-day computing.
There is a sense in which any complexity class which is self-low is "physical", or analogous to machines which we might build by hand with modular designs. A self-low class doesn't gain any additional power when relativized with itself as an oracle, and this directly corresponds to building a physical machine out of smaller machine modules.
We know that P is self-low. It seems that the universe insists that either we are able to build machines which tractably solve NP-complete problems, or not, but there is no in-between position.