People don't say no to increased density because they are worried about increased overall housing supply affecting their house value.
They say no mostly because they like the place they are how it is, and they don't see any reason for it to change because the upside doesn't affect them.
And the rest of the time they are saying no due to a direct downside like having a new large building near them that wasn't there before.
I'm not saying that this attitude should be allowed, but let's be honest about the cause of it.