(Importantly, no such restriction exists on population: within the U.S. at least, nobody has the right to prevent their neighbors or fellow citizens from having kids, while they do have the right to prevent their neighbors from building more housing units. There's an inevitable mismatch between reality here...)
So long as this idea is a a fundamental plank of YIMBY-ism, all of the pretty claims about how high-density living doesn't have to be a dystopian hellscape are literally worthless, because every one of them relies totally on things outside the residents' control that the YIMBYs will be first in line to try and destroy.
Yet this is not the case in California.
https://twitter.com/eean/status/1228182747149025280
https://twitter.com/neversassylaura/status/96939949395841843...
https://reason.com/2019/01/16/san-francisco-property-owner-f...
I could go on for days.
If you look at the photo of the lot you can see why: Beautiful rolling green hills. They didn't own the lot so they didn't have any right to stop the development. Nevertheless that's human nature.
That's why I chose the school bus example. You're supposed to get the part of the seat that your butt sits on, not the whole seat. But from childhood to adulthood, people just never change.