Further, there are 7.53 billion people on earth. If you conclude (generously) that, living in New York City, you have the freedom to associate and deeply rely on anyone in the tri-state area, you have access to about 20 million.

That's 0.2% of the total addressable people, and living in NYC puts you near one of the largest population centers in the world. It would take large amounts of capital for you to switch to another fraction of a percentage of the total population, and in doing so you'd be abandoning much of that original 0.2%.

You are far more constrained by your circumstances, by our social organization, and by physics than it might appear because of how easy it is to talk casually with others. Some of this we can overcome with changes to our society and development of technology, but some we probably can't. Your freedom to change your conditions falls off in power really quickly. Or, put another way, addressing the full set of choices gets expensive very quickly.

EDIT: I liked doing this math and, having lived in Tokyo as well as NYC, you could increase that 0.2% to about 0.5% by moving to the center of Shibuya; about 40m people live in and around the administrative unit. It's the densest area of its kind on the planet. That said, measuring Tokyo is a bit of a fraught problem because there are multiple ways to outline the city itself.