My suspicion is that the middle of the Venn diagram between "can afford a $500K aircraft" and "doesn't fly GA because they're not going to stay proficient enough to be safe" is vanishingly small. The doctors who fly their Cirrus once a year think they're Maverick so they don't care about the safety features. The guys who fly every chance they get aren't the ones stall spinning on base to final.
But I think the main reason why people don't do that who can't afford a multi-million dollar private jet is that if you take a compass circle and draw it around your location with the range of your average GA aircraft anywhere that's not on the east or west coast you get 4 or 5 cities that you aren't likely to travel to anyways. America is just too big a place for these types of aircraft.
Plus, while there may be all these airports that you are mentioning, can you really count on getting fuel to take you back at these minor airports or the FBOs to service outside a much shorter list of airports? That cuts the range away even more.