There's a few big reasons, at least in tech.
One is that your investors may want you to start your business somewhere close by, especially if they want to be involved in your business.
Another is that you need to start your business somewhere where you can acquire employees - if there are very few software engineers in an area already, you now need to convince people to move to your area for work. That's kind of a bad value proposition for employees because they need to relocate their lives to move somewhere that has only a single employer they could realistically work at. Usually the people running these businesses also are motivated by doing this to save money, so you might also be making about half of what you would in a big city.
Further, being in a big city has a lot of benefits from infrastructure and network effects. It is easier to acquire some of your first customers if they are close by. You have international airports which allow you to travel the world more easily than somewhere with an airport that only feeds into a hub. Your employees will appreciate having the ability to work at other places should their employer fail/they get fired/etc. so it's easier to convince people to move there.
What should be done, but which almost never is done, is that employers in small cities should start paying truly competitively with those in big cities. If your target employee could make 300k in NYC or the Bay Area, pay them that much in the cheaper city. Make the cost of living difference actually become a real perk, rather than a way to be miserly with how you pay employees.