Were did that came from? In a small town, you can get by fine with walking, bikes and so on -- like tons of Europeans do in tons of small (and even larger) towns, including towns where cars are totally forbidden and others were (because the town was built in medieval times and its buildings still stand just fine) they are impractical in large areas of a town.
And besides costly transit infrastructure like subways (which you probably have in mind), buses and trams are totally find for small towns as well.
In (most of) Europe you don't have to be a heroin addict or homeless person to ride the bus. Nobody has a problem riding the bus.
>When a small town grows into a medium, it makes sense to add car infrastructure rather than rebuild everything.
Again, what? Who said there's any need to "rebuild everything"? You can keep the "historic center" (as it's usually referred to), that is, the original small part of town, and build around it, all the while improving and extending transit infrastructure.
No reason to extend the city around car infrastructure -- like the horrible sprawls one sees in Texas for example.
In fact the US has a nice example of a "city" (speaking of the main area of Manhattan up and slightly beyond the Central Park) that has great density, and if there was the will, could be equipped with a great modern subway (not the way NYC subway is now), and lots of quality buses (modern, air-con, well maintained, not meant for the piss poor), a trolley line, and so on.
The area could be expanded from ~ 1 million people, to say 5 times the size, and with all those plus Uber/Taxi options, bikes and electric scooters, nobody would even care for driving their car there.