Could someone please explain the background to someone not from the US? Why is competition so scarce in this field in the US? Is it just the cost of infrastructure over large distances, or is something else at play here?
ISPs, like mail delivery or electricity, are classical examples of "natural monopolies" where the fixed price of installing infrastructure is high, while the marginal cost of each additional customer is low.
Such environments are tricky to manage: You want some market forces, because otherwise there's one lazy government-owned provider with bad service at high prices because they have no pressure to be better. At the other end of the spectrum, you have a private company with bad services at high prices that makes a ton of profit because competitors cannot afford the initial investment.
To balance these competing forces is tricky, and some countries manage better than others.
In short: Create a fair field for competition and ideas will come naturally.
More likely, the last mile provider does termination, so line speed isn't negotiable by provider. Instead you're competing based on peering and transit connections, oversubscription ratios, availability of static ips, other policies, customer service, etc.
The form of "deregulation" we get from Ajit Pai's FCC selectively advantages the incumbent oligopoly of telcos, leaving in place the most grotesque market-distorting regulations and removing controls on vertical integration which allow for the expansion of telco power into other industries.
Market failure can take myriad forms. The ISP situation in the US is a particularly maddening variant.
It's completely possible to corrupt an agency and for the FCC to behave the way you described through political interference but that is not their intended purpose.
Do you have any recommendations?
Leaving politicians aside, perhaps this is too harsh a judgment of FCC personnel? To an extent, yes, because some of them are janitors or IT helpdesk folks. However, most people at FCC have at least an inkling of how things really work. At the minimum, they know where the third rails are. They know that asking "why can't more of the spectrum be unlicensed like the wifi band?" is a career-limiting speculation. They know that every Chairman Wheeler will be soon followed by a Chairman Pai.
In the US there are over 50 metro areas with more than 1 million inhabitants, over 100 with 500k. Granted that they will be more sparsely populated than european metro areas but I'm just not too sure that the large distances are an issue here at all.