Have you ever wondered why most home internet speeds top out at 35Mbps? It goes back to legacy spectrum stuff, they never needed upload to get cable to you, so the reserved spectrum was small, on docsis 3.x it tops out at 35Mbps.
The internet backend isn’t some wonder of engineering, it’s tubes held together with duct tape and bubble gum. A lot of old ways of doing things.
There is a very real practical limit to “just building out capacity”. Because of exisitint equipment, legacy standards, etc.
Running fiber to everyone directly, wouldn’t fix things. Beyond specs and standards, it would become a space, power, and heat limitation at these isps, and this is all totally ignoring cost.