Because people inherently misunderstand the benefit of HS2, and how could they not if it's constantly being misrepresented by our media and politicians.
UK has one of the highest proportion of freight transported by road in Europe. That is fundamentally because our rail infrastructure is overloaded and unable to take any more freight. All non-perishable stuff that in other countries just goes on rail, in the UK is moved by trucks on our roads. Which as you can imagine, is causing tens of billions of pounds worth of damage to our roads, which we - taxpayers - pay for. All of these locals that live miles away from the train station are already affected by the lack of rail infrastructure - because every time they drive somewhere they have to contend with massive potholes and insane amount of heavy cargo traffic anywhere they go. If HS2 is ever finished, it will reduce congestion and our roads and reduce the wear and tear which again, is costing us billions in upkeep every year.
But according to our media, it's all about saving london commuters 2 minutes on a train from Birmingham, so every Dick and Harry is against it, because like you said - they live miles from the nearest station, why would they care?