Yes, cycling is a sport. Should we also flatten hills as they have the inconvenience of requiring to push harder on pedals?
If the physical aspect of cycling (involving stopping for taxis, pregnant women, children, or distracted pedestrian plebs) is too hard, solutions exist: public transports, walking, or electric bikes (whose riders seem equally annoyed to slow down, for some reason).
The problem with "building infrastructure" is that plebs' money is not infinite, so public works to please the high lords of the pedal may not be possible.
Space is also not infinite in cities, so you can't change the infra without sacrificing other users, be they pedestrians, delivery vehicles, or car users.
Last, cycling is mostly for a specific type of people, who are alone, fit and with a small cargo. It excludes older and younger ones, disabled plebs, and families. Those people are better served with quality public transport - which could be improved with the money used to make costly bicycle road arrangements. Public transport is also always convenient, not just when it's sunny. I live in a city with temperatures under 0 celsius for 4 months of the year, including some days at -20. I'm glad I can use my car in winter to take my children to school. Apparently, other cyclists seem to think the same since no one uses a bike in winter (the new urban equestrian class seems a bit shy when it's cold?).
Helsinki is a city I like, because while there are mostly non-invasive bike lanes on large arteries, it's easy to go around by car and by walk. The secret is that the public transport system is top-notch, so a pregnant woman can use it to go to the maternity ward - something that you can't do by cycling (but who is stupid enough to have kids nowadays in city centers, right?).