Welcome to society. I'm not sure where you live but in every country I know of there are plenty of things you can't do with your car and/or your free time.
The only one harming them is the tax collector. Taxes are the harm here; Uber is providing a means for some to avoid being harmed. It's just too bad that they can't help everyone else the same way.