Do we ban people from driving if they refuse to wear seatbelts?
A better example would be driving drunk, given that you are at an increased risk of injuring others.
Remember that a seatbelt also prevents your body from being a projectile that would be launched against other people in an accident, you're not just a danger to yourself when you don't wear one.