I'm not quite sure the car argument holds water either. Cars are very heavily regulated for safety concerns and they routinely ship with airbags and are designed to collapse in ways that protect the driver. Most states also mandate wearing a seatbelt. So there's precedent for mandating that operators of vehicles protect themselves. I don't see a helmet as any different from a seat belt.
Furthermore, motorcycles require helmets, which is a more appropriate analogy than a car.
The difference between walking down the street, jogging, being in the bath, gardening, etc, versus riding a bike is a matter of velocity. If I'm jogging and I trip and fall, I'm not likely to land head first, nor am I likely to do much more than fracture a bone or bruise myself. While riding a bike, any sort of collision is likely to happen very quickly and with a lot of velocity, and as basic physics tells us, mass + acceleration = force. The faster you're going, the more force is applied. With a crash on a bicycle, you're much more likely to break bones (as opposed to fracture) and suffer concussions -- or worse. A helmet is just plain common sense at this point.
Better safe than sorry, imo. I'd rather risk getting stung by a wasp than get a concussion or possibly brain damage.
But hey, if you're willing to risk it, then go for it. Not my problem.
Edit: I'm in favor of wearing helmets; I'm not in favor of mandating it.