That's a fallacious argument. There is no activity that could not be made more safe by wearing a helmet. Yet we don't because the arguments like yours are simple and wrong.
But I agree with your general point, if the marginal safety gain during cycling was enough to make them mandatory, few activities would remain without mandatory protective headwear (my last bike-related head injury happened while walking through the kitchen during maintenance).