The number of possible deaths is _far_ higher. The chance is almost certainly lower. I would not be at all surprised if on the balance it's way worse than 5 times more dangerous, in terms of expected lives lost (in statistical terms).
What is the statistical likelihood that any person of a specific religious group will commit an act of terrorism? Should we find that number and then jail a random swath of their population?
If I fire a gun into a building blindly, the chance I kill anyone is low. Should that be legal?
If we agree it should be a crime and just disagree on severity, that seems fair enough and (at least to me) not worth any further discussion than I already engaged in on it.