I don't think that's it at all - the counterargument to "sports gambling is bad" is "we live in a society that places an incredibly high value on personal freedom, so we should err on the side of allowing people to do what they want, even if it results in some amount of harm, because the inherent value of allowing freedom of choice is high."
That's not to say we can't ever ban things - obviously there is a degree of harm that is high enough that we should sacrifice particular freedoms (e.g. you can't have a nuke). And FWIW I think this one certainly sits close to the line of being worth banning, but the point is that it's not a clear case just because the direct benefits to society of people gambling (which are minimal) outweigh the harms (which are substantial).
I think the most helpful way to consider the tradeoff is by using examples of what we allow in society now. Alcohol's a good one - without question a hugely harmful substance overall, but we allow it. Is gambling more or less harmful than alcohol?