Constraining other peoples freedom, or otherwise possible ways to harm other people in any form is not included into the freedom people push for.
Your argument is kinda like saying if you prevent someone from hitting someone else your are against fredome because that person should be free to hit anyone they want to hit. Just that's a more subtle topic then hitting someone.