There is another famous quote from John Stuart Mill, "Your liberty to swing your fist ends where my nose begins."
For example, take the Constitutional Right to bear arms. Obviously we have limited individual liberty to bear arms for safety through both regulations and criminal statutes. I think that trade off can be pretty reasonable in some instances (say, no guns at the airport/on a plane) and since I don't mind sacrificing a little liberty for safety I don't think that trade off makes me undeserving of either liberty of safety, does it?
Please note, I am not trying to take a position other than public awareness and conversation about these things is always good; whereas, quotes about not deserving liberty or safety only serve censor an open conversation from a position of moral authority.