No it doesn't, it affects their parents, their children, their friends, their pets, their colleagues, the companies they work in, their clients, and everyone else actually.
You don't live in a vacuum, you live in a interconnected system where everyone has an effect on everyone, direct or indirect.
Liberty is extremely important, but we should always remember we're not alone.
This belief that everyone should be 100% free and have the right to be whatever he wants with his life is, if not the only reason, part of the reasons why we're in this mess today.