This is a good point! I think that the concept of "who do I want to associate with" is a different way to view things than "everyone I see in the world needs to treat me with agency"
I have a hard time knowing where that line is. Like you said,
> I AM a free speech absolutist (or close to it), even if it does result in some discrimination. (Even against me.)
People out there, due to their agency, may not agree with where my agency ends and theirs begins. I think if we have the privilege and feel able to "choose to avoid" those that we disagree with, we could have these discussions in the open without fear and actively change people's minds.
I don't know how to though. I've written about it here [1] but I still don't have a good answer for how do we draw that line of where your agency ends and mine/theirs begin.