I don’t see people finding the fact infuriating. People are upset because they feel like people are asking them to come in to the office because someone else wants to socialise.
Enforced socialisation doesn’t seem reasonable any more than denying that some people like to socialise at work. Two sets of people, two different needs.