You can find at least one person who is deathly allergic to anything. If I’m deathly allergic to carpets, that means no office in the world is allowed to have carpets?
You can find extreme examples — such as an allergy to sunlight — and sometimes the accommodation for those examples may be different to “banning carpet from the office”.
EDIT: And I'm not trying to advocate for whatever group there is. It's just reality. You make software that is used by people, you need to make it accessible. You make books, you need to make them readable and so on and so on.
And you know what I noticed? I noticed that when people make stuff accessible with least common denominator in mind, that stuff becomes easier and more comfortable to use for me as well even though I don't have any issues.