Excellent point! I see more and more of this hysterical attitude in tech. If your only concern with a conference is about the panel diversity then perhaps the conference is not for you.
This sounds like a straw man argument. You've created the fictional idea of a person whose interest in a conference is solely the diversity of the panel, as if there are people who don't even care what the conference is about or have any idea of the subject matter; they just like seeing different kinds of people on panels.