Anything that's worth participating to is elitist. It brings together interesting, fascinating and smart people. Most admirably, TED succeeded at keeping the audience eclectic. Nobody learns from homogeneity and confirming their thoughts with people who think the same.
As a TED attendee and TEDx organizer, I can definitely confirm on this. Whether for my event or for TED, I see the talks as just an excuse to get amazing people together. Everything — from the high cost to the application process — is about ensuring that every person in the audience genuinely wants to participate and contributes to the experience.
Case in point: the talks are posted for free online to weed out those who attend for the information in the talks. The Internet trivializes pure knowledge and makes human interaction so much more valuable.
When you bank your event on people (which is pretty much the only thing you can bank on in the age of the Internet), you have no choice but to make it elitist. In a good way.