How well do you think those students would fare if placed in normal class? Most of the normal people I meet either openly make fun of things I know or assume I'm full of shit for the volume of information I can pull out of things I've read. Most people also treat me as if I'm a geek because they wouldn't have a use for the things I remember.

You're thinking about this as if they're taking your resources. Let me turn this around for you. What if they aren't being given special treatment because they need the resources, but because they need an environment where the way they think is acceptable? How many of them would still show any talent or aptitude in normal class with normal classmates? Think about your reaction to negative numbers versus their own. Maybe gifted and talented classes aren't so much giving them more resources as they are giving them an appropriate education. Maybe it's about creating a safe environment for kids who tend to be more abstract to flourish.