The actual economic theory behind prediction markets (if you read Robin Hanson's work, efficient market hypothesis, etc.) is that these markets work precisely because informed traders bring private information into the price. That's literally the core mechanism that underpins them.

Yes prediction markets represent "wisdom of crowd", and they do this by rewarding people who contribute correct information. So informed traders are the ones making the system work, and they're putting actual money on the line to back up their claim. Without them you're just left with uninformed guesses, which isn't really "wisdom of crowds", it's noise.