Hell, no. If you have a car just sitting in your garage, should you have to surrender it to the DMV (department of motor vehicles)? I've purchased many domains that I planned to use; but plans don't always work out. That doesn't mean I should have to surrender my assets. Yes, I'm frustrated when a domain I want is taken and I visit the site only to see a "search page". But that's the owner's option. Maybe all those dormant projects on SourceForge should have to give up their project names, too?
Right I agree, this gets into the who fairness and equality debate, where everyone has a definition of fair and equal. The fact is, the people that purchased those domains played by the rules. Sure speculation can seem like a free ride and it is not that sexy but people should not be begrudged for getting lucky especially if they did so via a little foresight.