Generally this happens when the contest is run by marketing people as opposed to those from a developer background.
You've just got to build something you'll enjoy making and using regardless of the chance of winning.
From a marketing perspective, you hope to get a lot of cool stuff written for your API (and blog posts, discussion, etc) around your API. You think you can probably get 10 applicants, so you ask your boss for the ability to give a prize out for a contest to make it happen. Yet, then the only code that is submitted ends up being not far from a "Hello World" program even though 20 people said they wanted to enter. Now you look stupid to your boss that's wondering why no one entered and why they are paying $5000 for a Hello World program. You also look stupid when you post the Hello World program as the winner. Finally you look like an asshole when you close the contest instead.
As a developer you think, "Hey this is cool! I wonder how many other people are entering? My chances are probably pretty good." You get together some code, but hit some documentation issues with the API and you get distracted by your dayjob a bit. You end up being the 'winner' even though you only sketched out a small amount of code, but you feel totally screwed when they just cancel it.
Its easy to blame the product and marketer here, but I happen to know of one media center software company that was running an API contest. When I talked to their CEO I got the sense that they were dying for people to enter. It was well publicized, a good product and had good stuff to win. Yet, I personally got the sense that few were entering. It comes down to the fact that API contests are really hard to run!
Oftentimes as the marketer you're being pushed to do one (or pushing to do one), but don't have the full support of your development team to make it easy for people to enter. Maybe the devs release an API, but its only documented in LISP and its not a RESTful API (maybe for a piece of desktop software). LISP and its variants are awesome, but seriously few people who have spare time and are using your software can write well in it. Regardless of interest, few people enter. If you had gotten them to make a RESTful API, or documented it in Python, C, PHP, Ruby and Perl (with API wrappers), then more people would have entered. Yet you're the marketing guy and telling the dev team what to do is nearly impossible.
I think the key to entering contests is to have no expectations of winning as iamdanw says. Just build it because you want to use it and if you win that's awesome.
If people made some cool stuff for this contest I think they should be awarded, but I can also totally understand if there was a single entry that wasn't good too.
What you DON'T do is to advertise that you're offering a contest, then fail to pay out the prizes. That's not just sleazy, it's illegal.
Just to play the devil's advocate, what's stopping unscrupulous companies from canceling the prize and saying they did the above?
They say, "I'm sorry, you weren't the winner" to any contestant who asked, and without a public announcement of who the winner was, they could get away without paying the prize money to anyone.
Contest law is fairly well-hammered out, and surprisingly subtle; a few months back HN had a story about the legal dangers of doing a contest like this. If this story is accurate, it seems very likely GamePro is in violation of the law, if they are a US company. I don't know for sure as I am neither lawyer nor totally aware of the contest in question, but it's very easy to be in violation of these laws.
You could have easily made it viral by making it part of the game that at least, say, 1000 people had to enter, and that for everyone you refer to the contest who enters, you get something, or you get a small purse if one of your referrals actually wins or something.
They didn't think it through, they weren't smart about it, and they screwed people over.