I've done this many times in the past, if on a more private scale (e.g. on the developer mailing list of a particular project). In fact, a few years back, I started work on a website to facilitate this kind of project, though I never launched it (but I do still hold the ransomware.com domain name just in case I decide to come back to it). All of those projects ranged in cost between $500 and $20000, and most eventually turned out good (though I sometimes had to double down and hand the code over to someone else, or work on it myself, to get it polished and ready for production).
In the Open Source world, this kind of thing is not at all rare or unproven. It can work extremely well, and if it's something that works within an existing community of users and developers you get free quality control and vetting of the results...if it's a standalone project, I suspect you'll be doing your own QC. Though I'm sure you wrangle some of your fellow YC'ers into helping bang on it if it's generally useful.
Firefox plugins are a good choice for this kind of model (and I had a Firefox plugin made for about $4k about three years ago, in a very similar process), as they are small, easily testable by a wide audience, and can leverage existing infrastructure for the hard stuff.
The tricky thing, however, which you've noted, is that since only one prize is being offered, you'll have a hard time getting real buy-in from a large number of developers. $3k guaranteed is a great motivator for a student to spend a month working on something. A 1-in-50 chance at $3k is far less of a motivator. So, I've almost always broken it up into a "prove you can and really want to do it" phase and a "OK, now do it" (and the occasions where I didn't I already knew the work of the people offering to work on it from their Open Source contributions--this is tricky if you aren't following their projects reasonably closely). I didn't always phrase it as a "prove you can do it" project...instead I would pick the highest value low effort part of the project and offer a pittance for the first person to solve it--$500 is plenty, if the task is small enough. Then when the best version is delivered, I'd hire them for the rest of the project.
All of this begins to sound more like traditional contract work, rather than a contest, of course. But it does work, and making it a public process does tend to get more community involvement than a "I hired this guy to work on this project and now it's done, let me show you it" situation--if you need to get the project merged into the core project codebase, this separatist practice can make the merger very difficult...personalities get bruised if they think you're absconding with good ideas and dropping in a bunch of code they've never seen or talked about.