The problem is that only 10% (or something) of programmers are any good. Not 10% in each language community, but 10% overall. When you put out an open position for Java, you are attracting everyone that knows Java -- Java is big for "programming as a career" because there are lots of certifications, and it's mandatory for all college students. So that 10% isn't going to apply to Java, it's going to be 99% people looking to advance their career (read: collect money for clicking buttons in Eclipse), and 1% of people that are actually good at programming.

(Honestly, my resume claims I know Java. I do. But I would not be a good fit for your position because I hate Java with a passion. Even though I might interview well and be interested in collecting your huge salary, that just means I am advancing my career, not being a good programmer. And that's what you don't want, but what the word "Java" encourages.)

If you pick a less popular language keyword to search on, though, you might get better results. We added the keyword "Scala" and got a lot better resumes. Nobody learns Scala because they think it's "good for their career". They learn it because they like programming. That's who you want to hire. (Incidentally, this does make it good for your career.)

I've found that this works in a variety of language communities; C++ is popular in industry, so ask for C instead. Java is popular, so ask for Scala instead. OO Perl is popular, so ask for Moose instead.

This is not as good as knowing who you want to hire and just asking, but in a field with 1000s of landmines, you want to minimize your chance of blowing your leg off. Yeah, you are going to eliminate a lot of great Java programmers when you search only for Scala. But the people that you do interview are a lot more likely to be good at programming.