I think you are describing TREC[1]. Over on the chatbot arena, there is The Loebner Prize[2], also interesting, but in general the systems are over learned.
[1] http://trec.nist.gov/ [2] www.loebner.net/Prizef/loebner-prize.html
[1] http://trec.nist.gov/ [2] www.loebner.net/Prizef/loebner-prize.html
Jeopardy is a widely-understood game format; a competition-via-API would be understandable to a wide variety of competitors, from leading researchers to precocious cranks.
Outside TREC, there is the CLEF QA track[3]
[1] http://trec.nist.gov/data/qamain.html