There is something to be said for the novelty probably getting more interest, but I think that giving puzzles around the real work the hire will perform tells them as much about your company and the problems they can help you solve, as it tells you about the candidates ability to do so. A win/win as they say.

I already ask about 10 technical questions in each of the interviews I conduct, and having a problem online for each candidate to hack on would be really useful.

Now that I think about it, I may even consider using something like this to drive internal improvement projects. Some people just love the challenge, and the 'a' people always step up, you just have to ask.

And perhaps tedious.