There's definitely some merit in this approach over something like Fizzbuzz and whiteboard problems. At my previous place of employment [1] , after the initial interview, you were given 48 hours to produce a simple web addressbook app. You could use anything that you wanted to produce the app - at the end of the 48 hour period, you had to supply thme with the code and instructions on how to install it on a Ubuntu instance. It was actually quite fun (and then after getting the job and testing other people's entries, you begin to see how it separated people with impressive CVs who couldn't program for toffee, and others who had much less experience, but managed to come up with something that worked!)

[1] That would be Open Source Integrators, a great Durham-based outfit who I believe are still looking for people to staff their Chicago offices: http://osintegrators.com/