The task avoids assuming any knowledge of algorithms / advanced maths / databases or web frameworks. We're only interested in their approach to solve a problem with code, not what libraries they've spent years learning, or what project euler problems they've committed to memory.
I came up with the test after I was going round a job fair (for the free stuff) and was handed a coding test to produce a simple CLI twitter clone. I had no interest in applying for the job but I did the coding test anyway. It's only 2 hours of my time and I find coding / solving problems fun.
It's really helped us distinguish developers who see work as a 9-5 job that they only do because they must and those developers who just love solving problems. Maybe the technical skills of those 2 developers are roughly equal, but one of them is going to be far more valuable long term than the other. So far we've only hired one candidate after introducing the test, and they've been the best hire we've made.
As a side note, we always do a phone interview first so that we don't waste time (ours and theirs) giving the test to complete no-hopers.