I don't have much experience with mturk, but my one concern would be that respondents may simply click randomly rather than honestly answering the question; do you have any data on how honest and accurate your responses are?
I don't have much experience with mturk, but my one concern would be that respondents may simply click randomly rather than honestly answering the question; do you have any data on how honest and accurate your responses are?
For example, we used this feature to dogfood our own FAQ page: http://pickfu.com/4UQM7U
the FAQ example is a bad example: I can always answer "yes" and then supply as reason "every questions I had was answered". No need to read the FAQ page, of course.
I think if you want to insert any kind of validation system you would have to count the time it took for the turk to answer your question and make sure we are in that range for every hit.
I'm sorry you had a rough experience with that specific music-listening job - we've never had a more time-consuming question submitted. Most are in the order of "which site do you prefer?", like this current one: http://pickfu.com/AGSIBT. These get answered far faster by the MTurk workers.
What would be truly amazing if this service could work with non-mechanical turkers...