Let me suggest one solution for such cases: Use iterative tasks, in the spirit introduced by TurkIt.

For example, you want to create a caption for an image. You let a user create a caption. Then you take this caption and give it to another user, asking the user to improve it. Take the two versions and ask other workers, "which of the two versions is better?". Iterate until no improvement is possible.

Not a trivial setup, but gets around the binary accept/reject decisions problem and generates results of significantly superior quality.