We (the PLASMA research group at UMass, http://plasma.cs.umass.edu) developed a system called AutoMan specifically designed to automatically manage quality (as well as to automatically compute pay and time) for a wide variety of tasks. You basically invoke people as functions and it just works, with statistical guarantees (it also handles payment, etc. without any additional effort). Makes dealing with MTurk much nicer. Best used in Scala but also can be used from Java.

http://automan-lang.com https://github.com/plasma-umass/AutoMan

Paper here on AutoMan, round one: * http://cacm.acm.org/magazines/2016/6/202648-automan/abstract (CACM Research Highlight, 2015)

Original paper, not behind a paywall: * https://people.cs.umass.edu/~emery/pubs/res0007-barowy.pdf (OOPSLA '12)

New features described here have been rolled into AutoMan: "VoxPL: Programming with the Wisdom of the Crowd" (CHI '17, to appear): https://people.cs.umass.edu/~emery/pubs/voxpl-chi.pdf