There was a great talk by some AWS guys at the aws re-invent summit on ways to improve accuracy using ideas similar to cross-validation...

It's on iTunes - title is: "Getting to ground truth with Amazon web services mechanical Turk"

Video also available on YouTube: https://m.youtube.com/watch?v=vRtLdeNl7Tg