I graded for a class that took a similar approach to this - the students had a small amount of tests available to them that allowed them to iterate over the requirements for the assignment, but we had more exhaustive tests that we ran automatically. In addition to checking test completeness, we also looked at the source code itself and graded for quality of that. The automated runner handled a lot of the hard work for us, but it definitely wasn't a situation where we saw some failed unit tests and then proceeded to fail them or anything.

The system used to pull this all off is pretty interesting, I think. It's all open source: https://github.com/redkyn