Yep, I believe there is a way for sure. YouTube can evaluate users by how likely it is that their flag results in long term removal of a video. If a user cries wolf too many times or flags videos later marked as not controversial at all, then that user probably should not have flag powers.