I'm not sure what you're saying with this suggestion. If somebody has what they perceive as a minority opinion, they shouldn't feel shamed into not voting in kind. Anonymous voting subverts that awful societal problem.
The other point is whether the flagging is legit, or being influenced by special interests. If the same X number of users were flagging everything related to Y, there is no way to know under the current system.