If that user doesn't have many people blocking them, then this will barely affect the votes. If they have tonnes of blockers, then would indicate that they probably deserved it, and the downvotes would make sense anyway.
It would prevent echo chambers from forming.