It's not karma-driven at all, except insofar as downvoting itself requires a karma threshold—a restriction that has been the same for years. It's possible that we'll lower the threshold at some point.