Conceptually it seems relatively simple: each voter supplies a "desired score", which would be +∞ and -∞ for regular upvotes and downvotes respectively, and 0 or other integers for those who want to be fancy. Then you can decide whether n is the correct score, or too high or too low, by assuming everyone wanting a number above n votes +1, everyone below n votes -1, and everyone who wants n votes -1, 0, or 1 as necessary to keep the score as n. (If k people wanted n, then if the score from everyone else is within [n-k, n+k], then n is the right score.) Updating the score one vote at a time seems doable. It would require storing the entire voting history with each comment, though, which would probably require changing things and be a pain.
Then the problem I see is with the user interface: most people wouldn't want to use it, and it would clutter things up (a numeric input field, or a specialized "vote towards 0" button). I guess it could be a setting that individual users could enable. I'd hazard a guess it would be deemed not worth the trouble.