Count every user's up-votes and down-votes.

Change voter weight depending on how much it correlated with PG's votes in the past. Vote can even change direction if inverse correlation with PG's votes was observed.

Next step would be to create personalized page for every user when new submission is sorted according to correlation with that user (instead of PG).