Ask HN: Critique this Peer Review Algorithm
Given A group of peers. In this case a self selected group of persons who paint artistically, say 4 hours or more a week. Let these be the peers.
Hypothesis A "consensus" of opinion within this group should establish a qualitative analysis of the whole pool of submitted works; a curve. If each individual within this group of (approximate) peers is given the ability to actively criticize a significant percentage of the other artist's work, I believe the result would be simple but accurate stratification (indicating, roughly, the top dog).
Algorithm Hypothesis
-- 500 artists who all work in a similar category (realism, impressionism, anime?) submit 1 work of art each.
-- Each artist is required to view and rank 3 unique sets of 5 works. (5 works per set for ease of ranking, 3 sets to increase the amount of redundant input, "unique" meaning each set of 5 works is a permutation).
-- 1500 ranks (ideally) will be submitted. Each piece of art should get ranked 30 times against a completely different selection each time.
Why is this supposed to be good?
-- Those voting are work-proven peers
-- The ranks are not simple "hot or not" choices, but rudimentary curves.
-- Voting is limited, evenly distributed, and public
-- Recursive post analysis is possible based on discounting the rankings of lower ranked voters and, similarly boosting the ranking weight of higher ranked voters.
Disclaimer I am working on a project that will use this algorithm, and your comments, links and criticism. I am not a statistician, obviously.