Anyways, PLDI has the "Best of PLDI" papers from 1979 to 1999 (http://www.informatik.uni-trier.de/~ley/db/conf/pldi/pldi200...) and they establish the most important paper 10 after its publication. AFAIK/IIRC OOPSLA/SPLASH does this now as well. I think it's good, but it would also be very interesting to know, whether there is an intersection between the set of "Best Paper" awards and the set of "Most important/impact" awards.
Judging quality remains a difficult task for the initial reviewers, but also for the best paper award committee. Despite its honest efforts, the best paper award committee has not selected papers that are cited more often than other papers. In other words, the best paper award committee did not perform better than random chance. From this viewpoint, Desney Tan’s claim that “The Best of CHI awards represent the top one percent of research submissions to CHI” appears too optimistic.
http://www.bartneck.de/publications/2009/scientometricAnalys...
My own rant on this topic: http://raphaelwimmer.wordpress.com/2010/11/10/best-paper-dem...