Using an aggregation of the per-user lists to guide the evolution of the site-wide list would probably resolve that concern. It's also conceptually very similar to how submission flagging currently works here.