I wouldn't leave it to them, but I would allow them the ability to specify it. You could give users the option to only filter based on votes, if you so chose.