It's not obvious to me that if everyone has a downmod button the library will be better. Somebody is upmodding books you don't like. How do you know they aren't downmodding the books you like?
I suggest that if the library has a few titles you don't like, going without a downmod button is a good thing, just ignore them and move along.
But if the library is overrun with bad titles, giving everyone a downmod button won't solve the problem, because the problem is caused by the librarians.