You should have to enter an explanation when you downmod something. Then the recipients won't be left confused, and it will enforce responsible use of the privilege. I feel this is a better solution to the abuse problem than only allowing downmods for 24 hours.