I think the 500 limit is justified as a means to discourage vote-brigading. Age of the account, by itself, is not a good enough indicator, because a person might create hundreds of accounts to be used for brigading at a later date.

But, yes, both downvoting and flagging can and have been abused here, despite measures to prevent them.