Cleartext-based attacks and snooping (middleman, network peer) are simply far easier and far more likely.
You know those happen all the time, right? Literally all the time for certain users in certain places. A cleartext exploit is source-based (or route-based) and the server owner can't ever do anything to fix it besides forcing encryption (or shutting down completely to that source).
In contrast, a server compromise is destination-based and will theoretically only exist for the time that it is not known to the server provider.