This is not the only motivating factor for curve25519. There is also: that montgomery curves work well with the montgomery ladder, which is easy to use in constant time, and that any 32-byte string is a valid public key for ECDH.
> Now we are stuck with all these cofactor issues.
They are not a major problem for ECDH. If you are doing only ECDH and don't care about group structure, you can simply use the existing clamping mitigations.
The point of ristretto, and its precursor/similar project decaf, is to preserve group structure while using these curves, and also eliminating small subgroups.