Perhaps what is needed is a simple standard API that wraps OpenSSL, GNU/TLS, mbed TLS, Polar SSL, et al - that covers a small set of standard use cases.
[1]It's not /hard/ using a library, but it's hard using it properly.
And unless you're a cryptographer, you ABSOLUTELY SHOULD NOT roll your own encryption, and even then you're likely to screw things up.
[1] I know PKCS#11 and friends try to do some of this, but no-one fully implements the entire standard(s), and no-one implements a standard set of features either ... which makes finding a provider and a client with overlapping feature sets somewhat interesting.
I've done some (proprietary) work in this area, and I think that it ought to be the way to go.