It's not /hard/ using a library, but it's hard using it properly.
And unless you're a cryptographer, you ABSOLUTELY SHOULD NOT roll your own encryption, and even then you're likely to screw things up.
[1] I know PKCS#11 and friends try to do some of this, but no-one fully implements the entire standard(s), and no-one implements a standard set of features either ... which makes finding a provider and a client with overlapping feature sets somewhat interesting.
I've done some (proprietary) work in this area, and I think that it ought to be the way to go.