Sure, writing your own library is very difficult, but you have a simpler set of problems, which proper testing, another set of eyes and enough tools will take care of the big problems.
Now, implementing your own primitive and recommending to use it is bad. For a primitive to be deemed secure, it has to undergo years of review and testing by seasoned cryptographers.
This isn't a recommendation to write your own library, but my opinion that it is okay to do so if you want something that none of the major libraries offer.