No, you really can't, unless you stretch the definition of "side-hobby" to the point of breaking at the seams. I'd be shocked if a single person who professionally works in security or cryptography is going to agree with you in this thread. You need far more than a "little care" to assure that a new cryptographic library is safe. It's an endeavor suited to a collaborative academic or corporate environment, with robust human capital and strong oversight.

I'm willing to accept that someone can safely implement a primitive or a construction in a new library with significant time, effort and feedback from others. But that level of effort doesn't really qualify as a side-hobby, and until the implementation was formally checked by professional cryptographers I'd consider it suspect.

This is to say nothing of designing novel primitives or constructions, which I would consider as far removed from a "side-hobby" as a local 5K is to the Olympics.

However, I absolutely think people should attempt to implement their own cryptography if they're curious about it and want to learn. Just don't use it in production and assume it's unsafe. People who work in the industry don't make these claims because we think it's fun, we do it because we've all seen the consequences.