But the bug will still be there, waiting for an actual attack.
Of course there's no problem if you do it right. It's just that guaranteeing you did it right is never possible with 100% certainty if you wrote the code, evaluated the code, and published the code.
More eyes means less mistakes. More hands makes less work. If you roll your own templating system, and you mess up, then some text on a page is displayed wrong. If you roll your own crypto to protect customer data and you mess up, whoops, that customer data is out there.
That said, I think there's still a major difference between roll-your-own-crypto and other "unless you do it right" topics. Securely storing passwords and preventing XSS are things that can look ok and be insecure unless you do it right, but I'm not (always) against people (carefully) handling those things themselves. Rolling your own signatures or encryption are infamous because a "best practices" list and good code review won't save you; even if you root out all the actual bugs, your process can still be cripplingly vulnerable. As Schneier said, anyone can create an alogorithm they can't compromise.
This system is minimal, is running almost perfectly and has > 100 happy users. There isn't sufficient justification to introduce custom code layer and break all the existing functionality.
pgcrypto already does support PGP encryption after all, just not ECDSA.
Isn't this the #1 anti-pattern in cryptography?