A bigger security threat, I think, is that you have the private key both on your server and on your computer. It adds another location where you could mix up, hackers now have 2 possible attack targets, and it's more likely that your PC gets infected than your server. Either way, now, if your PC gets infected or your server gets hacked they will have the private key, while if you only have it on your server they won't necessarily have it when your PC gets infected.
The safest solution if you don't want to store the private key unencrypted is to generate an encrypted private key with openssl. You would however need to provide the encryption key every time you start the server. You will still have the unencrypted private key in RAM, but that's inevitable and also the case with your current method. The private key (even encrypted) should never leave the server.
From a threat modeling perspective, nothing you do will prevent an attacker that is able to run as your application user (or root) on your server. That's fine; The level of obfuscation you've put into place will (possibly) keep some of the script-kiddies who aren't targeting you directly from realizing they've stolen your key.
The attack you should be concerned about is on your distribution side: You can't copy that binary anywhere else without revealing your key. You can't put it in a docker image repository or a java package repository or even a s3 bucket - Because those become places that your key can be revealed. And you want to do those things. You want a copy of precisely the binary you deployed.
What about backing it up? Or if the server disappears in a fire, the key would be gone for good and one would generate a new pub-priv key pair?
What about server whole disk snapshot backups; they'd include the key?
> it's more likely that your PC gets infected than your server
That's a good point