It's asymmetric encryption. Even if the server got a hold of the public key, it would not be able to decrypt the contents.
How to ensure the server doesn't get a hold of the private key is the issue (can you really trust the code you're running?).
How to ensure the server doesn't get a hold of the private key is the issue (can you really trust the code you're running?).