Assume for a moment that the implementation of the encryption and decryption processes are perfect. You've now completed the easiest part of the problem (to be very generous, let's call this 10% progress).
You still need to solve the problem of distributing public keys in a way that both parties can be sure they're receiving and sending messages from the correct entity. Do you trust all of the certification authorities that are automatically trusted in standard browser or operating system distributions? If not, how do you communicate public keys securely between two or more parties?
And how do you distribute the application and ensure that users aren't tricked into using a modified/backdoored version?
How do you ensure that users aren't tricked via a phishing attack into inadvertently handing over keys, downloading software from the attacker or otherwise compromising their own security?
If one party to a conversation is compromised, do you care if the identities and messages from other parties to the conversation become compromised as a flow-on effect?
How do you ensure that parties to a conversation are receiving all the messages sent -- is an adversary blocking random messages? Is an adversary reordering the delivery of messages to change the overall meaning of the conversation?
Have you carefully considered the implications of replay attacks?
Do you care if the system is leaking important side channel information such as the frequency of communication and time between responses?
Have you considered what users will resort to in the event that their communication method is deliberately denied? Will users fall back to a weaker method of communication? Will users tend to perform a Google search that returns a maliciously placed help/reference page that executes malware on their system?
Does the system have a means for users to expose the presence of an adversary that has compromised the system?
Algorithm implementations are just a minor (but still highly important) aspect of a full crypto system implementation.