It depends on what you're trying to accomplish.
Rule of thumb is: just use TLS 1.3 if you can. That includes using TLS 1.3 in mutual auth mode if you need to validate both client and server identities. It comes with a lot of PKI for free to help address the "how do you know you can trust the other guy on the line?" problem. If you want the client/server to trust a restricted list of servers/clients, most clients and servers have that capability.
If SSL/TLS won't suffice and you just need to create an end to end "thing" with proof of identity, then libsodium is probably fine.
If you're looking for a more general purpose crypto library where you need to mix and match algorithms, I'd recommend something like Google Tink (https://github.com/google/tink), or the high level interface to the native API's that come with the OS. On OS X that's the native security framework (https://developer.apple.com/documentation/security), and on Windows that's the .NET cryptography library (https://docs.microsoft.com/en-us/dotnet/api/system.security....).
If you need even more flexibility and are willing to do the research to avoid the footguns, there's always OpenSSL/LibreSSL, but that comes with a huge footgun warning.