Then you are relying on that password manager, what if you lose access to it or the data with your keys? Welp...
It is. That way you only trust your password manager and not every single app you use (that can easily dump your password when you login, or might even store it as plaintext).
If a password manager is open source and you know how to audit it, you could do that. Otherwise you can ask somebody you trust who knows better.
> what if you lose access to it or the data with your keys
That is a possibility indeed. You should make backups, of course.
By reusing a password, you're basically doing this already.
We’ve been working very hard on this problem, and recently devised a scheme known as backups. It’s like a whole copy of your data someplace elsewhere!
Snark aside; popular password managers have extensive recovery and replication capabilities.