Even with strong crypto, if you're typing in your passphrase on a "dumb" machine that someone else owns and controls, you're risking the entirety of your security on how trustworthy (and security-competent) the owner of that machine is. No thanks.
I know there's no infrastructure in place (yet) but we're talking about some kind of future, not about today. It's certainly possible though. Even secured.