Store your TOTP secrets PGP or NaCL encrypted. I have done this for years. You'd have to get my private key (off of a smartcard) and get my private key password (out of my head) to decrypt my TOTP secrets before you could use my TOTP generated codes. I have much more faith in this approach than I do Android apps.
$ gpg -d encrypted-secret.txt | goathgen