Remove it from both. However make sure that you have quite a lot of backups of your 2FA backup keys, and maybe even one offline backup of your seed, if you lose them, the account is gone (which is a good thing, I guess).
A stolen phone can of course be a problem as well, but at least it's somewhat under my control and I may notice pretty quickly when it's gone.