This is my biggest concern about 2FA. On Google I have TOTP 2FA enabled, with my phone number as a backup. But everything goes through my phone - I'm not really worried about some stealing my phone to gain access, but what happens if I break or lose my phone.
A few years ago I was travelling abroad and had my suitcase stolen - which had my laptop (with FDE) and passport inside. Fortunately I had my phone and wallet on me, otherwise I really don't know what I would have done - I probably wouldn't even have remembered the name of the hotel I was staying at that night.