The threats that TOTP protects against are ones that don't involve losing your device. For example, if somebody breaches a password database or phishes your password, TOTP codes prevent them from using the leaked credentials.
Phishing/bulk password dumps are more common issues than device theft.