Sure, I guess. I would tend to argue that with hardware access they can do a number of things, though:
- They can install a wifi-enabled* keylogger that exfiltrates your OTP. Yes, they have to use the OTP before you do, but that seems solvable to me--just program the keylogger to transpose one of the six characters that comes after your (now known) password.
- They exploit any number of likely plug-and-play vulnerabilities in common OSes (e.g. https://www.thetechherald.com/tech-news/disable-plug-n-play-...).
- They steal your computer. Are you using FDE? If not, your cookies are sitting there on disk waiting for them.
- They...steal your phone next time you forget it at your desk. ;)
Are there scenarios where TOTP can protect against a non-phishing attack? Yeah, I can construct one if I really have to. (I think you're thinking too locally; my top argument for TOTP would be that I can rarely be sure the identity provider has implemented reasonable quotas against password brute-forcing--and who knows how well they secure things like server logs against rogue insiders?)
But if you tell me you use TOTP because you're afraid of a local attacker plugging in a keylogger--and not because of phishing--I think your priorities are out of whack. :)
* Or they use a covert channel, but I'm not serious about this part: https://dl.acm.org/doi/10.5555/1267336.1267341