I'll also go further and say you should be using a password manager, so even if you do end up getting scammed out of a login, they can't easily compromise your other accounts (obviously this depends on the kind of account being scammed).
Use a password manager and 2FA whenever possible!
You don't have your 2nd factor device (phone) and someone else has access to it and presumably an email account too? I guess they may still need the device's unlock code (PIN/finger print) to do more serious damage, but it'd be rough if you're trying to change your passwords, but can't because you are out your 2FA device. All the while the attacker is able to reset your accounts/passwords.