Passwordless login is generally as secure as most password reset strategies. (I.e. simply email password to an email address without doing further identity checks such as checking last remembered password or asking security questions)
It doesn't really matter if an advanced technique like you described is used, or if the email account itself is compromised, the security considerations are the same.
The one advantage that password reset emails have over magic link logins is they can provide users a clue that their account has been compromised when the old password doesn't work. (Though in practice, how many users will just reset their password and move on with no further investigation?)
In additional to single-use time-limited tokens, there are two additional measures that address this type of attack:
Providing details on the previous magic link that was used (date, time, IP address and maybe user-agent details) in the email with the magic link provides the ability to detect people who have gained access by requesting a link then deleting/blocking the email so that you don't see it.
Limiting the token to being used by the same IP address that requested the token prevents people from watching for a legitimate magic-link request and then hijacking that link for their own access.
With those two measures in place, I don't think there is any security downside to switching to passwordless logins.
I would note that if you employ both password and passwordless logins, this increases your attack surface and decreases the chances that intrusion will be noticed. (I.e. if the target uses a password to login, the can request magic links and then delete them and the target may never see this activity since they don't request their own emails. Conversely, if the target uses magic-links to login, the attacker can reset the password and the user may not ever notice). As such, it may make sense to limit users to a single login method (or always display a page showing last magic-link login and last password reset when logging in)