The downsides I can think of in that case are an attacker attempting to log in at the same time, or -- more generally -- a user thoughtlessly clicking the link when an attacker is trying to log in.
The first problem can be addressed by displaying a second readable token on the login screen which the user should verify is present in the email before they click the link. That is how banks and other high security systems do it here in Norway now when you select authentication via mobile, except they use some sort of mobile communication tied to the sim card instead of email. You get a popup on your phone containing two random words from a dictionary. You confirm that the words match what the browser is currently displaying and then tap "OK".
The second problem can probably be reduced by short token timeouts and using appropriate language in the email to emphasize the implication of clicking the link uncritically.