Another big one is that this method is not entirely device agnostic. I am not able/willing to log into my email account on every device that I might want to log into your service with.

One way this has been prevented for me so far is to include a one time pass in the mail along with the link, but some people might still not have a second device on hand every time they want to log in.