Instead of having to remember a password, now I have to break my flow, go into my email client, and wait for an email to arrive. I have actually tried this flow with Slack's "Magic Sign-in Links." In my experience, it is not uncommon for it to take a few minutes for an email to arrive. In some cases, that could be enough to make me give up and move on. (I do appreciate Slack's approach to this -- you can choose to either use your email/password OR the emailed link). The thing is, the first usability problem (remembering passwords) is one that I can easily fix on my own if I so choose (either via a password manager or, heaven forbid, letting the browser remember it). The second (having to wait for an email) is totally out of my control.
That said, the one place where passwords are still a major hassle is in native mobile apps that don't integrate with a password manager. I still think that bouncing out to 1Password and copying/pasting my password is overall preferable to waiting for a link in email, but it's far from ideal.