Mutt is great if you're not often dealing with images or rich text, and if you love keyboard-based workflows. I was a mutt guy for a while. But I've found apps like MailMate or even GMail's web client to be keyboardy enough, without some of the compromises and pain points (e.g. fighting OfflineIMAP)

Not sure why this is being downvoted. It works for a lot of people I know, is modular, stable, open-source and works across the Unixen.

The obvious tradeoff the relative lack of polish and overhead during initial setup, but if you're willing to put up with that it's a fine choice.

I have wanted to switch to mutt a few times but it feels complicated to setup and adapt to my workflow. Plus I want to POP down most mailboxes but not all of them.

are you using mutt's built-in imap functionality, or something like offlineimap?

I also use Mutt.