Putting email in its place with Emacs and Mu4e | Hacker News Reader

Putting email in its place with Emacs and Mu4e

eamonnsullivan.co.uk

110 pointseamonnsullivan54 comments

I wish there were a good workaround for those of us condemned to MS365/Outlook. Outlook desktop is very unstable and buggy, and Outlook web is full of weird antipatterns. For example, it is absurdly annoying to get a direct link to an e-mail message in Outlook web. If it were easy, I would just pass that url to org-protocol in the browser and keep my tasks organized with backlinks to the e-mails that originated those tasks or projects.

As it is, my emacs and e-mail are almost fully separated due to (I'm assuming intentional) lack of a simple method of interoperability.

I have been able to use mu4e with my o365 account using davmail (https://davmail.sourceforge.net/). I will say it was a bit of a pain to get authentication right, and involved a lot of trial and error.

Could you share a bit more details on your config?

Oh goodness, truthfully I'm not sure how well I understand it. I initially set it up a few years ago, and recently futzed with it after my organization changed something about their auth. My configs aren't really in a shareable state so I hope this ramble is at least somewhat helpful.

Basically DavMail connects to outlook at creates a local smtp and imap server which I connect to with mbsyc and msmtp. Mu indexes these emails from the local server created by DavMail, and Mu4e displays them and sends them with that local server as well. Once you have DavMail setup you can basically follow any standard mu/mutt/msmtp/mbsync tutorial, just use localhost and the ports exposed by DavMail.

Getting DavMail setup can be the tricky part, I remember having a lot of trouble, but I think it was related to the fact that the config I was editing wasn't being picked up systemd service that was controlling DavMail. The best advice I can give you is experiment with different authentication modes (davmail.mode in the config) and try sending mail to the DavMail server in an attempt to trigger it to do the authentication workflow.

In the end, set davmail.mode=O365Manual and davmail.url=https://outlook.office365.com/EWS/Exchange.asmx Upon attempting to send an email from mu4e it opened up a browser to do a microsoft authentication, and then I believe it saved a token in my config file (variable davmail.oauth.<your email>.refreshToken) which has been handling authentication without issue for the past few months.

Some miscellaneous notes. First, this may have been harder for me as it was not possible for me to use the DavMail GUI which might make the authentication workflow easier. I also have two email username@organization and first.last@organization. I have all of my davmail,msmtp, and mbsync configurations using username@organization, My mu4e config references the username@organization maildir folders, but my user-mail-address variable is first.last@organization and that is what recipients see (although mu complains about not knowing about the first.last account). Lastly, this DavMail setup isn't mu4e specific, I initially used it with mutt, and it worked for that as well.

I hope this is helpful, if there's interest I can try to go through the setup from the beginning and create a more in depth tutorial. I wish Microsoft did not make this such a pain, and I wonder if DavMail's days of effectiveness will soon be over...

This may help, it has an example pizauth config (scroll down to "Authenticating with pass and pizauth"):

https://stuff.sigvaldason.com/email.html

Is it possible to get the authentication done without running their proprietary JavaScript login page?

Have you tried this yet? Looks like it uses the ms graph API https://github.com/jgunthorpe/cloud_mdir_sync

I got briefly excited about this one, but I've run into the same issue of needing the IT department to explicitly permit me:

https://github.com/jgunthorpe/cloud_mdir_sync/issues/25

Check us out, we support Outlook and any other possible IMAP/OAuth provider. Cross platform, offline-first, and instant syncing:

https://marcoapp.io

For OAUTH2, I recommend starting pizauth (https://github.com/ltratt/pizauth) as a user systemd unit.

Then you can just do (eg. in mbsync)

PassCmd "pizauth show accountname"

I was in this world for like 8 years but switched to Thunderbird after one too many emails didn't send because I missed some notification or something in mu4e, and too many emails weren't rendering well in Emacs, and etc little problems that cropped up. I needed my email to Just Work, not be another aspect of my procrastination machi- sorry I mean, my IDE.

The need for email to Just Work is one reason I've never let emacs anywhere NEAR my email flow.

Everything in emacs becomes a Project.

The nice thing is that once things are setup in Emacs, they usually work without much maintenance or surprises for years to come. How many times have mainstream email clients changed UIs and featuresets in the last 10 years? How many of them do you expect to still exist 10 years from now?

Another benefit is that using standard mailbox formats and separate tools allows you to configure, replace, and integrate any part of the setup. With traditional clients you're locked into whatever they support and allow you to configure.

And yet: my mail flow on my Mac, using the native client, is effectively unchanged for 20 years. I've changed back-ends, sure, but that's a different thing.

And I never had to hack anything together.

Whatever works for you, but I don't believe that your mail flow on the Mac is "effectively unchanged for 20 years".

Wikipedia lists many UI and functionality changes over the years for the Mail app.[1] As recently as this year, people are upset about changes in the Mail app in recent versions.[2]

[1]: https://en.wikipedia.org/wiki/Apple_Mail

[2]: https://www.macrumors.com/how-to/get-old-apple-mail-back-ios...

Kinda peak HN crap to suggest I don't know how my own mail flow has worked.

Has the interface in Mail evolved since the early 2000s? Yes.

Does that constitute a material change, or even an adjustment, on my part? No.

Did I ever have to hack anything together to make this happen so smoothly for so long? Also no. ;)

I tried setting up mu4e once. It wasn’t worth it. It took me literally a few hours of reading random blog posts to figure out the configuration, and that was only to download email. Never got around to setting up sending them, which is a totally separate process. Even then, there were lots of issues. First, it’s slow. Loading an email had a noticeable pause and was slower than GMail. Also, you can’t avoid HTML email nowadays. There’s a very basic render, but expect all the formatting to be wrong. I also ran into rate limits from Google because we get way too much email at work. That’s not mu4e’s fault, but just another obstacle. Can’t really have my inbox be one hour behind real time.

I have been using mu4e for years, and am generally happy with it, and yet... I've never recommended it to anyone else. Unlike, say, org-mode or magit, which I'd happily evangelize.

The pain points are what other commenters have said:

- I don't find the default config a good fit for me, and run it heavily customized. As someone said everything in Emacs turns into a project...

- Performance can be an issue, especially indexing new mail (and especially if you like to lug around a copy of most of your emails locally as I do). On a laptop while traveling this used to be more of a problem, but newer versions are notieably quicker and newer laptops have better battery life.

- HTML rendering isn't great. Thankfully I don't get too many important messages that isn't just plain text. This might be a reasonable use case for xwidget-webkit though I'd imagine there are security/privacy issues to work out. (Another Emacs project -- yay!)

When I started I thought it would be an efficient way to get through lots of emails, and it has been for the most part. I'm just not sure I've saved time overall unless one counts the hours configuring it as "entertainment / hobby" rather than "work".

I too am a bit surprised this made it on the front page. Mu4e is definitely niche, and I wouldn't crow about it like I do org or magit. I've only been using it for less than a month and it will be a while before I know whether it is a net win.

Also, the real test would have been my much more voluminous work email!

The HTML rendering isn't great, as you said, but you are two keystrokes from opening that email in a browser, if you have to.

And I have tweaked the config several times now, but I think that's mostly because I'm changing my (and the charity's) email, which involves a lot of shuffling about. Again, in six months, I'll have another look and decide whether it _really_ helped.

> Also, you can’t avoid HTML email nowadays

This is the reason I haven't tried all the email tools that seem fun to play with, but not worth it :/

I use notmuch for email in emacs. I use w3m to process HTML emails for viewing in text, it does a pretty good job especially with tables, which are still used a lot in HTML email.

And the search.... fantastic. Best email search and virtual folder capability I've used on any platform.

I'll try that w3m viewer - good idea. I used to be on rmail but then decided to take a step back to mh-e

> Can’t really have my inbox be one hour behind real time.

Why not? Does your job mandate that you watch your inbox constantly, and respond immediately to all messages? How do you get anything else done?

My mail gets updated every 3mins... ?

This was my second attempt to get email working on Emacs and I gave up the first time, too. I persisted this time and I _think_ it will pay off. There is the obvious danger of this becoming another "project", but I'll make a note to check-in again in six months. It's an experiment!

I've not seen the other things you mentioned. I only check for email every 10 minutes, but opening and (especially) searching for emails seem much faster than doing it in Gmail. Plus, I can do searches across email accounts, like all unreads across all three accounts. That was definitely slower in the online clients.

Finally, there's a quick ('a' then 'v') way to just open a message in a browser if the HTML is too thick.

I'm also using Mu4e for personal email but stymied by Exchange auth for work email. I've been looking into using DavMail as an Exchange gateway, does anyone have experience with this?

https://davmail.sourceforge.net/images/davmailArchitecture.p...

I have been usong mu4e with davmail in Exchange mode for my Uni mail. It was a pain to discover how to do it but the FAQ has been updated since and it works like a charm. Deop me a mail at gmail if you wish.

Re: the OAuth issues: to remove some of the hassle of this, you can use my proxy/relay to allow any IMAP (or POP/SMTP) client to be used with a “modern” email provider, regardless of whether the client supports OAuth 2.0 natively: https://github.com/simonrob/email-oauth2-proxy. No need for your client to know about OAuth at all.

Thanks, keeping this as a reference — I'm trying to find some time to try mu4e. I used Gnus for many years, then switched to Apple Mail.app, but with the gradual decline of MacOS (and Mail.app) I'm looking to switch back.

I remember the two main reasons I switched from Gnus: 1) there was no good reliable search, 2) I couldn't drag&drop attachments into E-mails and back so I was spending a lot of time pointing to files. I hope both things have improved since then.

I am a Gnus user. My 2 cents…

concerning (1): I have no offline sync in place, all my emails stay on the server. The IMAP protocol has a decent server-side search included[0], combined with Gnus unified search syntax[1], I enjoy a hassle-free search experience.

[0]: https://www.gnu.org/software/emacs/manual/html_node/gnus/Sea...

[1]: https://www.gnu.org/software/emacs/manual/html_node/gnus/Sea...

gnus had some massive IMAP performance improvements a few years (probably close to a decade now) ago. Before that it was quite painful to use it on large mailboxes without a local imap - I used to sync that with offlineimap. When they had a massive issue moving from python2 to python3, and keeping that running on a modern distro started getting painful I tried it without local imap - and realised those improvements made things fast enough that you can run it on remote mailboxes, and even do so in your main emacs instance.

Gnus still sucks. It might improved on IMAP, but with NNTP it's 100x slower than SLRN, even if gnus reuses the SLRN spool.

I switched to mutt, started getting through my email in half the time in took me using a GUI, and never looked back.

Being able to write simple expressions to filter email, mass delete, and avoid embedded javascript are killer features. I can run all html through w3m and still have nicely rendered emails.

I still use a phone app for on the go browsing, but during work hours I have mutt open alongside neovim all day long.

I use offlineimap3 (https://github.com/OfflineIMAP/offlineimap3) to download email that I then read with mu4e. To make it work with my email provider I need to place a cert_fingerprint value in the .offlineimaprc configuration file. This used to change once a year, but now the value changes every 47 days (https://www.digicert.com/blog/tls-certificate-lifetimes-will...).

Has anyone figured out a solution to keep this value updated? One issue is that I'm never sure when the new TLS certificate has rolled out.

I use isync (mbsync), mu, and Emacs for my e-mail needs and it work fine. Both isync and mu4e needed a bit of configuration (and Emacs itself needed one change to stop automatically breaking lines), but it's been working well for years now. I previously used Thunderbird but its editor was garbage, so then I tried a few things (neomutt and I believe something called alpine or something like that) before settling on Emacs. I tried Notmuch, but it didn't work so well with the upstream mailboxes.

I have a very similar setup, but using gnus, mbsync, notmuch, and afew. All mail stays on the servers (including a self-hosted dovecot server on my home network). I manage about 10 email accounts with very little effort. It's easy to get extremely customized behavior by overriding defaults with elisp. Previously I was using Thunderbird, but I feel my emacs setup is much more productive.

Have you tried using the hardcoded Thunderbird (or similar) oauth credentials to authenticate to Google et al? You can also use davmail to proxy your requests to Office 365 / Exchange and it handles oauth also.

I tried both. The error from davmail suggests it was specifically blocked/prohibit and I failed using actual Thunderbird.

Hi, I guess you have already tried.

My Uni also made it difficult but I succeeded in setting up a working davmail, using the exchange protocol.

Mail my username at gmail.com if you think I can help.

Gateway: Exchange Protocol: O365Interactive OWA or EWS (Exchange) URL: https://outlook.office365.com/EWS/Exchange.asmx

(follow the instructions at https://davmail.sourceforge.net/faq.html)

My organization also explicitly blocks access from other clients than Microsoft Outlook, even if the credentials are correct and the protocol is supported. They also refused providing an exception, citing that I can just use the web interface via Microsoft Edge on Linux. (Which I prefer not to do for many reasons, e.g. backing up my emails locally, working offline, and authoring using native tools.)

Currently, only Thunderbird with the proprietary "OWL" extension somehow manages to connect despite the block. My understanding is that they somehow abuse the web interface to do so, instead of actually going through the proper protocols, but not sure.

If someone has another way to access Exchange servers that intentionally blocks non-Outlook clients I’d love to hear about it.

I am surprised as davmail with the Exchange protocol has worked for me since I set it up. They made offlineimap unusable but davmail works (it even has a small web client for the login when more than a month has passed). ??

Edit: they (my Uni) made offlineimap unusable, but it works with davmail.

I actually haven’t tried DavMail (but have heard about it), so if it manages to get around this sort of shenanigans I’ll happily give that a shot.

In my previous org I could also use offlineimap and msmtp to connect to their Microsoft mail server via standard protocols. But in this org I’ve so far tried the built-in Exchange support in Thunderbird as well as in Evolution Data Server based exchange clients (Evolution and KMail). All of them manage to connect to the server, kinda, but then I get an error message saying basically that my mail client is not approved and I’ll have to contact my admin to use it.

EDIT: I might add that the IT deliberately blocked non-Outlook mail clients a year ago or so, other Linux users told me that it worked fine before that. It’s supposedly a crackdown on people using shady third-party apps that they are concerned might exfiltrate data, but somehow they don’t allow exceptions even for reputable clients like Thunderbird.

You are repeating the very same history and excuses (from the IT Dpt.) I lived and heard. Davmail works for me on Linux and MacOS even from outside the intranet. Give it a try and I would be happy to help, I have a gmail account with my nick.

Edit: night time here now, sorry if I cannot help right away.

Thanks! Then I’ll give that a shot and email you if I get stuck :)

By the way, anyone have experience using emacs to analyse and visualise data (think spreadsheets and charts)? I’d really like to be able to use it to view any sort of data I happen to have.

My first thought was Emacs Speaks Statistics (ESS) to do this in Emacs via R, but I bet there are many alternatives.

Yes! Org-table is a spreadsheet. org-babel will run SQLite and gnuplot.

there is also https://gitlab.com/dto/cell-mode and https://www.gnu.org/software/emacs/manual/html_mono/ses.html... and https://github.com/ShuguangSun/ess-view-data

though I tend to stick with org

I use the `vnlog` and `feedgnuplot` shell tools HEAVILY to do data analysis and visualization. In emacs, these work well in shell-mode or in any buffer with `shell-command-on-region` (M-|). Not strictly emacs, but works great.

I read eMule and got nostalgic

I read MULE and got nostalgaic for the game, then remembered the same-named thing in emacs, which made me happy that nowadays we have unicode instead.

This another similar resource with some additional stuff about using mu4e-org:

https://stuff.sigvaldason.com/email.html

If you need to handle OAuth, I have a service to help with that: https://auth-email.com

Authorize once in a web dashboard, then use your accounts as if they didn’t need OAuth (ie. normal IMAP, POP3 or SMTP).

Reply on news.ycombinator.com