Guide to email in Emacs using mu and mu4e | Hacker News Reader

Guide to email in Emacs using mu and mu4e

cachestocaches.com

167 pointsgjstein71 comments

Maybe I'm missing something obvious here, but calling "mu index" from the offlineimap postsynchook is going to suck, because mu4e invokes "mu server"[1], that aquires a Xapian lock on the index, which means you can't run the "mu index" while reading your E-Mail.

This is the worst thing about mu/mu4e. You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI. After trying to get it to work in some automated way I gave up. I manually issue a key-combo to re-index and then read my E-Mail.

Another comment: I recommend not using Emacs to send E-Mail, but to outsource this to a local smtpd such as Exim (which I use with mu4e on Debian). Then if you don't have WiFi or whatever the smtpd just queues those up for later sending, and has queue inspection tools etc. All of the solutions for doing this in Emacs itself are dirty hacks full of edge cases and bugs smtpds have solved decades ago.

Despite some rough edges, mu+mu4 (plus Exim, or similar). is an amazing tool, and the most useful E-Mail client I've ever used.

1. https://www.djcbsoftware.nl/code/mu/mu4e/mu-server.html

The locking thing has not turned out to be a really serious problem for me, and I have several gigabytes of email. mu index is really fast. Also, using the Emacs sendmail hasn't been a real problem. It turns out I don't need to be productive during the ~1s it takes to send the email and I have not run into any edge cases or bugs in the ~5 years I've been rocking this setup.

It doesn't matter how fast the index is, only one process can hold the Xapian lock, if I'm reading my E-Mail in Emacs with mu4e (which invokes "mu server") and try to index on the command-line I get:

    $ mu index [...]
    mu: mu_store_new_writable: xapian
    error 'Unable to get write lock on
    <Maildir>/xapian: already locked' (11)

How do you send E-Mail with no network connection? Last I checked this was some ad-hoc thing where you neede dto queue them up in some queue managed by Emacs, and remember to turn that mode off when you were back on WiFi (or hack up your own hooks to automate this), v.s. sending it to a local smtpd which already has retry logic built-in.

Why on earth would I try and index it twice concurrently?

I never have a scenario where I need to send email without network.

No, you wouldn't index from two processes concurrently, but for Emacs to read your mail in mu4e and for you to concurrently issue "mu index" on the command-line two processes ("mu server" and "mu index") would need to poen the same Xapian database read-write at the same time, and just like SQLite Xapian doesn't support that.

... I still don't see why I would do that. Emacs runs offlineimap and then Emacs runs mu index. Why would I do it again myself from the command line?

Yeah I use notmuch with Emacs and it's the same. If you try to search, refresh search results, or read email while the index process is running you get an error. It's not a big deal for me, because I download and index my mail from the command line, so I'll never be reading or sending email at the same time.

This used to be true but notmuch has added support for Xapian's DB_RETRY_LOCK feature a couple of years ago so if you have non-ancient notmuch (0.23+) and Xapian (1.3.2+) then the second process will wait for the lock instead of erroring out. The feature can be disabled at compile time if you don't want it for some reason.

> You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI.

It's worth noting that multithreading recently landed in Emacs master, so it's now possible to avoid this sort of problem.

Now possible as in mu4e would need to use some API that's not in a release version of Emacs, or is it more transparent than that?

The former. I wasn't suggesting that mu4e should be converted right away to use this feature, but more concerned that an unqualified statement that long-running IO must block in Emacs would create the wrong impression.

You can configure mu4e to queue mail to send later:

https://www.djcbsoftware.nl/code/mu/mu4e/Queuing-mail.html

Yeah, that's the dirty hack I was referring to. I don't need to tell an smtpd "now's when I expect not to have an Internet connection", which this method relies on, and could happen as I walk into another room and lose WiFi. I just give it mail to deliver, and if it doesn't work it keeps retrying until it works (with configurable backoff and a setting saying when to give up for good).

Yup, I see where you're coming from. I've actually gotten used to mu4e's queuing behavior, and just leave it on all the time. For one thing, I like having a chance to edit/delete mail after hitting send (sort of like gmail's "undo"), and for another, I like the satisfaction of having a dozen or so messages sent all at once at the end of an email session.

The locking is only a problem because he's doing it wrong, I think. I'm doing this:

    http://notes.secretsauce.net/notes/2014/12/12_mu4e-and-desktop-notifications.html

I.e. there's an external script that invokes IMAP IDLE. And when IMAP tells it there's new mail, the script tells EMACS there's new mail. Then the reindexing happens within the emacs, and it uses the lock it already has. Makes this particular issue go away completely.

That must be it. I also trigger mu4e indexing using emacsclient, after IMAP sync. BTW you do not need to have mu4e open to index mail (and single-window mu4e mode will not have a * mu4e-main * buffer), it is enough to (progn (require 'mu4e) (mu4e-update-mail-and-index)).

I must also be missing something because I never explicitly index my mail. I fetch it with mbsync every two minutes and then use mu4e to read it and new mail shows up as one would expect.

¯\_(ツ)_/¯

How many tens of GB of mail do you have? I think this problem only tends to show up on big mailboxes.

10-25GB. But a fast computer, so?

Good for you. To save power and use smaller machines should always be an option. Indexing saves much resources here.

Agreed 100%. I'm just stating my case, not denying others'.

> Maybe I'm missing something obvious here, but calling "mu index" from the offlineimap postsynchook is going to suck, because mu4e invokes "mu server"[1], that aquires a Xapian lock on the index, which means you can't run the "mu index" while reading your E-Mail.

> This is the worst thing about mu/mu4e. You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI. After trying to get it to work in some automated way I gave up. I manually issue a key-combo to re-index and then read my E-Mail.

I tried both approaches (actually, I had mu4e reindex on `g` query refresh, no need for separate keybindings), and prefer having indexing run after IMAP sync. I don't notice any problems (and I mostly use very slow Atom hardware).

> Another comment: I recommend not using Emacs to send E-Mail, but to outsource this to a local smtpd such as Exim (which I use with mu4e on Debian). Then if you don't have WiFi or whatever the smtpd just queues those up for later sending, and has queue inspection tools etc. All of the solutions for doing this in Emacs itself are dirty hacks full of edge cases and bugs smtpds have solved decades ago.

FUD. smtpmail.el can do this. smtpmail.el can queue mail, and you can run it in a separate Emacs process periodically or on demand, to attempt to deliver those emails. The emails are queued as files in a directory, so you don't need separate queue inspection tools. I run smtpmail.el this way with mu4e, and think it is a far superior solution to having to install and configure yet another external program.

I use imapnotify to connect to my imap server and respond only when a mail is received by running offlineimap to sync the mail and telling mu4e via emacsclient to sync. Since mu is never ever called except by emacs and only exactly when an email has come in there is never contention for the lock.

I recommend notmuch in place of mu/mu4e and msmtp in place of a full-blown smtpd implementation.

I also recommend isync's mbsync over offlineimap

Agree with using Isync. Notmuch lacks the backward synchronization to the IMAP server and to another mail client then notmuch.

I primarily use mu4e but want to see changes on my mobile as well as on any remote computer using a web client in a browser.

As far as I know notmuch can just be synchronized to another notmuch instance.

If you use https://github.com/gauteh/gmailieer you can have bidirectional sync between gmail tags and notmuch tags.

Personally, I don't bother with it. When I have to do mail on my phone, it's usually so limited in scope that I don't need to retain state when I sit down at my main computer. I can imagine that's not for everyone, though.

My mail fetching script kills mu before calling "mu index", like so :

$ pgrep mu && kill `pgrep mu`

$ mu index

It works perfectly. Mu4e then restarts mu when it needs it.

I think I saw that as the recommended setup somewhere in the mu docs.

> $ pgrep mu && kill `pgrep mu`

This could be rewritten as:

$ pkill mu

With `pkill $(pgrep mu)` or even with `pkill mu` you can kill mutt and any mu$ process as well.

`pkill -x mu` matches the whole name without substrings.

For anyone thinking about a setup like this: skip offlineimap and use mbsync instead. It's much faster, and syncing (rather than indexing) mail is the main speed bottleneck to reading email in emacs.

Could you share your mbsync config? Every time I have set it up it only pulls emails down. It doesn't sync back with the imap server for emails I have read or deleted.

`doveadm sync` is also a nice option if the server supports it: combined with a custom `sendmail` command, one can both send and receive/synchronize messages over SSH, avoiding additional authentication.

mu/mu4e author here... please let me know if you have any specific questions.

No questions. Just a "thanks" - mu4e is great.

As you are here: there is a constant issue with address autocompletion on osx... both emacs25 and 26...

But on the other hand. great job and THANKS.

For rendering HTML emails I use either shr or w3m. To get it more readable I make the text area smaller with writeroom-mode/olivetti-mode in the mu4e-view-mode-hook. But both modes apply just after opening an email a second time.

Do you have any hint?

Is there any Emacs-based solution to access a GMail account with 2FA, without enabling "insecure access" on Gmail?

I've looked at a lot of solutions, and AFAICT:

- SMTP + app-based passwords require "insecure access" being enabled

-Using the GMail HTTP API does not require insecure access, but none of the standard CLI email tools support it.

I'm going to break the HN code of ethics (oh gosh) and explicitly bump this. All of the solutions I have found just point at the insecure access option.

As far as I can tell, when 2FA is enabled app passwords are the only non-2FA way to use your account. I have an app password in my .authinfo.gpg that I use for authorizing offlineimap and SMTP through Emacs. Obviously this isn't as secure as 2FA, but since the password is PGP encrypted I don't worry about it ever being comprised.

Just looked at my account. There is no option for insecure access when you activate 2FA.

From https://myaccount.google.com/lesssecureapps

"This setting is not available for accounts with 2-step verification enabled. Such accounts require an app-specific password to access less secure apps"

Can you use XOAUTH2? https://developers.google.com/gmail/imap/xoauth2-protocol

offlineimap supports it, and the config file has setup instructions: https://github.com/OfflineIMAP/offlineimap/blob/master/offli...

It may be possible with Thunderbird and Maildir. There is an issue: https://emacs.stackexchange.com/questions/38412/read-and-edi...

How does this compare to using notmuch-emacs? And can someone say whether notmuch or mu is preferable these days?

I don't have a lot of experience with mu, but I went with notmuch and have really liked it. It reminds me a lot of the standard gmail interface.

I have extensively used both, and spent some time using Gnus too. Plus a lot of time on Mutt.

In my opinion, Notmuch is better than Mu because it has a really neat architecture that allows customizing it to embrace any possible email workflow with a few trivial Bash commands.

Notmuch is purely tag based. It will never ever modify your mailbox. It's your task to map tags to imperative mailbox actions (delete, move...) using some shell hooks. Most workflows can be trivially implemented by calling the notmuch backend and piping the output to mv or rm using xargs or parallel.

Since searching in Notmuch is really quick, I recommend never manually assigning tags and tagging everything with rules written in the backend for those repetitive searches that you want a shortcut for. This is not specific to Notmuch, but also applies to Mu.

Mu has a hybrid tag/folder model, and the interface resembles Mutt in some ways. The other key difference with Notmuch is that Mu stores tags using a special header in each message. Notmuch uses a separate database, which is not synced across computers (unless you rsync it explicitly). I prefer the latter approach, as modifying your mailbox to store tags is a bit ugly and slow. I also favor not tagging things manually, but using rules. Then syncing tags becomes unnecessary.

Thanks for the comparison. I'm wondering what the benefits of mu4e are over notmuch (your comment doesn't list any - which makes sense because you preferred notmuch).

The reason I ask: When I look around, I tend to see a lot more references to mu4e than to notmuch. Is it just easier to start with, or does it actually have some nicer features that notmuch doesn't?

Not sure if it's relevant, but: I download all emails and do not want to sync with the server.

A case for mu: it is NOT tag-based, which is a feature: it can work with normal IMAP, and thus other email clients that aren't mu can look at that mailbox, and have all the data. Notmuch's tags do NOT live in IMAP (as far as I know), so if your workflow relies on them, you get annoyed if you ever look at your mailbox in any other way.

With mu, you just do the normal thing: instead of tagging a message as "x", you put it in folder "x". Clearly this breaks down if you want to apply more than one tag to a message, but this hasn't been an issue for me in the 20+ years I've been using email.

Before I read this, I had no idea that mu even did any tag-based anything.

That's not true. You can still have Notmuch to play well with other mail clients. You simply need to work out a good mapping between tags and folders, which is trivial.

IMAP only supports a few tags: read, unread... But not custom ones.

The only possible scenario where Mu is better than Notmuch is where you have many many custom tags and you want them to be perfectly synced across Emacs instances in different computers. Since Mu stores them in messages, this is very easy. With Notmuch, you'll need to separately sync the database. E.g. by rsyncing it.

But note that other clients won't understand these tags.

People typically use Mu because Notmuch was not very well documented. So the typical newbie came to Notmuch, found no way to delete a message and got frustrated.

I think there is more or less feature parity.

However, out of the Emacs world, Notmuch is very very popular as a Mutt search backend.

Does notmuch handle calendar invite indexing and tracking?

I've been using mu4e for a while, and despite some rough edges (e.g., I have the same dissatisfaction with timers and xapian lock that others have), I've been pretty happy with the set-up. I did find that for working with gmail, mbsync works more consistently than offlineimap.

I found trying to do this sort of thing (e.g. gnus + dovecot) a very helpful introduction to software engineering; once I got a job as a software engineer I found I didn't have time for a local setup that needed so much attention.

It needs no attention once you set it up. And the first steup is a bunch of copy-pastes and minor customization.

No, I'm speaking from experience unfortunately! Things break, emacs changes, your OS changes, you decide you need a more recent version of something in order to get a certain feature or bug fix. You get a new laptop. Etc.

I agree. I’ve learned immensely from using tools like mu and offlineimap, writing utilities for a maildir, and it can be great fun...but I need direct and unmediated access to email, I need it to ‘just work’ and let me copy-paste html text and images and drag and drop attachments. I have to use email to get other stuff done, I can’t spend time focusing on email itself.

Yeah, I see what you mean. I tend to agree.

I guess exploring around with setups like this is part of the command-line-and-lots-of-email-subculture's current zeitgest. I hadn't heard of mu as a CLI tool before, though I knew that mu4e was one of the emacs mail-reading options.

I'm currently playing around with mblaze, which is another command line mail-reading toolkit (https://github.com/chneukirchen/mblaze ), and building some notmuch-friendly tools on top of that ( https://github.com/dannyob/mblaze-much ). I'm not sure it's the right way forward, but it's proving to be a fun experiment.

Not completely related, but I have a question regarding Emacs and email:

Is there a way to read/write emails on Emacs, without having any emails or credentials stored locally as plaintext? I'm heavily prioritizing ease of configuration and security over speed or other additional features.

Sure, you sync them locally with e.g. offlineimap, and specify a "remotepasseval" key in your config which can invoke a command that invokes whatever password manager you use, same for sending.

I don't do this and just keep my password in plain-text and use full-disk-encryption, because I don't want to have to be constantly unlocking a keyring to have some background E-Mail syncing job work.

If you save the passwords in a gpg-encrypted file, emacs primpts you for the password to the file when needed.

Otherwise, password-read or something like that.

I don't know if the others use it but Gnus can read credentials from an .authinfo.gpg file.

https://www.gnu.org/software/emacs/manual/html_node/auth/Gnu...

That's what I do actually: https://www.emacswiki.org/emacs/GnusAuthinfo

When I set it up with gmail tough, I think you either need a 2fa token (which requires 2fa on gmail) or tick "allow unsafe apps" in the admin panel, otherwise you get rejected by gmail. I'm sorry I don't have much info, it's already been a few months I've set it up. However, if anyone's struggling, I'll be glad to help ;)

Sure. Although the documentation doesnt make it explicit your .authinfo file can have a gpg extension and Emacs will encrypt and decrypt it for you using gpg automatically

The actual reading of email with mu is about what's on disk. Using offlineimap is just a common configuration, and in its configuration you can specify a function, so if you wanted it to prompt you for a password whenever you get mail, you could do that. You could also just have Postfix deliver mail to your directory, then just have mu4e run the mu index command periodically to find new mail.

As for the writing of mail, it culminates in running some kind of sendmail program on your message. If you have sendmail or postfix setup locally to forward mail up to another server, you already don't need to have a password set up in Emacs. But you could also make that program do whatever you want, it just needs to be a binary that accepts certain arguments (or an emacs function that operates on the file).

I wrote a small program do exactly this on OSX and Linux (Gnome):

https://github.com/thejefflarson/getpasswd

It accesses the OS keychain to get the right password. I don't use offlineimap -- I use mbsync instead which has a PassCmd option where you can pass in a shell command (e.g. getpasswd <imap-server> <imap-user>). And another tool to add passwords to the GNOME Keychain:

https://github.com/thejefflarson/setpasswd

Adding passwords to the OSX keychain is easy enough, but I found it fiddly in Gnome's GUI. YMMV.

FWIW, here's an alternative setup to consider if you don't want everything to do in Emacs (I use it daily for its wonderful Org-Mode):

* OfflineIMAP -- to download and synchronize local Maildir with remote IMAP.

* Mutt -- E-mail client (understatement).

* Notmuch -- for fast local indexing and querying.

* Postfix -- the wonderful mail transfer agent.

Here[1] are some config files (from my colleague, Paul Frields) that you can adapt from.

Been using the above set up for about five years. They heavily help with the high-traffic open source projects that use mailing lists for patches. (Even for general mail setup, this is still useful).

These tools are a part of my day-to-day happiness at work.

[1] https://pfrields.fedorapeople.org/configs/

If you're using notmuch and Gmail, I'd recommend taking a look at gmailieer - https://github.com/gauteh/gmailieer. It uses the Gmail API to fetch mail into a local maildir and keeps the tags synced bidirectionally.

The downside is that for whatever reason the Gmail API doesn't support the mute label, so having that tag sync is not possible. (If a Googler reads this, please fix that! For others, there's already an issue filed as referenced in the gmailieer readme.)

I'm really happy with my e-mail setup of offlineimap, afew, notmuch, bower.

http://www.offlineimap.org/

https://github.com/afewmail/afew

https://github.com/wangp/bower

I used to use gnus and liked it. I don't think I can use this at work because I need the exchange invitation crap which thunderbird thankfully supports. I just can't be bothered to put significant effort into making that work. But I think I should definitely look into this for home.

I use Fastmail and the interface is so good that I can say this solution is really crappy.

I use Fastmail and think their web interface is great, but I still also love having offline access to my mail[1], and being able to create an org-mode TODO from an email with a link back to the message, and being able to hit a key to open a new message while coding. Also, every day I get an email with some log files from a service at work; because `mbsync` puts these in my Maildir, I can run a script (using `ripmime`) to extract the logs (going all the way back in time) and show me a report etc., which is so much faster than if I were to get those attachments from a GUI.

[1] I'm still spooked from the time my Gmail account was suspended for no apparent reason, having to contact support and being without email for nearly a week.

Reply on news.ycombinator.com