OfflineIMAP: sync and backup tool for IMAP | Hacker News Reader

OfflineIMAP: sync and backup tool for IMAP

github.com

117 pointsmatches-malone71 comments

I use OfflineIMAP to fetch & emacs/mu+mu4e+exim to read+index/search+send my E-Mail. It works, and overall I'm happy with it.

I used to use mbsync but when my employer migrated from Cyrus to Exchange IMAP I couldn't get mbsync to work with it, but OfflineIMAP worked just fine.

Now they've migrated to GMail, which works fine with both, and even though OfflineIMAP is more flexible & easier to customize I'll probably switch back at some point.

OfflineIMAP regularly pegs my CPU at >100% just downloading a few dozen E-Mails, whereas mbsync's CPU usage never went above 2-3%. When my laptop fan starts spinning it's usually OfflineIMAP in the background.

So if you're looking for an IMAP->Maildir retriever or backup solution consider trying out mbsync first.

It's a PAIN to switch between these programs, or even using the same data when switching between different servers (e.g. Cyrus -> Exchange). Inexplicably neither of them have some sort of reindexing mode where they can take an existing Maildir and reconstruct their local database from local/remote Message-Id's. They both use & maintain an opaque local database. That's why I haven't simply gone back to mbsync.

> So if you're looking for an IMAP->Maildir retriever or backup solution consider trying out mbsync first.

I'll second this: I used OfflineIMAP at first, but switched to mbsynrc after being unhappy with OfflineIMAP's performance. (It also seemed to hang semi-sporadically for me, which made it a pain to launch as a cron job.)

Lots of factors affect which tool is the better for a given user, of course, but I personally have been happier with mbsynrc.

There is the future version of offlineimap called https://github.com/OfflineIMAP/imapfw

Are you using the OfflineIMAP SQLite driver? If not, the maintainers did say it was more performant and reliable than using text files which was the default before version 7.

Yes, I use the SQLite driver, and use & regularly pull the latest version from the git, 7.1 right now.

I haven't dug into why it uses that CPU in any detail. Looking at its output / tcpdump of what it's doing it goes to >100% CPU for 10-30 seconds when it's finished fetching the status from the remote & is presumably in some tight inner loop comparing it with local data to decide what to do before it starts downloading messages, and then briefly spends a few more seconds at >100% after the messages are downloaded, presumably updating its local state.

When it's just issuing IMAP commands or downloading messages from the remote it sits at 1-5% CPU.

As I pointed out OfflineIMAP works just fine for me, but in retrospect I'd be happier if I had stayed with mbsync. I don't need any feature from OfflineIMAP that it doesn't have, and it both uses less CPU/Memory & it finishes deciding what to download & downloading it in way less time.

I have just checked how much it uses on my side: doesn't go above 50%, but I do consider that pretty high and can totally understand how it could use 100% with bigger mailboxes. For me, this was never a problem since I only run it once daily for backup.

Thank you for mentioning mbsync, I will keep it in mind.

I'm in a situation similar to the one you were in (mu4e, Exchange IMAP at work), and mbsync (version 1.2) works for me. Not sure what Exchange version is there, and I had to configure it a bit both initially and after an MS Exchange update -- but just some basic settings, such as `SSLType STARTTLS`, setting CertificateFile manually, `AuthMechs LOGIN`.

I like mu4e because it integrates with orgmode well. However, it is not the best client for managing Exchange meeting invites and plain text responses are not very readable in outlook. Were you able to solely rely on mu4e without using Outlook?

Yes, and I've actually never used Outlook with Exchange. But I'm only sending plain text responses, not trying to use any advanced features, and it works fine to receive and read meeting invites (just as messages).

What goes wrong with plain text responses in Outlook, by the way?

Outlook uses small fonts to render the response and there is a noticeable gap in readability between HTML formatted and plain text responses.

Thirteen years of thunderbirding - downloading all imap for local keeping, and with numerous migrations between servers - has not lost me a single email. In fact, I have all mail correspondance since the mid nineties - around a hundred thousand mails - neatly archived in Thunderbird folders.

For what it's worth.

If you have >100K emails, how do you know that nothing is missing?

It always says 100% complete when it runs /s

I have kept count.

Me too. There is one caveat, though: I've found TB's UI starts to struggle when dealing with folders with a large number of messages, particularly on Linux. It's not uncommon for the UI to freeze up entirely when I switch the view from one folder to another; usually this just means having to wait a second or two for TB to become responsive again, so it's not the end of the world, but it is annoying.

Yes, feet are dragging from around 10.000 items in a folder. A mild annoyance at worst.

Can Thunderbird do full-text search, or do you use an external program to index the folders?

Not parent, but yes. TB can do both an index-accelerated search, or a full-scan search.

Alternatively, since everything's downloaded locally as mboxes / maildirs, use grep / ag / ripgrep, etc.

It can. It's clunky, but it can.

I have Thunderbird installed on an ODROID-U2 board solely for searching my email archives via VNC when I'm not at home. It works very well.

Same here, only with many many clients, including Thunderbird. Oh, and POP

Oh, and POP, yes, in them olden days of yore.

I've successfully used imapsync (https://github.com/imapsync/imapsync and https://imapsync.lamiral.info/) in the past. It's been around for years, has tons of options, and has proven very stable, and is still maintained.

Does imapsync set INTERNALDATE correctly? I ask because naive IMAP uploading doesn't do this. (It's the date Gmail shows in the mailbox.)

Yup. Just used it last week to migrate my IMAP account to a new box. Used it years ago for a similar migration. Always works great.

This saved me at least 2 hours of awful fetchmail/getmail related work last month; Highly recommended.

fetchmail or getmail? I once used a very-slightly customized version of getmail (the Charles Cazabon program) in production to sync customers' GMail accounts into Salesforce.com, and it worked great (continues to work great, AFAIK).

The problem I have with ThunderBird is how poor it is an writing text only emails.

What is the difference between the GitHub version and the paid version on the website?

Binaries it seems (for mac/win)

I wanted to use an IMAP sync but after much experimenting with existing tools concluded that the protocol wasn't flexible enough for a proper two-way synchronisation.

I can't remember the exact reasoning now but several years on I am using dovecot's excellent dsync protocol over SSH to keep my laptop in sync with a private mail server, and have never looked back.

Combined with pine, exim and the excellent mairix (surprised not to see that mentioned yet) I enjoy a decent search on a fast MailDir implementation, with full offline support.

For those of you using GMail I'll repost a comment of mine [0] about gmvault [1]:

I have been using this for years and it's great software. One tip, store each different email account in it's own "database" my crontab looks like this:

    #Email backup
    0 0,6,12,18 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/joshstrange --resume josh@joshstrange.com >> ~/mail-backup-logs/joshstrange.com.log 2>&1
    0 1,7,13,19 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/otheremail --resume other@otheremail.com >> ~/mail-backup-logs/otheremail.com.log 2>&1
    ....

I sync it all locally to my house then back it up to Dropbox as well. The reason to store them in different databases is you cannot "filter" them out when restoring so if they all go to the same DB if you restore you are restoring ALL your email across all accounts to one new account.

[0] https://news.ycombinator.com/item?id=12987670

[1] http://gmvault.org/

> IMAP's main downside is that you have to trust your email provider to not lose your email. While certainly unlikely, it's not impossible. With OfflineIMAP, you can download your Mailboxes and make you own backups of your Maildir.

A professional mail host ('professional' as in 'someone that somehows makes money by hosting email') is way more unlikely to lose data than most of us are.

Making backup is great by the way, but for such task I'd strongly recommend the good old fetchmail or the newer (and better) fdm (https://github.com/ft/fdm). Both support imap(s). Both can save to either mailbox or maildir.

If you need a local copy, go for fdm and a local copy of your mailbox, plain and simple.

> A professional mail host ('professional' as in 'someone that somehows makes money by hosting email') is way more unlikely to lose data than most of us are.

I think their point is not that you're less likely to lose it yourself, it's that the probability that both you and the provider losing it at the same time is extremely unlikely. I.e., two points of failure required to lose your data.

The fdm project is dead. The Sourceforge project page returns 404, and the Github repository hasn't been updated since 2014. (And the Sourceforge mailing list returns 404 as well.)

The code probably works, but don't expect any help or updates.

I used it a couple of months ago and it worked perfectly.

Not all dead projects are actually dead. Some are just "completed".

A bigger worry with a professional mail host is they just cut your account for some arbitrary reason, and if that host is Google then good luck resolving the problem.

See also the older but somewhat more reliably and faster isync AKA mbsync.

http://isync.sourceforge.net/mbsync.html

My first thought was "older? really?" - but hosted at SourceForge, I guess it could be.

I started using offlineimap in 2008, which feels like a long time ago but of course is a lot more recent than the first appearance of imap.

The first version of isync (0.1) was released in 2000.

The first version of offlineimap (1.0) was released in 2002.

I remember using OfflineIMAP in 2002. It's pretty old.

Would anybody please like to recommend a good sql schema (that makes sense) for storing emails (possibly downloaded with this fine piece of software). Also a hint about a maildir -> sql import tool would be interesting to see.

If you have already made your experiences with storing email in a sql database, it would be very appreciated if you would like to write about the pitfalls to avoid and the problems that occured.

Thank you very much for your attention!

I'd have a look at dbmail.org (an Imap server with a rdbms backend). If I've understood correctly, after some years they landed on a fairly straightforward schema with appropriate indexes.

There's also "Archiveopteryx": http://aox.org/

Another question is what do you want to do? I've not played that much with sql store for mail - but notmuch (and mailutils) make good use of Xapian for quick index/search of mail, if that is your end goal.

This is good as a standalone IMAP client. We have done a IMAP Client Server, which will run multiple IMAP accounts which means this particular server will fetch/sync all the mails and monitor for new mails and flag changes. Almost works with all the IMAP email server without any issue.

Some stats: * We have more than 25K IMAP accounts running across 3 different 8core machine. * Application is written over our custom async web server and our nio tcp client. * Handling more than 50k connection per server instance.

Running a standalone client is much more easier because we have to concentrate only on syncing/fetching 1 account per instance. Rather in our IMAP Client Server we have to deal with multiple accounts with connection restrictions,failover handling, proper retry mechanism etc... We have done this backend engineering for one our product.

https://www.zoho.com/salesinbox/

Back when Fetchmail would segfault on certain spam messages, and a bug report to ESR went nowhere, I wrote my own mail fetcher in Perl. Then I extended it to IMAP, and later IMAPS, working around some bugs in some popular IMAP servers. I had it deliver mail to a local mda (so I could use Maildir and mda filters) and configurable to keep or delete mails from the remote server. I think it only supports one remote mailbox, however. https://github.com/psypete/public-bin/blob/public-bin/src/po... (It's very old code that sucks, and not secure, fwiw)

I guess fetchmail is more stable now. I've never seen any crashes, and I've been using it against IMAP servers for at least a decade.

I use imapfilter[0] to sort imap mails on the remote server and move older ones to a local dovecot instance for long term backup. That way I have current mail available on all my devices and access to the backup via mutt or via file in the MH format on the backup system.

That works, but you have to program a bit lua in order to setup imapfilter correctly.

[0] https://github.com/lefcha/imapfilter

If i backup my Macbook hard drive (thus including all Mail data/my imap mail account) does this backup include ALL the data stored at my imap mail server? Or would i be missing certain parts, like attachments or whatever?

Would OfflineIMAP download 'more' data than i already have backed up?

I just installed nylas mail to see whether it support well plain text emails.

It seems to me all this mail solutions requires that the password of the mail server must be somewhere visible in a poor plain text file which is a no-go for me.

It seems to me all this mail solutions requires that the password of the mail server must be somewhere visible in a poor plain text file which is a no-go for me.

There's an addon that lets you store Thunderbird's passwords with Gnome's keyring. That said, in the end you must either type it each time or it must be stored in a way that can be read in an automated way.

https://developer.pidgin.im/wiki/PlainTextPasswords

For backing up GMail, can anyone compare this to https://github.com/gaubert/gmvault ?

Oops, I just saw this after I post at the top level: https://news.ycombinator.com/item?id=14222691

I can't compare it but I have been using gmvault for quite a while and love it.

Also, is there a similar tool to migrate from MSN/Yahoo to Gmail?

There is.

https://github.com/imapsync/imapsync

Sad to see:

    Python v3.4+ [STALLED]

But looks like development is pretty active so, hopefully they finish making the jump to Python 3 soon!

I need this and real full text search. Who will give me full text search? ;)

https://notmuchmail.org/

http://www.djcbsoftware.nl/code/mu/

Another +1 for mu/mu4e. Been using it at work for three or four years. Having instant access to ~8 year old email is like a superpower.

question: how is mu/mu4e at handling multiple email accounts?

Delightful. There are several ways to deal with it. I do this:

- For READING mail, my offlineimap knows about multiple accounts and puts them in different directories on disk

- mu indexes the PARENT of all those directories, so as far as mu is concerned, there's a single mail store with ALL the mail. The m: filter can be used to select specific directories, which map to different accounts here

- For SENDING mail, mu4e knows about multiple contexts that are detected based on rules you can set (for instance, when replying to email, it'll set the context to whichever account the mail I'm replying to was sent). Or one can set the context explicitly with the ';' binding.

Works quite well

+1 for mu (and mu4e)

On Windows, Mailstore is fast, free, stable and works with IMAP, webmail & Outlook: http://www.mailstore.com/en/mailstore-home-email-archiving.a...

You can use recoll. It's based on Xapian and has a nice Qt-based GUI.

+1 for Recoll.

Link to the docs: http://www.lesbonscomptes.com/recoll/usermanual/usermanual.h...

If I use this, is the result different than from using Thunderbird?

OfflineIMAP is not a graphical email client. It is simply a tool with which you can either backup emails from your provider using IMAP or sync a local Maildir with your provider. The tool uses the Maildir email format. More information: https://en.wikipedia.org/wiki/Maildir

Works great with mutt!

i use a similar program named isync aka mbsync. it is written in c and i like it more than offlineimap!

I believe this is called POP3.

Error: No available formula with the name "imapfw" :-(

I don't think imapfw is widely adopted yet, it seems to be work in progress so most people probably opt for the more stable predecessor OfflineIMAP.

Reply on news.ycombinator.com