How to Back Up Gmail In Linux | Hacker News Reader

How to Back Up Gmail In Linux

mattcutts.com

50 pointsjsonscripter22 comments

Since this is probably posted as a followup the the Justin Lilly debacle, it should be mentioned that Lilly himself speculates that is was a mal-configured fetchmail script that caused the lock-out.

http://justinlilly.com/blog/2009/aug/07/google-account-suspe...

i dont like pop based backup options. i prefer doing an imap sync to local for which offlineimap has been doing an excellent job for me:

http://software.complete.org/software/projects/show/offlinei...

What are the specific advantages?

Some other nice things:

• Offlineimap uses maildir format, used by some IMAP servers like Dovecot. So if Google ever goes evil and locks you out, you can just use your own server instead.

• Theoretically you can use Offlineimap to reproduce exactly all your mail and folders/labels onto another IMAP server. This would be pretty useful for migration, but I've never actually tried it.

maildir works very well with mutt as well and because of the 2 way sync nature of offlineimap, you can work offline with all of your mail ( something that the gmail offline doesn't allow me, it only makes some available. ).

yes, i am a big offlineimap fan.

if i delete mail from gmail, it gets reflected in local copy. it has the negative, of if i lose everything from gmail and run a backup, it gets deleted from the local. but, i protect against that by having local backups. so my backups of local machine reflect the state of the gmail through time without growing and growing.

Or just use archive.py in libgmail (it asks for username, password, then you select 'all' folders).

Forgive me if this violates the RTFM principle, but I've researched this in the past without finding a concise answer. If someone has the answer at hand, it might save a lot of time and effort.

My understanding from past preliminary research is that IMAP is (or was, as of a year+ ago) a better option in terms of performance and convenience (e.g. no crapout after X number of messages fetched). However, I'd like to be able to remove messages from my gmail hosted mailbox image without losing them in my locally hosted mailbox image. The IMAP instructions I found were not clear (to me, at least) on how to achieve this. (It seemed possible / a feature at least nominally supported, but I had trouble identifying clearly defined instructions and implementations.)

Two questions: Is this indeed a possible/good/best approach? If so, does anyone have a pointer to a decent set of instructions?

IIRC, getmail and fetchmail both are focused on delivery of email and not with local management of emails. Using IMAP, to fetch emails in such a fashion doesn't really work. You would basically have to tell getmail/fetchmail to download only new emails... and even then you would end up with duplicate emails between runs.

On the other hand, Gmail's POP is non-standard. It records the last time that you pinged the POP server and only shows messages that were received after that time to the POP client talking to it. That's why it tells you the last time you fetched in the POP3 settings. Gmail's IMAP is more like a standard IMAP server and will show you all the email in the folder regardless of the last time that it was run.

To use IMAP effectively in the way that you want to, you would have to use something like offlineimap that syncs with IMAP, and somehow tell it to not delete emails that were fetched from the server. Or you could do full syncing with offlineimap and then do snapshot backups (rsync,rdiff,etc) of the gmail-backup folder. That way, you have the state of your Gmail account at different points. You could even have it do the snapshots as part of the syncing process (e.g. run the snapshot updater, then run offlineimap to sync new changes).

commercial online services to do this:

http://www.messagebunker.com/ (works very well) http://backupmymail.com/ (i helped make this) http://www.lifestreambackup.com (soon)

I guess it is risk spreading, to some degree, but using one commercial online service to back up another doesn't sound optimal for me. I like one backup of my data on a physical hard drive in my possession.

offlineimap + {mutt, dovecot}

- Two way sync

- All my other accounts (university etc..) are sync'd into gmail

It's worth mentioning that the usage of dovecot makes this configuration. I've had problems with using offlineimap + {mutt, maildir}. When offlineimap does maildir syncing it doesn't have the same information that it does if the maildirs are managed by an actual imap server.

Specifically, if I moved an email from my inbox to a tag/folder, offlineimap would end up just tossing it into the Trash. I think this was because offlineimap stores metadata in the file name of the emails in the maildir, and mutt doesn't preserve that when moving from one maildir to another. So offlineimap would delete when syncing the inbox, and then gmail would maybe block it when it tried to sync it to the specified tag/folder (?). It's worth noting that this was with the Gmail IMAP option of 'delete when all visible tags removed' was enabled, and 'All Mail' was hidden from IMAP (but that was because without doing that it was a PITA to delete emails, b/c I would have to remove them from the folder and from All Mail... not to mention that syncing All Mail means downloading all your emails twice).

I think the offlineimap option `realdelete' is related to your problem.

The nice advantage of using a desktop mail client is I can have it download all my email always. So I have a local backup of all my mail, just in case Gmail ever goes down.

the pain with that is the dissonance between folders and labels.

It's not that painful, but then again, I don't use labels that often, I just search my mail if I need something specific.

i was waiting for labels from email programs for so long. well, i had it once when i wrote my own mail client for the BeOS but since then, it was mostly frustration so for me, that dissonance is truly painful.

Why would you want to back up Gmail? I bet Google already backs it up, at different geographical locations even I dare say. Just in case... well, in case of what exactly? A fire? Earthquake? In case they decide to delete your mail account?

This must be one of those things that belong to the paranoid hacker stereotype.

In case Google decides to lock me out? There are plenty of stories of people who were locked out from various google accounts without warning and without being able to contact anyone about it. The probability may be small, but the impact of losing all your email, at least for me, is enough that some insurance is warranted.

In case you want to look up something critical (airline flight information?) when there happens to be a 30-minute GMail outage?

In case you want to take your data with you when you migrate to a different service?

In case you want an archive of your correspondence that will live on even if Google goes out of business decades from now?

In case you want to keep snapshots of your mail folders and be able to compare them, to see when a message arrived or was moved or deleted?

So you can access your entire mail archive when you're offline?

i'll give you two:

1. i can do local snapshots and go back and get mail that i've deleted etc. yes i actually like to delete rather than archive mail.

2. some people have actually had google lose their mail. i am one of those people, i had a gmail account 3 years ago that all the mail disappeared from. why? i don't know. maybe it was broken into and someone deleted it all. maybe google had an issue ( o lord forbid that would ever happen ) and my mail went poof. i contacted google and got nowhere.

so ya, maybe i'm paranoid. but i think of it as just being safe at this point.

Reply on news.ycombinator.com