Hacker News Reader

Comment by magicalist | Hacker News Reader

I'm still hoping they'll someday announce Drive (the local file sync part) as the universal Google Data Liberation mechanism.

For instance, it would be awesome if with this change you could click a button that says "also sync my gmail with drive" and you'd get a local backup of your email in your drive folder. Many people wouldn't want to waste the GB of local space for the backup, and I don't know what format they should actually use, but having the option would be amazing.

Eventually it should be that all your google data should go through Drive and by syncable. Imagine the ease on people's mind -- no matter if all the Google datacenters were struck by lightning or your account alone was disabled for some reason, good or bad -- if you already had your data saved locally with no more than clicking a checkbox.

View on HN

It's already pretty trivial to set up a standard IMAP client and sync all your mail to your desktop. Some clients even support GMail's labels so you don't have to deal with that. (Each "label" shows up as a different IMAP folder, so normally you end up with a copy of each email for each label you give it.)

> It's already pretty trivial to set up a standard IMAP client and sync all your mail to your desktop.

No, not by any means. What magicalist is describing is "pretty trivial". What you are describing is a big pain in the ass.

gmvault.org or if you hate the command line, most desktop mail clients (Apple Mail, Mozilla Thunderbird).

What, just setting up an email client? Most mainstream apps now have presets for major services so you don't have to pick port numbers or figure out what authentication methods to use. Just type your address and it looks up the server settings for you, one click and you're done.

Where is the pain in the ass? Launching the client, clicking on the Google icon and entering your username/password? Is it really so difficult?

Currently have over 90,000 emails (~9 GB worth) archived in my GMail account. Any idea how long it would take to transfer it all to my desktop via IMAP?

It'll take a little over 2 hours on a 10 mbit line.

Unscientifically I've seen between 1-3 messages per second via imapsync so ~a day. The bottleneck seems to be the IMAP protocol or rate limiting on google's side, I've certainly never seen it approach wire speed.

Google has published total "bandwidth limits" where they say[1] "To address migration needs, you can download your entire mailbox approximately 3 times per month."

[1] https://support.google.com/a/bin/answer.py?hl=en&answer=...

Those limits only seem to apply to Google Apps for Business accounts.

I believe it also applies for anyone grandfathered into the free Google Apps standard. I'd like to know the limits on free accounts if that is not true.

Furthermore, you have to jump through several difficult-to-recognize hoops to get that 9GB on your disk. IMAP will only sync what the sever feels are recent emails, and envelope information for the rest.

offlineimap will do it without hoops.

OfflineIMAP crashes a lot. I have been using it for many years and with each upgrade, I'm optimistic that stability and/or performance will improve, but it still sucks a huge amount of CPU and hangs several times per day.

I haven't had a problem. Have you reported the crashes and hangs?

Depends on your email client. Thunderbird downloaded full copies of two of my Gmail accounts when I added them and selected the "Keep messages for this account on this computer" option.

See also: https://github.com/rgrove/larch - "Larch is a tool to copy messages from one IMAP server to another quickly and safely. It’s smart enough not to copy messages that already exist on the destination and robust enough to deal with interruptions caused by flaky connections or misbehaving servers."

If you are capable of storing a backup more safely than the countless google datacenters, I would assume setting up an SMTP server is the easy part.

We are talking about migration plan here - eg: moving from gmail to hotmail without losing your data, or better transferring them to your new provider.

In any case, your data may be physically safe in Google datacenter but you are still one Google policy breach away from losing your access to it.

Yes, I should clarify that I think it's great that IMAP support exists, that you can certainly backup your email relatively easily today, and that the Data Liberation/Takeout thing is great in general. I'm using email sync as the example here, but the benefits would really come from a universal (or more universal than today) sync, not just an easy email sync.

I just think Drive would be a great way to unify backup methods across the Data Liberation options. Gmail backup is already relatively easy, but some of the others are less so, and being able to turn them all on with a few clicks (rather than remembering to visit the web interface and exporting every few weeks) would seem to be a better solution. Ideally IMAP support would continue, and, even more ideally, the hypothetical Drive backup format for a gmail account would be some typical email client save format for IMAP accounts so you could (at least somewhat) transparently open the backup in a major email client.

All of this is certainly not trivial, and less so now that I thought through the above. If you treat the local "backup" as a first-class version of your mail by actually opening it in a client and then altering emails in there, drive could then in theory sync it back with the gmail account just with IMAP.

Some account data you'd probably have to specify is a read-only backup and not as read/write. For instance, exporting your Reader data gives you a whole json format export, as does exporting your g+ data, and I'm sure other products as well. I'd imagine it wouldn't be worth the extra engineering effort (and possibly a recipe for disaster) to support random changes to that json being synced back up to the cloud (if it was even data you had permissions to change in the first place...for instance, if you edited the text of a post you commented on, but it wasn't your post).

You could probably get around those by just explicitly labeling the subfolders as "Google+ backup" or something, so there's no expectation of changes there being synced back to your account. Should it overwrite any changes you made locally next time it syncs, though?

Luckily I don't have to figure those things out. Still a good idea though :)

Or if you had a network outage and wanted to access your files.

I use Mail.app to sync my gmail locally, but it'd be nice if Drive actually synced my Google Docs locally. Currently Drive just places a small json shim file on the disk, rather than syncing a usable document.

Data Liberation does export your docs in a usable format, but it's a rather awkward, manual process.

Currently Drive just places a small json shim file on the disk, rather than syncing a usable document.

If you use Chrome its easy to get Drive to store docs locally (but not images pdfs etc) [1]

[1] http://support.google.com/drive/bin/answer.py?hl=en&answ...

Chrome will only store a few items locally, and often you still need a network connection.

If you use Syncdocs [1], it syncs all docs locally as Word or Open Office files. It also syncs other files and Google calendars.

[1] http://syncdocs.com

I use insync[1], it also converts all documents, but why I use it mostly is it's linux support.

An advantage to Syncdocs is it's one time payment, instead of a subscription fee.

[1]https://www.insynchq.com/

If you want your actual Google Docs files synced locally, I recommend SocialFolders. It runs in the system tray as an unobtrusive service. It also syncs Twitter images and some other services. The free version has worked for me as I don't have many thousands of files in Docs.

https://socialfolders.me/r/k1pb5 (referral link; gives both you and me a higher file limit.)

https://socialfolders.me (no referral link; normal file limit.)

Or how 'bout this: you click "also sync my photos with drive" and then you enable the same option in Flickr and iCloud, and now you can use any of the three photo services to organize your albums, and people can use any of the three services to view them, and it's always the same fully-synchronized data for everyone.

I also want to be able to drag my iPhoto Library folder into Google Drive and have it get magically parsed and turned into G+ photo albums -- even if this is just a one-way process.

Can't you do the one-way import with Picasa?

Yeah, exactly, I don't understand why they don't make "Photos" a folder in Google Drive and vice-versa. Subfolders can be albums, so when you upload something from your phone to G+ it also shows up on Google Drive, and when you drag a photo in your Photos folder it gets uploaded to your G+ account.

I don't really care that much, though, since there's no Drive client for linux (how hard can it be to port the thing?).

There is a Drive client for linux: insync. I use it since more than a year and it works perfectly.

Oh hum, thanks for that. Does it use inotify/is light on resources?

It's actually pretty well done, I used to use it til I moved to ownCloud. Interestingly it's more stable than the official client on my Mac, on Ubuntu 12.04

It's nice that there's a third party option (yay for open APIs) but it's pretty weird that Google still hasn't bothered to make a Linux Drive client themselves. This is basically the only reason I'm still using Dropbox primarily, though I'll probably start moving some files to Drive when the bonus space I got from their college competition thing expires.

Do you know how it compares with fuse-google-drive?

>I care that much, though, since there's no Drive client for linux (how hard can it be to port the thing?).

Not hard, it has been available forever internally in Google. I guess the hard part is supporting it externally.

Use CloudPull ($10) from http://goldenhillsoftware.com and it's by far the best Google backup tool I've found. It supports Gmail, Google Contacts, Google Calendar, Google Drive/Docs, and Google Reader. The app backs up a max of 10 accounts in the background and all the data is available in standard file formats.

If you want true sync then use insync ($10/account) from https://www.insynchq.com. It gives you two-way sync of your Google Drive. You can edit your document in Office offline and it syncs to your Drive. It works very well.

I've been using Backupify (https://www.backupify.com) to accomplish just that. Actually would prefer the backup data to be elsewhere, not with Google.

Reply on news.ycombinator.com