Hacker News Reader

Comment by ir77 | Hacker News Reader

i'm very curious about something -- people here rave about the takeout -- does anyone actually use the thing?

i've tried the takeout option about a year ago when i wanted to switch from google photos to apple photos and it was an absolute mess, it exported random zipped folders, all broken up (i think there were 14 of thema ll together), multiple images were duplicates, somehow it managed to corrupt apple format of pictures and i would have one picture with the same name that was something like 24KB and another one with 1.2MB, etc.

if it wasn't for the option (which they took away) of syncing photos to your google drive and copying them out that way, i have no idea what i'd have done, i'd have lost about 120GB of photos and videos going back to 2008.

so seriously, do all of you just use the takeout and store it away without actually opening up and trying to use it?

View on HN

I think breaking up is a ZIP limitation (max archive size is 4GB)

I'm exporting it regularly (although I certainly don't have 120Gb of photos on it). You can choose an option for regular exports (every two months), and delivery method (e.g. Google Drive). Then I have a script that runs daily, mounts google drive, moves the takeout locally if it's present and removes from google drive (so it doesn't take space).

Then indeed, inside you have mess with some data in HTML, some in JSON, etc. But well, at least you can parse it... I have a library which I'm using as an API to various data exports, in particular, archived takeouts too (so I don't even have to unpack them to access)

- https://github.com/karlicoss/HPI/blob/master/my/google/takeo...

- https://github.com/karlicoss/HPI/blob/master/my/location/goo...

- https://github.com/karlicoss/HPI/blob/master/my/media/youtub...

Described this in more detail here: https://beepb00p.xyz/my-data.html#takeout

Hmm, I've seen ZIP files over 4GiB - not from takeout but from others

The original specification was 4GB, although there is an extension to the specification that allows for over 4GB (64 bit is required).

That'd be ZIP64 -- see https://en.wikipedia.org/wiki/ZIP_(file_format)#ZIP64

"Then indeed, inside you have mess with some data in HTML, some in JSON, etc. But well, at least you can parse it..."

how is some regular schmuck that wants to move his data out of google to another service supposed to determine what they actually have to parse? the user simply uploads pictures into the system but gets garbage out?

the scarry part here is that google makes it extermely easy to suck in the data but for an average user it's extremly difficult to get back out and takeout is absolutely not a good solution.

They seem to use standardized formats where possible: vcard for contacts, mbox for email, image files for photos, etc. I'll grant you that they do some not-nice things (like separating photo metadata into json files), but I'm curious what format for search or timeline activity would be useful for a "regular schmuck"?

If said person wants to view the data on their own time, HTML seems adequate. And JSON seems ideal if they plan on sending this data to a new service that ostensibly supports parsing Google's takeout.

I think a big part of the problem is even if Takeout is using standard formats, none of their competing services or software platforms are set up to ingest those formats.

Like mbox is fine for opening in a desktop client, but if you move from Gmail to Fastmail or Outlook or whatever, mbox might as well be a ClarisWorks spreadsheet file.

I just finished found through the takeout process for photos last night. 39 2GB archives. Request took a couple of days to process, then I got an email with a bunch of links that they said were good for 7 days. I planned to load them into my Synology download manager, but the takeout system seems to rely heavily on browser state, so Synology couldn't download them. Each took 5 minutes to download at home over WiFi, and the takeout interface demands authentication every 10 minutes. It also reloads the page and reset my position in the download list on every click.

I've uploaded and expanded about half the archives on my Synology, and it's currently indexing everything, so I can't comment on the photo issues you've mentioned quite yet.

Overall, I'm happy there exists a mechanism to get my photos, but the quality of the experience is truly awful.

Almost like they made the ux really hard to use for a reason, eh?

All that engineering talent and that's the best solution? Lol.

More like, they're just wanting to make sure they're legally covered. There is no added benefit for the company in making that experience better.

Source: working on a product engineering team for a larger company. Why would we upheave our roadmap for something that doesn't help provide explicit value to our product offerings? We've got a list a mile long of improvements and new capabilities to deliver on, and no engineer likes working through legal hoops anyways.

I don't see how your statement contradicts your parent comment's. Looks like Google just did the bare minimum to cover themselves legally (as you said). But that shouldn't mean that they can't make the experience better.

As for no engineer liking working through legal hoops, my observations disagree. Many absolutely don't care. Give them a well-described Jira ticket and they'll happily chip away at it for a year if it's necessary.

I exported my data this week, and was given the option of either many zip files (default) or a single 7z file.

I downloaded the 7z without any problems.

My brother passed away very suddenly a couple of years ago. Google Takeout and similar services were a godsend for extracting and archiving everything from his accounts. As far as I know, everything exported fine, thiugh he didn’t use Google Photos.

Gzipped tarballs are an option now with what looks like a 40GB max.

  $ ls -lh takeout*
  -rw-r--r--  1 ben  ben    39G Sep 26 18:29 takeout-20200925T172738Z-001.tgz
  -rw-r--r--  1 ben  ben    38G Sep 26 19:20 takeout-20200925T172738Z-002.tgz
  -rw-r--r--  1 ben  ben    35G Sep 26 20:06 takeout-20200925T172738Z-003.tgz

The limit is 50gb on tgz takeouts.

I use this one every other month -- download everything and throw it in cold storage. Mainly keeping it for the scary possibility of being kicked out of my account.

What do you use for cold storage? I've had bad luck with external hard drives dying over time.

I just used it to pull over 350gigs of photos. Not fun. They really make it hard to reimport photos elsewhere. I wish their app just had a download all.

Use the tgz option for exporting, the zip format has all sorts of limitations and weirdness.

As I wrote elsewhere, previously. I agree it's a mess, but not the end of the world. Extract the file -not- preserving directory structure(-j flag). Now you just have everything in a single folder. Then delete the metadata files by file extension. This took all of a few seconds to do for each zip.

Shameless selfplug: As part of ongoing privacy research me and colleagues developed a website that parses personal data exports/takeouts from google, twitter, instagram and facebook and visualizes the data in a treemap and a timeline. We aim toincrease the awareness of personal data and the effects of online behaviour.

The data is not uploaded and instead parsed in the browser.

For the Google takeout make sure not to include data from Photos, Gmail, Youtube and Drive as they make the export too big. Also select "JSON" for "My Activity".

Feel free to try it out yourself:

http://transparency-vis.vx.igd.fraunhofer.de/visualization

I do a full takeout every month or so, both for myself and also my parents. It's largely so that we don't lose anything if Google does one of its awesome sudden account disablings.

Used it to export all my google plus social stuff (several gb over the years) before they shut it down. It kinda worked okay.

Exports as tar instead of zip.

Every photo has an additional json with some kind of metadata.

For me it was fine.

I think people love the idea of takeout more than the thing itself.

Reply on news.ycombinator.com