Hacker News Reader

Comment by r3trohack3r | Hacker News Reader

Anecdotal, but with Google Photos, I've observed data loss to the point where I no longer trust the service with anything valuable.

Photos I know I've taken are missing. There are periods of time when I walked around a city 10+ years ago taking a large volume of photos, like when I first moved to Seattle. Going back to that window of time in Google Photos, I only have a handful of photos from that walk.

There are also partially corrupted photos from many years back. Photos that only partially render, or render as noise.

Luckily, all of these are random low-value photos from my youth. They aren't core memories of my kiddos growing up or anything. I'm glad I discovered the data loss in that window of time, and not by losing photos of my kiddos' birthdays.

View on HN

Tangentially related, if you've ever had the pleasure of migrating away from Google Photos/Drive with Google Takeout, prepare to spend some time fixing photo metadata in your library and re-embedding exif data with exiftool. Takeout strips out the exif data into a non-standard JSON sidecar, as opposed to something more standard/well supported like XMP.

It's unfortunately hostile for non-technical users that care about their photo metadata, which I assume is most people since it includes data such as creation timestamps and location. It's not too bad to script this if you're savvy (and careful), but otherwise you'll have to pay for a third-party tool[1]

[1]: https://metadatafixer.com/

> Tangentially related, if you've ever had the pleasure of migrating away from Google Photos/Drive with Google Takeout, prepare to spend some time fixing photo metadata in your library and re-embedding exif data with exiftool. Takeout strips out the exif data into a non-standard JSON sidecar, as opposed to something more standard/well supported like XMP.

I heard this claim before but I was never able to reproduce it. Does this only happen with the "Storage Saver" option enabled and/or when metadata has been manually changed? At least with the "Original Quality" option my takeout data seems to exactly match the files that were originally uploaded.

Worse than that videos shared with you are reencoded to 1080p with crap bitrate. IIRC the JSON sidecar is also missing for these. Doesn't matter if you save them to your library. You can watch them in original quality and see all of the metadata on photos.google.com but they don't get taken out.

I have backup both to Google Photos and iCloud.

The probability of both failing and/or both deciding to just delete my account because I look "suspicious" is very small. But I _have_ had iCloud mysteriously not being able to retreive old photos.

I have Photos set to download all the files locally, and those get backed up with Time Machine and Backblaze.

People forget even s3 will lose files. The reliability is high sure but it’s still sub 100%

Has S3 ever lost a file yet? I know that the reliability isn't 100% as that isn't possible in the real world. But I've never heard a report of a file that was confirmed as stored being lost, and I was under the impression that this was in the category "bound to happen eventually, but hasn't happened yet".

In December of 2012 AWS admitted that four files were lost: two completely deleted and two more truncated.

S3 is designed for 11 9s of durability. What this means is that they model a "correctly functioning system" (i.e. replication and/or erasure coding) and the durability guarantee is about HW failures only. However, such models do not (& really cannot) account for the existence of bugs or introduction of new ones. That's a huge part of why S3 doesn't really do a whole lot of feature development (well that + it's hard to maintain a 20 yold codebase).

Also, we're talking about Google Drive here which isn't GCS (Google's S3 competitor) but a higher-level product layered on top of GCS but with it's own book keeping / ACLs etc. Thus there's more room for error. My hunch is that the data is permanently lost.

Additionally, S3 stores an enormous amount of data such that probabilistically they're bound to lose something to HW failure. 2 years ago, S3 stored 100 trillion objects [1]. With 11 9s of durability annually you'd expect to lose 100 objects a year. The saving grace is that most objects aren't accessed (maybe not ever again) & they detect & correct durability errors on access to ensure that accessed objects definitely aren't lost. So while they "admit" to 4 objects, that's likely an under count because I wouldn't expect them to regularly check if all 100 trillion objects are accessible because of how long that would take.

[1] https://www.zdnet.com/article/aws-s3-storage-now-holds-over-...

Probably not in the way you're thinking.

But with cloud systems you can always lose your files when a hacker gets access to your root account and holds it hostage, or you lose the phone with the 2fa secret on, or your payment doesn't go through and you miss the reminders, or your account is wrongly closed for abuse by a malfunctioning bot.

s3 losing a file is a statistical anomaly and would get a lot more than a blog post and support deflection

My concern about my own backup strategy (which, for Photos, is exactly the same as yours) is that I don't have a system in place that notifies me if files get corrupted or randomly deleted, and I'm not sure how I would go about implementing something like that. Any ideas?

I use a synology for TM (raid 1) and it uses btrfs which has checksums to prevent bitrot. Either that or something running ZFS would be the obvious options:

https://kb.synology.com/en-us/DSM/tutorial/How_to_enable_fil...

Never had iCloud lose any of my data and I've been using it since not long after it launched, but now I'm a little concerned.

With photos I do keep them all local to my MacBook, but not documents. Maybe I should.

iCloud is not very reliable for me. It did mysteriously delete photos and files. I always use it just as a secondary backup since then, never as a primary backup

Worth noting that archived photos don't show up in most places, including the timeline. Not saying it's not buggy, but if you haven't already specifically checked the archived photos section, it's worth checking.

Good callout. Yes, I've checked and they are in fact missing, not just hidden.

I do a takeout quarterly and stash the data someplace safe. I've been burning the data to M-DISC blu-rays for a while now as a deep archive copy, they seem pretty reliable.

I also now have multiple apps syncing all the photos from my phone so they're stored in a couple different cloud photo locations on top of that.

I do this too, and I include par2 files on each disc. You sacrifice some space on each disc but the incremental cost of a few more discs is minimal.

I did not notice direct data loss yet, but I noticed other behavior that made me reconsider how much I trust it.

For example in the past I had several occurrences of duplicate photos where each of the two instances has exactly the same bytes as the other one. Usually that should have gotten deduplicated, but it hasn't. What's even stranger is that deleting one of the two pictures also deleted the other one. Re-uploading the original photo usually made both duplicates show up again. The way how I was able to get rid of those duplicates was to wait a day or so until I re-upload the data and to attempt that process several times. At some point only a single photo showed up.

Also those duplicates only showed up in the regular photo view. Even if both of the duplicate instances were added to an album viewing the album would only show the photo once.

For photos i backup using dropbox. Then my synology fetches the photos from dropbox.

Google photos only has the low quality photos.

The Google photos user this is really fucked up. I primarily use Google photos to manage my kids pictures as well what did you move to?

There really isn't any single service you can move to, to be "safe". This is why 3-2-1 backup schemes exist for important data, although they have morphed a bit with cloud usage. The basic idea is to be robust to any single point of failure (at minimum, but extra redundancy costs). So if your laptop or your google photos blows up, you still have everything.

If you have things you really care about not losing, you need a backup strategy.

The solution isn't to move. Every storage mechanism is going to have some amount of failures. The solution is redundancy. Store your valuable files on multiple medium. I know a lot of people that have both google photo and icloud - that works okay. You can also schedule backups to a NAS or some other cloud storage.

The failures being reported in the comments here are unacceptable. Better to have redundancy across services that are more reliable with better customer support. Choose companies that specialize in file storage, not a shitty advertising company (Mega, Dropbox).

https://mega.io

The quality of their software is vastly superior to Google Drive. They have mobile, desktop, and CLI apps on Windows/Apple/Linux. Been using them for a year without issues.

This company is owned by Kim Dotcom, isn't it? Are you ok with entrusting him anything of value?

It was started by him but he is no longer involved. Even if he was still running the company I would trust him infinitely more than Google or Microsoft. Also I like that they are based in New Zealand.

Their software is well engineered and is open source. https://github.com/meganz

You want to pursue uncorrelated redundancy more than you want a signular alternative.

People who came up in the cloud era were promised the false (impossible) idea that large conpanies could and would eradicate your risk of data loss if you just paid them a few dollars a month.

That's not a thing. If you really can't tolerate some loss, you need to coordinate something more robust across a few different and dissimilar options. And even then you'll still have risk.

That said, even just making sure you keep local copies of your archive or print your treasured favorites can go a long way towards greater reliability that without a lot of extra effort.

This is not for everyone, but I host my family photos myself, most recently with this: https://piwigo.org/. I have been doing this since 2007 (started on a different software, called "gallery". Was able to migrate from gallery2 to gallery3 and now piwigo), and so far no major issues. Advantage: I can easily share photos with family, no need for iCloud, Facebook, or indeed any service- they just need a web browser on their desktop computer or phone.

Personally I have turned off google photos and had turned off apple photos when I had an iphone for a couple of years.

Instead I use sync.com (yes, UI is horrible and it's slow, but it works) and my synology NAS that is mirroring to a synology NAS at my fathers house (we mirror each others photos).

Basic rule of thumb, if it's free they don't care.

Everything in my family dropbox - I make sure all of them can fit on both my and my wife's laptops. With backblaze backup. And a backup to a local external hard drive. I also keep a checksum of each file that I regularly... check.

I now back up valuable files to three regions in S3, three regions in Google Cloud, and offline physical storage I keep on a shelf in my office.

I have a weird gap on icloud, and the photos are on google photos thankfully, and other periods it’s opposite

they are both supposed to have all

dangerous

This has been happening to me across all google products. Emails - random emails are just gone. Sometimes I can see evidence of them from body of replies. Maps - I am a HUGE user of maps, starring thousands of places as I planned travel over the years. Randomly 1-50% of my stars just won't be there. Sometimes they come back. Photos - same issue as OP in this thread, random clusters of photos (mostly 10+ year old ones) just gone. There are a lot of support threads out there of people reporting the exact problem with no real fixes too.

I have the same with starred places on google maps. Sometimes they're there, other times they're MIA. Super annoying.

I noticed a while ago the UI would just show some photos for a day, and I'd have to click the down-arrow at the top right of the group of pictures for a date to get all the pictures of that date.

Anecdotally, I know a handful of my photos that were posted via Google+ are gone after that was turned down. I have been able to find all my other photos that were uploaded through Google Photos.

Reply on news.ycombinator.com