Users reporting artifacts appearing in old images stored in Google Photos
support.google.com
support.google.com
But because it's Google I'm sure they can break whatever rules they want as long as it doesn't involve trying to avoid Apple's 30% fees.
Trusting a dubious central authority up until you get burned is a bad heuristic.
Just location tracking. The lies. Over, and over, and over, and over again the lies.
I see no reason not to believe that Page and Brin sincerely desired Google not to be evil; but it's the institution decides to do evil, not any individual, so they did not have (or could not maintain, as the organization scaled) the power to guarantee this. No one does.
Maybe they store physical photos and the albums got soaked in water. Sadly it happened to all our 90s photo albums during a flood. Some of the photos were stained really bad.
I scrolled back and checked some pre-2015 photos, they look fine, and then deleted the app.
I uploaded a photo to google maps. A week later it found some random picture of a local park and offered up a notification suggesting that I upload this photo to the destination it had located in the maps app.
How on earth are they doing this without "scanning" the photo? Seriously, please tell me.
It’s less about Google wanting to scan your data today and more about Google’s stingy strategy for photos plus Google’s lazy engineering (too many other interesting games to solve). Google Photos was born out of Google Plus, and is under heavy revenue efficiency pressure.
But then I think about the convenience of my 2TB Google Drive and I can't do it.
And I think, too, of the Polaroid pictures my parents took and gave me in a box once I got married and are now fainted. Mayhaps algorithms deciding that it's time to let go of your own data is just the modern form of yellowing.
I highly recommend it. It feels like pure freedom, putting a price tag on my personal data. The migration is something done in two steps: set up your alternative service, but also keep the old one (gmail/gdrive/calendar/...) running. Then start to use the new email service, while keeping the old one.
There are enough affordable options and you do not have to migrate everything at once. I started with my emails - gmail is still running and forwards all incoming emails, so I wont miss anything. I already updated all my accounts/logins to the new email address.
The next step was copying all my files from Dropbox, GDrive, and so on to {service I am using}. The data in the other clouds are still there, but I am going to delete it all soon.
There are things that are not perfect - like accepting appointments, which I still want to show up in my Google calendar. To those mails I still have to respond in google mail. But I am getting there, one step after another. The win so far is much bigger than the loss/pain.
As for pictures, I do use Google Photos, but it's not my primary photo video store. It's more for the app that lets me lookup pictures.
For storing original versions of pictures/videos, I sync from my phone to a home computer using Syncthing, and store backups of my files on AWS S3 using Restic for encrypted backups.
If I lose Google Photos, no big deal. I still have local and cloud copies of my data.
Why not sync them to a cheap external drive on a regular basis?
Speaking of which, as an aside there's no excuse for big tech to not support the Linux desktop (I also use macOS and Windows 11 so not a die-hard fanboy). It doesn't matter if the userbase is tiny. Linux enables massive profits in the cloud so any cloud provider that supports multiple platforms but not Linux, whilst also generating huge profits from it, is doing a very bad thing. And that includes MS, Apple, and Google - full support regardless of the (comparatively small) cost as thanks for the profits they generate from it.
M-DISC use a more stable media, and should last much longer, but are (much) more expensive and require more recent burners. There are dual-layer M-DISC with higher capacities, too.
I did a bunch of research on long-term data archival and wrote it up here: https://photostructure.com/faq/how-do-i-safely-store-files/
I refuse to relinquish my files to some insouciant third-party. And what recourse would you have if they lost them?? Very probably them saying "Sorry. Not our problem."
You can just hash them locally or use AI to identify modifications.
Agreed. And that promise has been fulfilled - although it costs more than nothing and you need to do your homework.
You know these providers are proven bad actors. You know their business model(s) are antagonistic to your goals. You also know there are stable, time-tested alternatives.
What did you think was going to happen here ?
Edit: to pay a minimum of $12/month??
This is already the case with different image compression algorithms improving on one another. Users couldn't and shouldn't tell the difference between JPG, or slightly smaller / resized JPG, or WebP, or whatever. I personally don't think image hosts should have to alert users when making undetectable changes to their images like this.
It's simply not the same thing. If I opened my file browser one day and saw that all of my FLACs had been converted to MP3, I would go on a murder spree.
Well, let's see:
ssh user@rsync.net sha256 photos/IMG_1234.HEIC
Yep, checks out.
The checksum commands:
(md5, sha1, sha224, sha256, sha384, sha512, sha512t256, rmd160, skein256, skein512, skein1024, cksum)
... are available to run as you see fit.
Nothing about rsync.net per se, other than the general idea that any data you put on "somebody else's server" can't be trusted to stay the same if you don't have a high-quality integrity-comparison content-hash of that data kept somewhere.
0: I'd be willing to bet money at 1:1 odds that MD5 preimages still cost at least 2^96 bit operations (out of a nominal 2^127 hash invocations) 5, 20, or 100 years from now.
Can you? Sure. Are their weird edge cases where it’s mildly helpful to know as opposed to having to ask someone? Debatably. Should you need to in order to not have things break for no reason? Absolutely not.
Maintaining a house or yard, for example, or taxes. Driving also has a pretty high learning curve -- high enough to mandate driving lessons -- but once we learn it we're set for life. (Tech is the same way here....)
By the principle of explosion[0, eg].
As someone who works in generative modeling, that's probably what is creating these issues in the first place. Kinda like how you can see certain GAN type of patterns on Netflix when there is upscaling or if you don't turn off the fancy features on your TVs. Of course, finding how these images are corrupted leads to better models so they will be solved. This isn't the first time Google has edited photos and made them worse and it won't be the last. But the trend is that the photos get better.
My home backup is going to be it's own problem after my death.
Exactly, and setting up a NAS has never been easier.
People use so many cloud services and having it all private and self-hosted is absolutely amazing. It’s really not that hard to set up either, and there’s little to maintenance outside replacing a failed drive.
That started me down the path to owning all my own media.
It also didn’t help that iCloud Photos is a privacy nightmare.
[0]: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
I trust any cloud provider a lot more than my lazy self.
> and external storage is becoming cheaper every year
Sure, but electricity isn't. At the current price in the UK it seems like running a small NAS would be 10 GBP per month, more than most online storage solutions.
That is without counting the initial investment of buying the NAS, the cost of maintaining it (if I have to spend an hour a month on maintenance, that's $1000+ a year of time wasted), the footprint of the NAS, the noise, the heat, etc.
A backup in one place isn't a backup. The 3, 2, 1 rule and all.
Meanwhile, there hasn't been a cloud provider without massive reliability or data loss issues. My NAS has better uptime than Amazon, Microsoft, or Google.
> costs maybe pennies to run.
Pennies per hour, per day, or per month?
I'm not saying it isn't under 40W - as you note, if it's idling properly, it's pretty easy to have a NAS including disks under 10W idle. But the electricity cost is not negligible when you are running something 24/7. Even 10W24hr30day is 7.2kWh per month, and most places cant count on 8c/kWh electricity anymore.
> My RPi 4b runs on a 18W power supply, which would put it at 432Wh per day
Which would be 13 kWh per month. If we're comparing to monthly fees for cloud storage, it's important to look at the same time frame.
I have two SSDs on it that are powered by the RPi itself, so the whole package is < 18W. Also, SSDs use way less power than HDDs.
> Which would be 13 kWh per month. If we're comparing to monthly fees for cloud storage, it's important to look at the same time frame.
Yes, running at max load. A more realistic load will be 3W/hr -> 72W/day -> 2kWh/month. That would put it at ~0.3€/month at a realistic load and ~1€/month at max load (0.15€/kWh where I live).
You also have to take into account there's other stuff running on such "NASes", like the PiHole, a git server, a plex server, which are going to cost you separately if you need them. On the other hand, my setup cost me ~300€ (RPi, charger, drives, USB-SATA adapters), which buys a whole lot of VPS and a whole lot of online storage.
So you took it out of the shipping box, plugged it to power, and pressed the power button, and everything was up and running?
And since then you haven't had to update any software on it, reboot it, unplug and replug an ethernet cable, shut it down and opened it to add storage, etc.?
> it makes no sound while running
Looking at a basic NAS on Synology shows that it's around 20 dB, which is admittedly very little, but is not 0.
> and costs maybe pennies to run
Again, looking at the same NAS and using my actual price per kWH, it's around $10 a month.
> there hasn't been a cloud provider without massive reliability or data loss issues
My searches haven't turned up any article about a data loss from any of the 3 big cloud providers. Even the story here, while scary, is just about compressed files, not originals.
---
I'm not denying that a NAS has advantages (the fact that it's a lot more than just a data backup), but in the context of backing up files, it's not at all convenient compared to using online solutions.
I think it’ll be possible to create inorganic ‘brains’ that have human-like subjective experience, because to think otherwise reduces to some form of vitalism, but I doubt we’ll be able to move specific individual consciousness across technologies.
Are you sure about that? That does not sound right to me.
Though, saying that, OP’s point about needing a human body to have human experiences may be correct if souls get a new human body in the afterlife - I’d just assumed liberated souls, like ghosts, had no physical form.
I'd say that the human body is just one very sophisticated I/O device.
The fact that we can't replicate it now doesn't mean that it wouldn't be possible after a millenia of engineering, even though this replacement device might as well be a partially organic one in the end.
That said, there are no guarantees that we don't fall into a sort of "dark age" of no more growth/research into any new directions, or don't wipe ourselves out altogether.
Nowadays it's only available inside the world of Black Mirror and similar fiction.
My experience is that when someone dies, all their paperwork gets left in a drawer for years until the next house move and then it's all thrown out.
The digital archive or cloud storage will likely in many cases not even be discovered. And if it is discovered, it will probably be such a load of images, good and bad, that no-one will have the time to really look it through and appreciate it.
But for sure, both options combined is probably best.
If I had a gigabytes of photos of my grandparents or great grandparents I would spend hours pouring over them looking at what their life was like before I knew them. That would be such a neat keepsake.
Given their is no real customer service team that handles their free tier accounts, I am personally reluctant to assume my Gdrive data will be salvageable.
https://support.google.com/accounts/answer/3036546?hl=en
(That's not to say there aren't other concerning reasons why that data may not be accessible.)
How do you do this?
My experience has been the opposite, with relatives eagerly snatching up the old photographs.
My boss is young and one of those people for whom digital is the only way. Then one day she got a box of old photographs and was absolutely amazed that she could turn them over and read where they were taken, on what occasion, and who was in the photos.
She now prints out all of her photos and writes notes on the back of them and keeps them in a box, just like people have done for the last century.
They're absolutely nothing like block-based JPEG, that's for sure. When I inspect the images Google Photos serves me in my browser, it is serving up JPEG (from the response headers). But is this an artifact that shows up in AVIF or WebP? I wonder if mobile clients are getting a different encoding.
The only objective thing I notice is that the artifact lines tend to track a line of constant brightness, so you seem them appearing perpendicular to gradients of light and shade. And that each artifact line is black/dark on one side and white/light on the other -- and that the white edge seems to be in the direction of darkening, while the black is in the direction of lightening.
Someone here who works with modern image codecs must be able to hypothesize what part of encoding/decoding must be bugging out here?
(See, e.g., http://graphics.cs.cmu.edu/courses/15-463/2019_fall/lectures... for a nice overview of gradient-domain image processing.)
But seriously, I am also very curious. I wonder if they are playing with an in-house compression algorithm.
These artifacts aren’t where I’ve seen them before. Normally you’d see this sort of posterization and clamping around global highlights and shadows, especially if the incorrect colorspace profile was applied, but this seems like the affected border is around a localized area, possibly due to a buggy pseudo-HDR implementation (what you see when you move the “pop” adjustment slider, which increases localized contrast ranges). Google+ images had a mild pop adjustment applied automatically.
With every passing week, every Google fuck up like this and with ente's great pace of development, I draw myself nearer and nearer to the precipice of full switchover. I will soon be having my last Google Takeout.
[0]: https://ente.io/
I switched to the service completely.
edit: RTFA, guess ente does - https://github.com/ente-io
If you happen to revisit and find something lacking, please write to vishnu[at]ente.io, we'd love to make ente work better for you.
[0]: https://community.hetzner.com/tutorials/install-and-configur...
It looks like Google has unpublished our app, will figure out a fix.
E.g. if this was about storing monetary data, this would never have happened.
Personally, I use Syncthing to sync between devices, including a NAS as a target, which regularly backs up to Backblaze.
rsync -av /Source /Destination rsync -h
QNAP or Asustor has standalone apps my parents can use as well. Without getting your parents involved, you could either pay for remote hosting for your off-site, or trade RAID space with a friend.
Another good test: upload your photo collection, then download it and binary diff the two sets of files.
Let’s hope none of the images were NFT “assets” then haha
Also, when was the last time you heard a banker say that everyone makes mistakes all the time.
That becomes apparent if you ever had any issue with a google product. There's no way to resolve issues outside of canned answers from "AI" systems and public forums.
https://twitter.com/samwcyo/status/1569897392560050178
Discussed in https://news.ycombinator.com/item?id=32835190
Of course they do. How can they possibly put any value on endless PB of holiday snaps? They only care that you are now reliant on them to store your memories.
I tried to open a Word document that I hadn't viewed for several years, but got an error that the file was damaged. It looks like this happens sometimes, after searching for the error online. Luckily I was able to recover an earlier version of the file (via version history), but it was alarming that it happened in the first place (also I'm not sure if important edits were made in the latest corrupted version).
It's unfortunate how many cloud services default to storing on the cloud only instead of also keeping a local copy, and don't even provide the option to opt out. Even if you choose the "keep offline" option for several services, for unknown reasons, this doesn't seem reliable (Maybe I'm 'using it wrong'? But in practice, I've found myself having to download files that I was sure I set to keep offline.)
The same isn't true for GCP - there they lost some writes to customers persistent disks - but only ~50 megabytes worldwide, which is still rather good when you consider they store millions of terabytes.
It seems that every other day you learn about people being permanently locked out of of their Google account. Every byte of data stored in there is permanently lost to the person losing their account.
"We have always had better storage durability than Europa."
Note, I am not arguing that this is necessarily what happens. The thread full of contrary views on cloud storage just tickled this cynical take loose from wherever it was lodged.
As far as I know, they just haven't admitted to losing consumer data. Until they define that and put even a little bit of effort into checking whether it might hold true, they don't really have anything to be proud of.
Back in the aughts Google was infamous for disappearing mail, if not entire accounts just disappearing into thin air.
https://techcrunch.com/2006/12/28/gmail-disaster-reports-of-...
https://www.theinternetpatrol.com/has-your-gmail-email-disap...
Years later:
http://www.cnn.com/2011/TECH/web/03/01/gmail.lost.found/inde...
That being said my photos are the only digital files I really care about not losing. My photos and videos from my phone get backed up to iCloud, Google Photos, OneDrive and Amazon’s photo storage that comes with Prime.
My videos get backed up to all of the above except Amazon’s storage.
But, I don’t have a personal computer anymore and soon won’t have a personal residence. My wife and I are going to be digital nomads traveling across the country for a few years.
My backup flow is: phone, iCloud, local ZFS array served by PhotoPrism (synced using PhotoSync), Backblaze.
Details on ZFS checksums: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Chec...
It’s an old project and information on the internet is getting thin but the math is sound, the tool works, and there is a wikipedia page:
https://en.m.wikipedia.org/wiki/Parchive
Edit: btw use of par2 files can be added on as an extra step with minimal storage overhead, without replacing anything you have now.
Apparently I am not the only one[1][2]. Other people have a mix of errors[3]. It is not the reliable data extraction I was expecting for something as valuable as my family photos. Google is losing a lot of trust from me based on this.
[1]: https://news.ycombinator.com/item?id=25591630 [2]: https://sysc.org/what-is-google-takeout/ [3]: https://www.reddit.com/r/googlephotos/comments/khited/google...
It works extremely well, and you can re-run it any time to sync new files locally too.
I used it like that for a few months before I finally installed syncthing on my phone and stopped using Google Photos altogether.
Now what I do is take photos on my phone, have them sync to a NAS. And on the NAS I used a modified version of this https://forum.syncthing.net/t/android-photo-sync-with-exifto... to build up a YYYY/MM folder organisation and move files older than 30 days from the syncthing folder into my archive. My archive is then in my Plex so it's still accessible to me.
In essence: 1) Take photo (implicit sync to NAS), 2) wait 30 days, 3) archive photo into long term directory naming convention, whilst making available to Plex and deleting the version from my phone (by deleting the syncthing version it will delete the one on the phone after 30 days too).
The only thing that it gets wrong is that it cannot restore the file created date to video. But in the grand scheme of things, that's a nit.
I've been putting off a Google photos back up for years, and this was my excuse - waiting for this issue to be resolved. But I'm being a fool, 60% compression is better than losing everything.
I have just compared originals I had which were uploaded to Google Photos vs the ones downloaded.
Visually there is a barely perceptible difference in a side-by-side comparison on a high quality and calibrated monitor. I wouldn't have noticed if it weren't side-by-side and zoomed in.
On the file size, it's gone from a 6MB jpeg down to about a 1.5MB jpeg.
On the smaller files that difference is far less, but jpegs above 3MB seem to see the biggest reduction in file size.
Thankfully I have all of my originals that weren't taken on the phone camera so I don't mind this so much as running my exiftool script over the original archive produces the same naming convention and I can merge those folders and still have the originals I care about.
Sheesh though... such a thing for Google to screw up on.
Use anything else you want for day to day usage - Cloudflare, Digital Ocean, Google Drive, etc. whatever fits your budget and needs. But sleep safe knowing your data has a backup.
Same thing with YouTube - you never know when Google might decide to wipe out your channel, and if they do you’ll never know why they did it, and you’ll have no recourse after they have done it. There are plenty of stories on HN where this has happened. So the master copy of all your videos should be in AWS so you can start again if you need to. IMO.
I hope to fix up my prototype and launch it next year.
If anyone knows of a quality source of bulk SD cards or flash drives in high capacity please let me know.
The idea would be to subscribe to have this data shipped once or more per year so hopefully some of the data would survive.
Any advice on error correction metadata and tools?
It's been great so far although I do need to figure out why it powers on all the time (probably SMB shenanigans).
But the main thing I got it for is to set up freezing folders full of RAWs I've taken to back-up to Glacier.
Don’t bother getting all frustrated in a big support thread. The engineers who caused these bugs DO NOT care about you. They are way way too busy solving dynamic programming problems. Chargebacks are the most effective tool here.
You are flirting with having your entire google account disabled. As this may end up un-personing you, I would suggest strongly taking another route.
I had a disaster when using DropBox way back around 2011. Ever since then I have been responsible for storing and backing up my own files. Nobody else gets to have them in their 'safe'-keeping.
Update: my limited knowledge on this is proof I’m wrong.
It totally depends.
Even most data transmission components don't use forward correcting codes. They typically just have error detection.