My Cloud Storage Crisis | Hacker News Reader

My Cloud Storage Crisis

tiramisu.bearblog.dev

48 pointsmemorable64 comments

For some reason I found this article hilarious.

  - Google ends unlimited Google Photos storage for new photos
  - User is apoplectic that hitting free storage limit may affect email
  - $2/month is an affront to user's anti-subscription sensibilities
  - User migrates photos off of Google Photos
  - User realizes 90%+ of his photos were a waste of space

It may very well be because I'm seeing myself in this. I just migrated a lifetime of email from Google Workspace to iCloud Mail when Google threatened my free legacy plan. It worked great, but I'm sure 95%+ of those emails are a waste of space.

I'm sure 95%+ of those emails are a waste of space.

That's a bit like the old proverb 'Half of all advertising money is wasted'. Now, if only we knew WHICH half it was.

It's always surprising to me how often I try to search though years of emails looking for some particular snippet of information.

Just yesterday I needed to know what model of TV I owned, and rather than try to crawl behind it with a flashlight I searched for the BestBuy order confirmation in my email from years ago. I do that kinda thing all the time.

About ten years ago I deleted the 10 gig of email I had floating around. I delete everything now. Haven’t needed any of it yet.

If I do need to keep something I print it to a pdf and file that in documents and that’s about 20 emails so far.

User migrates photos off of Google Photos == user spends $60 on external hard drive + x hours of their time moving everything across. Seems a shit setup compared to forking over a pretty small amount of money to google for a very convenient service (especially when the hard drive dies and you realise the backups are out of date).

The problem is that things like photo storage and emails are really non negotiable. Most people need these things as a basic digital necessity. Google is well aware of it, and it knows that it can increase fees or reduce storage space in the future, and most of the folks will keep sticking around because the company holds a lifetime of emails and photos as hostages. It's just like the pusher who increases their rates for cocaïne, well aware that people who are hooked onto it will keep paying whatever.

I've spent quite some money and time to set up my own cloud, but the initial investment paid itself back after 2 years already when compared to the monthly cost of Google One. Companies like Google need to understand that, if they try to pull too much the rope of profit, even if that means hurting services that people have started to take for granted and built their lives around, then people will just start leaving.

Everybody is different, but my privacy and desire to not use my life as a feed for google AI & ML makes this a very small price to pay to move away.

Couple this with migrating to something self hosted like nextcloud, and I cant see why anyone would possibly pay google $2 per month to be part of their data aggregating empire.

I'm almost positive my home server (which is a lower end, but newer desktop) costs more than $2 a month.

I don't think Nextcloud's photos app would run well on something like a Raspberry Pi either, considering when I tried my photo collection in it, it would consume enough RAM to crash the 8GB VM it was in and give up.

While I have time and knowledge to troubleshoot, realize most don't and $2 is less than an XL coffee.

It is completely subjective and relies heavily on how much you value your privacy.

For me, the cost of running a home server is far less than the cost of Google having my entire digital footprint.

I really want a hybrid solution. Where my files are in both the Cloud and my Local NAS. I can cut off my subscription whenever I want, and I can change and sort the files on my Server at much faster speed. I also get the extra protection of offsite backup and redundancy check for files corruption.

> I really want a hybrid solution. Where my files are in both the Cloud and my Local NAS.

IPFS?

The alternative is to pay Google a fee each month forever. And when they decide to raise the fee, give them more money. And go along with any current and future privacy policy. As opposed to spending x fewer hours scrolling Facebook.

> You can only download 500 photos in one go

Uhh, no? https://takeout.google.com/ allows you to download your entire library at once.

There's even a transfer wizard where you can export your photos to an external service like Apple Photos! https://support.google.com/accounts/answer/9666875?hl=en

If you have lots of photos, Takeout can take days to prepare the archives. It splits archives in multiple fixed-size files, each has to be downloaded separately. The downloads sometimes get stuck in the middle.

It is better than downloading 500 at a time, but still a pain.

If you are offboarding fully off from Google photos then I imagine waiting few days is fine.

I actually never had a problem with Google takeout:

- You can schedule an automatic Takeout to happen every 2 months, I do this.

- The “multiple files” I never found to be an issue. I select the 50GB option, and get back ~8 50GB

- Personally, I never had a single issue with file download. Downloading the eight 50GB tar files always succeeded and fully saturates my 1Gbs connection.

Btw, during my last takeout download I opened up nettop to look at traffic, and realised the downloads are happening over UDP. Researched a little bit and I think that’s related to QUIC protocol? Was new to me. If you used a different browser, perhaps try downloading the Takeouts from Chrome next time since it will use QUIC.

https://en.m.wikipedia.org/wiki/QUIC

If you use tar.gz the split is at 50 gigs. That seems perfectly reasonable to me.

I try to backup my Google Photos to somewhere else every couple of years.

It is a huge pain since (as far as I'm aware) it allows you to do "takeout" but with folder granularity. If you don't have them neatly organised, you'll either have to download the whole thing again or risk missing lots of data. It should have a "New photos since date X" option.

This smells of a dark pattern to discourage migration to other services.

"New photos since date X" doesn't sound useful to me for migrating to other services. It sounds useful for backups while remaining on Google Photos.

Disclosure: I work at Google, but not on Photos.

What's the recommended way to back up photos from Google Photos?

I don't work on Google Photos and don't use it much, so I don't know.

if you want to rely on takeout, you better not have uploaded youtube videos with the same account.

even tho you get a lower quality than what you uploaded, takeout will contain those as well, and trust me it balloons the overall download.

“How dare they” (from the post) limit this person to the 15GiB they paid to use. Absolute horror that you might have defined limits and be notified of the consequences before you exceed those!

I mean, Google did originally say that they would provide the author with unlimited storage, and then once he was in, they altered the deal. I doubt he would have used the service if it would have meant a monthly charge when he started.

Another way to look at this: users got years of free unlimited photo storage before google decided to monetize. IDK, storage ain't free, it was nice while it lasted

Another consideration for Google may a decreasing level of value in scanning data from photos or acquiring new training data. Like they have all they need and more after years of collecting photos from everyone and have seen nearly all the permutations. No more value for them means now we pay for the storage.

Google Photos images have never been used for model training. It's also unlabelled UGC so I don't know this conspiracy theory has ever gained any traction.

~ ex-photos engineer

Thanks for weighing in. The app sometimes asks me if it has correctly labelled people in photos, and I can confirm or give another label. I would assume some of this is going back to the server, right?

Yep, storage isn't free; I'm one of those people who just keeps their own NAS, so I'm well aware, but what I wonder is - was the plan always to get people in with a free offer and then charge them later, or was there some other bet going on?

...and continue to get free storage on anything uploaded before the transition date.

I stopped using Google Photos in favor of Synology Photos which comes pretty close in terms of functionality and polish.

Google Photos is excellent.

Synology Photos is really good. And this seems to be a worthwhile tradeoff for me.

Disclaimer: I founded 2 photo startups and have documented the solution I've finally landed on and have been using/evolving for 7 years now.

1. https://github.com/jmathai/elodie

2. https://medium.com/@jmathai/understanding-my-need-for-an-aut...

3. https://medium.com/@jmathai/introducing-elodie-your-personal...

4. https://medium.com/@jmathai/my-automated-photo-workflow-usin...

5. https://medium.com/@jmathai/one-year-of-using-an-automated-p...

I was going to rant about Synology Photos being not that great at many things, but looking at your workflows it makes plenty of sense. You seem to cover a lot of the weakness through these batches.

This still feels a bit fickle as Synology app quality hasn't much improved IMO (I'm mostly using the DSFile and DSUpload apps, both in android and iOS, and they still work but are far from getting the love they deserve from the Synology dev team). But at least you still have all your photos at hand and can move to any other alternative import and viewing system that would fit the bill.

I use DSFile for getting photos off my phone. It leaves a lot to be desired.

Read tha article and came here to say just this

I pay for Google One 100GB (Annual Plan - 62 GB used) and YouTube Premium (Monthly) to ensure there is peace of mind and fewer things to manage. I moved to Tutanota for email and DropBox for storage before this. Life became fairly complicated and when I started missing few emails in Tutanota, as they went to spam, I turned back. Tried one drive also and it actually made me cry.

I am a big privacy nut but in today's world there isn't a single provider with suite of apps so well connected and evolved, as Google has. I would like to pay for Google for next 5 years, if Google allows, to ensure that my email address continues to work even if something happens to me.

Life should be simple, Google Workspace aids in that.

I personally think it's a bit mad to trust Google, the champions of shutting things down / changing terms, with your email. Companies like Fastmail make switching easy and they don't muck around as it's their only product.

But I do suffer with the issue of finding my Google photos is full of annoying received memes and screenshots. Is there any software out there that can reliably identify screenshots. They should be recognisable by the familiar dimensions and menu bars at the top+bottom. Bonus marks for being able to also recognise stupid meme pictures that contain computer generated graphics or text. I just want photos in there. Which I know is partly my fault, but I'd like to now fix it.

> Is there any software out there that can reliably identify screenshots.

Google Photos. Their web interface has a special section where it collects screenshots[0], or at least what it thinks are screenshots. It's not perfect, but is better than nothing.

[0] https://photos.google.com/search/Screenshots

Anything offline? Sorry, when I said "my google photos" I actually meant the photo folder on my NAS that is partially populated with the result of a Google Takeaway last year. I used to use Google Photos but I stopped.

I wrote an article on my blog on how to move photos from google photos to Synology, if you're the kind of guy that wants to host things on a NAS: https://michelenasti.com/from-google-photos-to-synology-phot... There's also a way to overcome the 500 photos limit (using google backup options).

Nice write up. I’m currently in the process of moving all my photos into my new Synology NAS. I can see that you prefer the terminal, but the downloading could be performed via the download station and the copying of the files via the file manager/explorer. This way the whole procedure would be more user friendly to novice users ;-)

So I used to pay for the $2/m 100gb plan as I wanted full resolution not low quality. Then google removed the ability to sync your pictures with you computer which was my main reason for doing so.

I then went to onedrive's 100gb offering. I bit slower to sync but not bad.

These days I use syncthing. Easy to setup and just works. I take a photo and its on my computer in seconds. I keep forgetting to clear out onedrive and cancel it though.

A while ago I purchased a 7 TB hard drive, paired with another 2 TB on SSD for things that I need to access faster, and another 10 TB on a slower disk used for long-term backups.

I plugged to a mini pc running Arch, put it in my utilities room, and I've been running my Nextcloud instance on it for a while.

When NC media management became better (it's not on par with Google Photos yet, but there's a good app for Nextcloud Photos that now even supports face tagging, and even the media tab in the official app is now getting better), I just downloaded all of my Google Photos pictures, uploaded them to my server, spent a couple of days cleaning up the collection and fixing the timestamps, and I haven't looked back since then.

Google is a parasitic company in financial distress because the primary source of its income (ads) is going through a major crisis, and they are desperate to make money out of other things, even if that means depriving the user of "essential" services by bundling email, photos and drive under the same meter.

I decided long ago that I'm done with trying to understand the financial and product motives behind their deplorable decisions. And I'm done with them holding my emails, files and photos as hostages. And nowadays you can have a service that is almost on par with Google Photos. You just have to invest on a large drive, run your own NC instance, and use the right apps.

> but upon reviewing my photos I've come to the realization that the vast majority of them are useless. Many of them are videos for my 1SE vlog, screenshots or pictures of things I wanted to share to a friend, or pictures of food. I'll never again look at 90+% of all these pictures, which leads me to the conclusion that 1. I should be more mindful of when I choose to take pictures (and take fewer in general), and 2. I should do something about the best 1% of my photos, like post them on this blog.

I would say way more than 90% of my photos are useless. Heck, every time I take a photo I actually take 3+ and then share the one that looks best. Meanwhile, dropbox automatically backups all the photos and my photo folder keeps getting bigger and BIGGER and BIGGGGER.

QUESTION: What program works best to go through a lot of pictures and delete those that don't matter? I tried the default windows once, xnview, and a few others, and they are all so slow at navigating photos that I would never finish :(

I used my recent migration from Android to iOS to download and clean my photo library. Even though I'm the kind of person who aggressively culls their photographs immediately after taking them for the slightest imperfections or redundancy, turns out that a very small part of my photo library are the "first trip with girlfriend" kind of photos that I like to go through every few months, and most of it are the "look, a nice tree" kind of photos.

After deleting most of the latter kind, my photo library sits firmly below one photo a day, and I can have a full copy of it on every device I own.

I don't shoot many videos though.

I think one valid point is that the coupling of storage between Email and other services can have rather annoying side effects. In most cases receiving email is much more important than being able to store more Images. Especially as receiving emails is triggered by the outside world, while storing images or files is something you do yourself or have set up in some automated way.

Not receiving emails because you messed up something and filled up your storage quota with images seems like a very clear drawback of this kind of combined storage.

What's even more infuriating is that if you're on the legacy Google Apps for Domain plan, the only way to upgrade storage is by going to Workplace for the whole domain, even if only one person (my mother) requires it.

So instead of buying her a $20/year plan for 100 GB, they're forcing a hundreds of dollars yearly plan on me.

And obviously there's no way to detach that one account.

Looking for a better option here.

If broadband hadn't stalled and we kept on the 00's curve it would be standard to have 10G or faster at home by now, and the services developed would be different. It seems like "cloud xxx" is synonymous with "networked service that has ingenious engineering applied to cope with slow networks".

I bought a Synology and moved everything to their Photos app. It’s really good. But if there was something better I could always mount the SMB share into a Docker container of whatever that is. It’s nice to own my own shit again after 20 years.

And dividing your cost of the Synology plus harddrives + power required by year, how many years before you break even with a cloud storage plan ? Add to that the cost of a local backup drive and the cost of your time.

Being based in Europe, where power costs €0.7/kWh, of course doesn't help, but powering a 4 bay Synology here costs around €23/month (45W x 24 x 30,5 / 1000 x 0.7), and that's the power costs alone.

I have around 10TB of cloud storage (including backup storage), and i pay €22/month for that, so even before factoring in the hardware cost I'm saving money on the cloud.

I have a small ARM machine running to synchronize cloud data locally in "real time", which has a USB drive attached for local backups, and the machine also makes cloud backups of the mirrored data. It uses around 4W idle, so costs are around €10/year, but of course doesn't offer much in the way of redundancy, but still i manage to live up to the 3-2-1 backup principle with data being primarily stored in the cloud, backed up locally as well as backed up to a different cloud provider.

That being said, if your use case is more than 10TB, the cloud starts getting expensive, and the Synology might make sense.

I don't have a Synology, is it really 45W 24/7?! Is 45W the lower power mode number? I'm assuming it has a mode where the cpu sleeps and the disks spin down, and it must spend most of its time in that mode. If not, I'm very confused why they are so popular

The load for the Synology model itself varies, but is usually around 10W - 20W.

Add to that the load provided by the hard drives themselves, which is usually around 4W - 9W per drive.

Low estimate is 10+4x4 = 26W

High estimate is 20 + 4x9 = 56W.

Wall measurements from my own DS918+ model with 4x8TB WD Red are 40W (idle) - 48W (full load on drives) - 52W (CPU load and some drive load).

https://www.tweaktown.com/reviews/8231/wd-red-10tb-nas-hdd-r... has the drive power consumption at 0.5W when "sleeping", which is hopefully most of the time. You'd hope a Synoplogy would put the disks to sleep. All in all, surprisingly high power for something that's meant to be on 24/7.

The "major consumption" from a 4 bay nas is usually from the drives.

I used WD official documentation for my consumption figures (https://documents.westerndigital.com/content/dam/doc-library...), which state 5.5W idle, 8.8W access.

Being WD Red (Plus) drives, they don't support hibernation (not supported in firmware), so they're spinning 24/7, though supposedly using only 5.5W doing it.

So in theory the NAS should be able to go as low as 20W, but in reality it rarely does, at least not when running the drives in raid. Because it uses the same drives as its root/boot partition, anything that logs will write to disks, essentially keeping them active most of the time.

Over a weeks usage, my DS918+ rarely dipped below 40W, and the average consumption was 44.8W over that week. There could be an easy "fix" for it, as the 918/920 models have 2 NVME bays, so in theory if Synology would allow it, those could be used for boot/root, allowing the spinning rust some rest every now and then.

Instead i've turned it off, and replaced with a small ARM device with an SSD.

I'm also experimenting with a NUC7CJYH with a large Seagate Barracuda drive and a USB stick for root/boot, and that averages a power consumption of about 5W over a week being used for daily backups. I tried with a RPi 4 with a bus powered USB drive, and that ended up at 4W, so if power consumption is "almost equal", the NUC has a lot more power.

It's not about cost. I'm privileged enough to not have to worry about a couple dollars one way or the other per month. It's about owning my own photos. It's about knowing that if Google gets some bug up their butt and shuts down my account, I haven't lost every photo I've ever taken of my kids.

I've chosen to "counter" that by mirroring data at home (without redundancy), and making local as well as remote versioned backups.

If my cloud provider of choice locks me out, i can simply restore my files to another provider from either of my backups.

As for photos, i also make yearly Blu-Ray M-disc archives of the past years photos (or more frequent if i have a disk worth of photos). Identical sets stored in different locations (>100km apart), without any encryption, archiving or compression, and instead relying on physical security to protect them.

Alongside the Blu-Ray discs i also keep a couple of USB hard drives with a mirror of my entire photo library (at time of archival) that is rotated whenever i deposit a new archive Blu-Ray disc. These drives use a simple file system. I can't at present remember if it's ExFAT or Ext2, but it's one of them, and it was chosen (at the time) because it is relatively easy to recover data from in case the filesystem gets corrupted.

The hard drives are then subjected to a full (non destructive) surface read/write test, updated with the latest photos and stored. The surface test may be overkill, but there was a discussion some years back, that drive firmware will detect and correct weak magnetic sectors on the drive itself when it read data, which is what i'm trying to accomplish, simply a refresh of the storage medium.

> Being based in Europe, where power costs €0.7/kWh, of course doesn't help, but powering a 4 bay Synology here costs around €23/month (45W x 24 x 30,5 / 1000 x 0.7), and that's the power costs alone.

This is crazy expensive. what is the 30.5 / 1000 ?

45W for 24 hours for 30.5 average days per month to get Watt Hours per month, then divide by 1000 to get kWh then multiply by € per kWh.

It's always good to remember these costs but I think €23/month on power is a bit much. My workstation with a 3970x threadripper idles at less than 45w when not in use.

The measurements (45W) was the average power consumption at the wall by my DS918+ equipped with 4 x 8TB WD Red drives. WD Red drives do not support spin down, so at most they're idle.

Synology does publish power consumption for each model, i.e. for the DS920+ (https://www.synology.com/en-us/products/DS920+#specs) model they state :

- 32.17W access - 9.69W HD Hibernation

But, they also specify that the drives used for their calculation is

    Power consumption is measured when it is fully loaded with Western Digital 1TB WD10EFRX hard drive(s).

And looking up power consumption of that particular model we get

- read/write 3.7W - idle 3.1W - standby 0.6W

so for the "access" vaule we can subtract 4x3.7W (14.8) and see that the device itself uses 17.37W. Then lookup the power consumption of your drives, i.e. WD80EFAX (https://documents.westerndigital.com/content/dam/doc-library...), and find

- read/write 8.8W - idle 5.5W - standby 0.8W

So you add the 4x8.8W (35.2W) to the 17.37W and arrive at 52.57W for "access".

And what happens if your house burns down, or two drives from the same batch decide to die?

This is only part of the solution - offsite backups is the second part, and ensuring you can restore those backups the third.

I've got a home-built NAS with 6 4TB IronWolf drives in RaidZ2, but then the volumes I don't want to lose are backed up with restic and rclone to JottaCloud, which is $8 for "unlimited" (in reality after 5TB your upload slows down progressively to them).

Yeah, it's all encrypted and immutably backed up to rsync.net at 1am every night. Gimme some credit. ;)

I ended up paying Google One. Google Photos is good enough, and moving to Synology Photos (the other option) is more expensive. Also, having another device running at home is uncomfortable.

The whole post is about -- "because I am allergic to subscription models and don't like being told what to do."

I respect the author's choice and philosophy in life. However, the free services are subscription. One just do not pay but the it will have be some day or you become the product.

I believe there are still lots of people who believe in not paying for non-tangible services online but would happily dole up enough for multiple coffees each day.

I had Google One 2TB for quite a while as a backup for my photos (primarily on iCloud) but I moved out[1] because of the weird way Google Photos was storing photos (they seem to be everywhere). Some were on Drive, then some where in Photos or something like that.

My other use case for Photos was to share with my family (a Shared Family Album, if you will). Before Ventura/iOS/iPadOS, we had dropbox "Camera Uploads" and then hazel would import it into our main Photos Library[2]. Now, besides the few hiccups that I'm experiencing, the new Shared Library in the Apple Ecosystem is working for us. Prior, I have done the auto share to a common library juggle with Photosync[3] but it keep dying/crashing.

Finally, during my usual digital chores (calendared as weekly, monthly, quarterly, and yearly) I include the task of just exporting all photos in the possible highest quality JPEG formats for the archives. One day, I will move slowly to an all open-source tools, methods, and simplified workflow for these contents.

I'm not a professional photographer. I keep photos not for me or just now, but I hope that few generations down the line, they discover funny photos and can be interesting to them. Think 50+ or even 100+ years from now.

The other reason I take care of my photos is that I feel nice when others use it and are happy with it. I have been a Flickr Pro since its very early days and stopped only a few years back. I have also gifted a lot of Flickr Pro to students and budding photographers. I have also been lucky to have "earn" low double digit thousands (USD) from some photographs and have been published in big publications around the world.

1. https://brajeshwar.com/2021/how-to-delete-all-photos-and-get...

2. https://support.apple.com/en-in/guide/iphone/iph1a6c849ab/io...

3. https://www.photosync-app.com/

Just a question, does google's photo storage store the bits exactly as you send them?

I've got a strategy of saving a local copy and then transferring to Google Photos, then using rclone sync those to Google Drive and purge local copies that I remove from Google Photos. It works really well and assures me that Google won't strip any metadata or reprocess the images.

In my experience, yes. You have to use Google Takeout to get the full bits back.

Reply on news.ycombinator.com