Hacker News Reader

Comment by hn_throwaway_99 | Hacker News Reader

Wow, 4.3 petabytes a day. Operating at Google's scale these days is just nuts.

One of the advantages of being on an exponential growth curve is that this figure has been way lower in the past. Just go back 5 years and it's only a fraction of that. 5 more years and it's even lower. The gmail/drive/etc. data that come from Google's first decade of existence probably fit into those 4.3 petabytes.

How is this an advantage?

Advantage in the sense that they have to store less. 4.3 petabytes times the age of Google in days is way way more than they actually have to store, as the time period when they were adding 4.3 petabytes daily is short in comparison.

So the advantage of being on an exponential growth curve is that today's problems are tiny in comparison to tomorrow's problems? That seems like a disadvantage!

The other implication of exponential data growth is that shortening retention periods is only a short term solution to storage costs.

I'd say it's the main implication. Google will have to use more drastic means soon. It will be interesting to see how this plays out.

I used to work on a certain Google product and I remember thinking a petabyte was big when our raw content store crossed that milestone. Now Linus Tech Tips can put that much storage in a single box.

Thanks for posting that, most people don't realise how much compute/storage you can actually put on a single rack cabinet nowadays.

I remember about ten years ago when 40TB of usable storage was a mid sized five five figure investment. That’s nas territory now.

I remember a time when 3mb of black market data was enough to get you out of the sprawl. Now you can barely get a Uber downtown for that.

Whoa.

Yep. Thats the flip side the the growth is that storage capacity keeps growing relative to price as well.

I spent $600 over a decade ago and put 7.5TB in my server. I spent $600 this year to add another 28TB.

I’m at nearly 40TB in my home server for maybe $1200 invested over the last decade.

One problem that we run into though is that spindle count /random access capability is not increasing correspondingly. It's an interesting challenge.

I remember a bit over 20 years ago when we had a full 19" rack full of storage. It was about 500 GB if I'm not mistaken.

You can rent systems from Azure or AWS with 20 TB of RAM.

That's mainly media. What's the deal with deleting emails of all things, as if they couldn't support that aspect indefinitely?

People use email as a sneaky method of storage

Sometimes it's the most practical way to share photos and documents, especially when iphones are involved.

We just use shared albums or text them to each other.

How prevalent is that? Some nerds doing this is a drop in the ocean. Rather my bet would be on Youtube as the hungry beast that can't get enough storage. There's a loooooooot of videos that have single digit views yet need to be accessible at any time, instantly. Video is large. Doesn't youtube even keep the original file nowadays?

Sounds like we need a way to encode arbitrary data into video that can survive lossy compression. Youtube filesystem?

In my experience, this is super common. Lots of average computer users send video clips, uncompressed 48 megapixel photos from their cell phones, PowerPoint presentations with embedded videos and 48 megapixel images, pdfs, and on and on. At the company where I work now several people have a 1MB gif of the company logo that they put in their signature.

Combine that with Gmail telling users for years that they had plenty of space, so there was no need to ever delete anything, and people believed it. Now you have a decade of emails with giant attachments or embeds that may or may not ever get looked at, and that keeps growing.

Deduplication solves that problem. There is only one (or very few) instance(s) of that gif on Google servers. That's why encryption will not be available anytime soon. Storage requirement would explode.

You can do deduplication and encryption. The harder part is how to make it paranoid-level-privacy-friendly.

it would be nice if they allowed deletion (or even dedupe!) of attachments

Right! Maybe they are cutting email because it is not as profitable.

I agree with the other poster who brought up storage costs not keeping up with growth of stuff to store.

I wonder what this means for youtube...

Email is the tightest grip Google has on me. Maybe YouTube.

But their Google music fail, their rising phone prices, their bait and switch photo storage, and low quality search results have caused me to change.

Any alternatives to Chrome, Firefox on both Android and Desktop are buggy, I want a different company?

If you're okay with Blink under the hood, take a look at Vivaldi. It's made by some of the same people who did the original Opera, and is very much in the same spirit - instead of the modern trend of bare-bones browser + third-party extensions, it comes with "batteries included".

+1 for Vivaldi. I can promise you the people behind it just want to make a great browser.

I tried Vivaldi for a long time, but a couple of things drew me back to Firefox eventually. The biggest problem was that it kept signing me out of Google for some reason. I've seen threads about this elsewhere, so I'm not the only person facing it. Then, the containers feature in Firefox was very useful as I have multiple "identities" in different spaces, so I could isolate them in their own containers to prevent spillover.

Sadly, Vivaldi was sold a few years ago and no longer seems the ideal choice if you're focused on privacy and security.

https://www.reddit.com/r/brave_browser/comments/iq33nv/brave...

When it was launched a year or two ago, Brave didn't integrate with the Chrome extension web store, but now it does (just like Vivaldi). Of course, also like Vivaldi, you may not need another adblocker since Brave has "shields" which are very easy to enable/disable (but it works fine if you install ublock origin, privacy badger, quick JS switcher, etc.)

You have mistaken Vivaldi for Opera.

Opera was sold to Chinese investment fund, they are scamming short term loans in Africa now.

Vivaldi was created by old Opera exec with same goals as old Opera. Their problem is not being scammy, neither is privacy. Their biggest problems are code quality - UI in javascript is sloow, closing tabs or fullscreening youtube video takes one full second. Then there is culture of secrecy - pretending open source by dumping uncompilable tar.bz2 code shreds from time to time and keeping bug tracker behind closed doors in order to not show how (not) many people work on it and what bugs they ignore for years.

Ok, that's fair, thanks for pointing it out, but I thought also that Vivaldi also had part Chinese ownership.

There's also this: https://vivaldi.com/blog/vivaldi-browser-open-source/

I interviewed at G and eventually matched with teams writing software for core networking functions. In that process, the discussion came out that the software was responsible for roughly 11% of traffic...for the entire Internet.

I think the main issue is storage Moore's law has really hit a wall and it is no longer the case that the data Google stores grows slower than storage per dollar like in the early days. Google's freebie business model is going to have to drastically change if the two exponents have crossed paths and they'll have to do it fast because, exponents.

I wonder what kind of HDD setup they (and AWS, and Azure, etc) are using.

A reasonable price point is that of a 10 TB HDD, you would need 430 new disks of that size everyday.

Would love to learn about what kind of contracts they enter in with hardware manufacturers.

The storage company most willing to explain what they're doing is Backblaze. Their blog is great. https://www.backblaze.com/blog/category/cloud-storage/storag...

Is there any tutorial type doc on how businesses use the years old data? I understand the use or hot data, need for reports, queries, etc.

Looking to beef up my knowledge.

Reply on news.ycombinator.com