Hacker News Reader

Comment by kache_ | Hacker News Reader

The scale at which Meta operates at really boggles my mind. I work with an ex facebook guy who was on the infra side of things and the numbers he told me.. I couldn't even imagine. And I'm working on the order of magnitude of 100m/h, but still, completely different set of challenges.

View on HN

Same. I remember asking one guy at FB the process to ask for a new server. He said he can't even open a request for anything less than a thousand boxes. The largest fleet I'd worked on at that point was 12... different worlds.

I mean, that's not true in the general case. That'd be incredibly wasteful.

(Work at Meta, mostly on capacity)

Yeah, I guess it probably is some sort of criteria that requires a formal request? The resource under that number might be automatically provisioned via "freebie quota" per team. I don't work for Meta but I believe most of the big techs work in a similar way.

Curious - do you know what he could have been referring to? SRE position, 6-8 years ago.

A bit before my time, but maybe a physical capacity order, to a non fungible pool. The infra is quite different now (mostly to avoid this kind of inefficiency)

I once read that Facebook was opening 2 or 3 massive new data-centers in the US for the purpose of hosting stale content.

You may have posted a photo 7 years ago, and statistics show that basically nobody ever revisits it. However, in case you do, it needs to be there. So these enormous buildings do basically nothing, but still need to be there.

It makes me wonder how it can go on like this. Users only keep adding content and never remove it. The income per user cannot grow forever, storage cannot get infinitely cheap, the model has to break one day?

There's no meaningful benefit to dedicating any amount of DC equipment just to stale content. Those are spindles (and networks) that could be taking meaningful hot reads and writes, and colocating stale and hot data is generally a better use of capacity than concentrating hot data in fewer locations.

What you say totally makes sense but still, even if new media takes more storage space, the accumulated stale date in the long run will win.

Or maybe I'm underestimating how much space newer material needs?

Exactly. And that doesn't even take into account higher res photos and 4K video. I remember the staggering statistic where just Instagram sees 100 million photos added per day, every day. And that was years ago.

Instagram does not guarantee photo quality. They can resize photos anytime they want. Eventually they turn 10 year old photos that have not been viewed in 9 years into thumbnails or just delete them.

I looked back at some photos from Christmas in Facebook Messenger and they looked noticeably degraded in quality.

That is just Instagram, with photos. Imagine Youtube.

And again I have been saying this since ~2015/16, we dont have any meaningful roadmap for cost reduction on storage, whether that is Hot as in NAND, Bulk as in HDD, or Cold as in Optical Disc. I dont see 2TB SSD dropping below $100 in next 5 years, or 10TB HDD below $120.

Remember when Google promise infinite Gmail storage?

> we dont have any meaningful roadmap for cost reduction on storage

Then maybe we could:

- stop encouraging users to post shit just so we can track them

- stop tracking them which requires many data points and a lot of processing power (for 0 benefit for the user or society at large)

- stop the copyright non-sense and actually use hyperlinks instead of reuploading the same content 500 times across platforms? maybe even do content-addressed storage (Bittorrent/IPFS) who knows?

The amount of ephemeral content (stories, expiring content) being created/shared has a larger impact on capacity and provisioning than you might expect.

This isn't how scaling works though. Across all applications the hot data growth outpaces the cold.

So if you're designing capacity for exponential growth, the future point at which you stop experiencing exponential growth and only have to worry about roughly linear growth is a much easier problem to solve.

Reply on news.ycombinator.com