Hacker News Reader

Comment by roadbuster | Hacker News Reader

Not much tech content here. They racked a dense array of drives in a JBOD. There's no description of a complex or sophisticated ingestion engine, system for distributing data across the array, file system, error correction, nothing.

An Exabyte of disk is definitely eyebrow-raising, but companies like LinkedIn had an exabyte of storage in 2021 just in their HDFS clusters.

https://www.linkedin.com/blog/engineering/open-source/the-ex...

View on HN

It's hard for me to grok that LinkedIn (or any company, really) has more than 1 exabyte of analytical data.

That's about 1GB per user. If you imagine them storing every interaction anyone has with the site-- every file upload, every page load, every mouse movement-- and throw in some duplication-- it's not completely impossible.

Do these logs ever get pruned? Is it worth knowing that 10 years ago, Johnny took 1.5 seconds to click the upvote button? Or is it just easier to keep it, imagining some what-if value extraction?

Isn't 1gb per user absurdly large?

Makes you wonder how much facebook and google have per user.

not for gmail, or photos, but it does seem that way for linkedin, probably, unless they like storing uncompressed logs.

If I remember correctly, JBOD means that if a drive fails the data on the drive is lost. If you have so many disks then you have an expected fail rate of roughly 1.7% per year (according to the popular drive statistics from back blaze).

Isn't it a bit harsh to lose 1.7% of data every year? Why not a dead simple RAID that can tolerate the loss of a disk without data loss?

Redundancy is handled at a higher layer. They are probably using ceph on top.

A JBOD is as the initialism, they're Just a Bunch of Disks. They are simply enclosures that allow a large number of disks to be hooked up to a server. Any RAID or whatnot would be done on the software side, with Ceph or something similar as another reply states.

More information on the software side of CERN storage can be had here: https://indico.mathrice.fr/event/143/contributions/174/attac...

Reply on news.ycombinator.com