Hacker News Reader

Comment by synicalx | Hacker News Reader

1EB with only 30k users, thats a wild TB-per-user ratio. My frame of reference; the largest storage platform I've ever worked on was a combined ~60PB (give or take) and that had hundreds of millions of users.

View on HN

Most humans don't handle sensor and simulation data for a living, though. CERN just so happens to employ thousands who do that for a living.

My frame of reference; the largest storage platform I've ever worked on was a combined ~tens of EB (give or take) and that had over a billion users.

Facebook?

Google. I can't imagine Facebook having tens of EB.

When experiments are running the sensors generate about 1PB of data per second. They have to do multiple (I think four?) layers of filtering, including hardware level to get to actual manageable numbers.

It depends on which experiment. We call it trigger system. And it varies according to each experiment requirements and physics of interest. For example LHCb is doing now full trigger system on a software side (No hardware FPGA triggering) and mainly utilizing GPUs for that. That would be hare to achieve with the harsher conditions and requirements of CMS and ATLAS.

But yes at LHCb we discard about 97% of the data generated during collisions.

Disclaimer: I work on LHCb trigger system

>1EB with only 30k users, thats a wild TB-per-user ratio.

33TB per user is a lot, but is it really "wild"? I can fill up well over 33TB of pro-camera photos in less than a year if I shoot every day. I'm sure scientists can generate quite a bit more data if they're doing big things like CERN does.

That's the scale of the universe, compared to data generated by humans

Reply on news.ycombinator.com