For some reason, wrapping my mind around this is like trying to fathom the Grand Canyon or the scale of the universe.
You and me both. It's made even worse when you learn that the first-level trigger removes 90% of the input stream before passing it to 2,000 computers, which in turn select only 0.2% of that data for storage and further analysis.
Fermilab in Batavia takes data from CERN over a 40Gb/s optical link (CMS Tier1, CERN CMS is Tier0); when I was there several years ago, we were staging the data to ~500TB of spinning disk in Nexans and cold storage were 2-3 Storagetek tape libraries the size of school busses.
Computing and particle physics is where awesome meet.
CERN and its associated teams (CMS and ATLAS) are awesome. They are the only outfit outside of Google that seems to be handling data management and computing properly.
Not to disappoint you, but I could tell you horror stories about my time working on the CMS team at Fermilab. Horrible, horrible stories.
I hope the CMS and ATLAS teams onsite at CERN were much better.
I would be curious to hear some of these stories.
I'll have to put a blog post together about it. Extremely poor planning, no accountability, a boss who was verbally and emotionally abusive to a co-worker (the reason I quit after only a year there and went back to the private sector).
ATLAS guy here. I agree with @toomuchtodo, we're stuck in the stone age. It's basically just batch processing and lots of copying data around for no reason.
You could say the same thing about google's backend (batch processing and copying data), but it seems to be working for them as well.
In the future, the Square Kilometre Array is forecast to require 300-1500 PB/year of archiving, somewhere between 10 PB/hr and 1 EB/day of raw data. (That's 3-12 TB/s...)