Hacker News Reader

Comment by maybeben | Hacker News Reader

i mean, they also have one of the largest ceph deployments. anything is scalable with no budget.

slide 22 states that the cost is 1 CHF/TB/month (on 10+2 erasure coded disks), though it would be interesting to do a breakdown of costs (development, hardware, maintenance, datacenter, servicing, management, etc..)

1 CHF/TB/month is a bit expensive for storage at that scale, so it would definitely be interesting to see what they're spending the money on and what they are (and aren't) counting in that price.

No budget often tags along with no accountability

Tape backup, accessibility, networking, availability... At 1CHF/TB that's a lot better than my local university still charging >100x that for such services internally

Economies of scale in storage are significant. Also, I don't know why you put up with your university charging 100x that when you can store things on AWS for $5-10/TB/month (or less). That comes with all the guarantees (or more) of durability and availability you get from the university.

Don't forget bandwidth costs at this scale are significant to say the least.

I assume most of that exabyte is stored in 2-3 datacenters, in which case the bandwidth cost is actually relatively small. Downloading it would cost a fortune (or take an eternity), but if it stays in the datacenter (or stays in AWS), bandwidth is cheap.

They probably consume Panasas, IBM, DDN, and BeeGFS gear and licensing too.

Nop.

Most internal data is spread between Ceph and home-made distributed storage system named EOS (https://indico.cern.ch/event/138478/contributions/149912/att...) running over commodity hardware.

The only commerical-backed storage system is the long term storage tape system. Still it has an home-made overlay API over it to interface with the rest of the systems.

Good god no. Nowhere near anything so crass. CEPH and EOS all the way

All their important "administrative" stuff (think Active Directory/LDAP user database, mail and other essential services) run on proprietary storage systems from a commercial vendor (not IBM though), with enterprise support and all.

At least that was the case a few years ago when we last talked to one of the heads of IT at CERN, but I guess it hasn't drastically changed

Reply on news.ycombinator.com