Hacker News Reader

> Let's do the math.

Offers no math.

At retail, 625 16TB drives is $400000. This is about 2x the MONTHLY retail s3 pricing. Further, as we all know, AWS bandwidth pricing is absolutely bonkers (1).

I think your conclusion that S3 is "pretty good" needs a lot more math to support.

(1) https://twitter.com/eastdakota/status/1371252709836263425

The math should also include the price of the staff who babysit 625 spinning metal disks, who likely drive to a data center multiple times a week to swap failed drives. I shudder to think if this job fell in my lap!

Sure, but you actually have to go through the steps. The very first step, back of the napkin, indicates savings. More than enough to warrant a more detailed evaluation. Then you can start considering the more complicated factors (redundancy, staffing, power, ...).

My response was in the context of someone who didn't do any of that.

Also, my tongue-in-cheek response to you is: the price will be offset by the SRE engineers who were babysitting your AWS setup that you'll no longer need. (More seriously, I don't think finding quality sysadmins who enjoy this stuff is particularly harder than finding quality roles for any other tech positions).

Been there, done that, at all levels. I would much rather be working on a 10PB set of hardware racks, including all the drive replacements. When you factor in the costs of compute hardware (to make that useful), networking equipment, etc, it's trebled again the cost, and then again for the power, cooling, and cage space to run it all. The actual break-even point of running your own hardware is more like 2 years.

But it's not about price: It's about control, and it's about the expertise you gain from running all of that. If you have 10PB of data, you should have someone in-house who knows how to work with 10PB of data at a low level, and the best way to get that is to employ people at all levels to make that work. You gain significant advantage from having the direct performance data and the expertise of having techs whose 9-5 is replacing disks.

>>> My response was in the context of someone who didn't do any of that.

I did and you ignored all of it -_-

* To store 10+ PB of data.

* You need 15 PB of storage (running at 66% capacity)

* You need 30 PB of raw disks (twice for redundancy).

>>> At retail, 625 16TB drives is $400000.

That's only 10 PB of disks. That's about one third of the actual need.

Please triple your number and we will start talking. That's about 2000 disks and well above a million dollar.

You can't just call a supplier and get for a thousand 16 TB disks (or even a hundred). They don't have that in stock now. The lead time might be 6 months to get a few hundreds. They might not have 16 TB disks for sale at all, the closest might be a 12 or 14TB.

Handling large amount of hardware is a logistic problem. Not a cost problem.

With over 600 drives you would have at least 20 hot spares, and i don't think you'd have more than 2 drives fail per week if you don't have bad batches of them.

> 625 16TB drives is $400000

how much is the real estate cost of 625 drives and associated machinery to run it?

At a guess, AWS has an operating margin of about 30%, so you can approximate their cost of hardware, bandwidth, and other fixed costs as 70% of their sticker price. As a start up, can you actually get this price to be lower? I actually dont think you can, unless your operation is very small and can be done out of a home/small office.

Their margin on bandwidth is literally over 1000%. Quick google says that S3 costs 320% more than Backblaze (which, presumably, isn't running at a loss).

> At a guess

The comments in this discussion that try to provide actual numbers show a fairly lopsided argument against S3. The comments that are advocating for S3 aren't as detailed.

You can look at this at the macro level, as on comment did, and see that one 1.2PB RackmountPro 4U server is $11K. Yes, of course you still need space and power. But at least this gives us actual numbers to play with as a base (e.g. buying 10 of these is less than what you'll spend on S3 in a month)

At a miro-level. You can spend $650 on a 16TB hard drive, or $650 on 16TB for 2 months of S3. Now, S3 is battle-tested, has redundancy, has power, has a cpu, has a network card (but not bandwidth), and is managed - unquestionable HUGE wins. But the hard drive (and other equipment) come with a 3-5 year warranty. Now, the difference between $650 for the hard drive, and $12000 for S3 over 3 years, won't let you: get the power, rent the racks, hire the staff, and invest in learning ceph. But the difference between $400K and $5million will.

> one 1.2PB RackmountPro 4U server is $11K

An empty 4U server with 96+ bays looks like it will set you back ~$7k minimum. At $500 per drive (I have no idea what volume discounts are like) filling it with drives would be in the range of ~$50k. You'd still need RAM. And (as you noted) space and power.

I have no idea how the math ends up working out, but a 1PB appliance in working order is nowhere near as cheap as $11k.

>>> You can look at this at the macro level, as on comment did, and see that one 1.2PB RackmountPro 4U server is $11K

protip: $11k is the cost of the empty enclosure. disks are sold separately.

FWIW you can get great redundancy with far less than 2x storage factor. e.g. Facebook uses a 10:14 erasure coding scheme[1] so they can lose up to 4 disks without losing data, and that only incurs a 1.4x storage factor. If one's data is cold enough, one can go wider than this, e.g. 50:55 or something has a 1.1x factor.

Not that this fundamentally changes your analysis and other totally valid points, but the 2x bit can probably be reduced a lot.

[1] https://engineering.fb.com/2015/05/04/core-data/under-the-ho...

https://en.wikipedia.org/wiki/Parchive

Basically they use par2 multi-file archives for cold storage with each archive file segment scattered across different physical locations. Always fun to see the kids rediscovering tricks from the old days.

> If you talk to Dell/HP/other, they can advise you and sell you large storage appliances. Problem is, the larger appliances will only host 1 or 2 PB. That's nowhere near enough.

This is just incorrect.

If you talk to HPE, they should be quite happy to sell you the my employer's software (Qumulo) alongside their hardware. 10+ PB is definitely supported. (The HPE part is not required)

If you talk to Dell EMC, they will quite happily sell you their competing product, which is also quite capable of scaling beyond 1-2PB.

Most (all?) enterprise vendors will go well beyond 1-2PB.

Four years ago, one of the all flash vendors routinely advertised “well under a dollar a gigabyte”. Their prices have dropped dramatically since then, but the out of date numbers translate to “well under a million per PB”. That’s at the high end of performance with posix (nfs) or crash coherent (block) semantics. (Some also do S3, if that’s preferable for some reason)

With a 5 year depreciation cycle, those old machines were at << $16K / month per PB. Today’s all flash systems fit multiple PB per rack, and need less than one full time admin.

Hope that helps.

Always fun stumbling across another Qumulon on here :)

I've checked what I could find on Qumulo. It is software that you run on top of regular servers, to form a storage cluster.

It seems to me you're only confirming my previous point, that you need to invest in complicated/expensive software to make the raw storage usable.

>>> Then you're going to have to handle "sharding" on top of the storage because there's no filesystem that can easily address 4 racks of disks. (Ceph/Lustre is another year long project for half a person).

There's no listed price on the website, you will need to call sales. Wouldn't be surprised if it started at 6 figures a year for a few servers.

It looks like it may not run on just any server, but may need certified server hardware from HP or Qumulo.

AWS is ridiculously expensively at their scale, both for storage and egress. But the choice is not only between that and building a staffed on-premise storage facility.

You can compromise at a middle ground - rent a bunch of VPS/managed servers and let the hosting companies deal with all the nastiness of managing physical hardware and CAPEX. Cost around $1.6-2/TB/month (e.g. Hetzner's SX) for raw non-redundant storage, an order of magnitude better than AWS. Comes with far more reasonably priced bandwidth too.

Build some error correction on top using one of the many open-source distributed filesystems out there or perhaps an in-house software solution (reed-solomon isn't exactly rocket science). And for some 30+% overhead, depending on workload (you can have very low overload if you have few reads or very relaxed latency requirements), you should have a decently fault tolerant distributed storage at a fraction of AWS costs.

I agree that considering Hetzner is a good idea. I have used them often, never any problems, and very low pricing.

> You need to improve your CDN and caching layer.

Depends on usage patterns, but if this is 10PB of users'personal photos and videos, then you're not going to get much value from caching because the hit rate will be so low.

> If it's the bandwidth. You need to improve your CDN and caching layer.

What would you recommend for this?

(considering data is stored in S3)

Verizon and Redis has worked well for me.

> * To store 10+TB of data.

> * You need 15 TB of storage (running at 66% capacity)

> * You need 30 TB of raw disks (twice for redundancy).

Did you mean PB?

Corrected.

Very good advice!