Hacker News Reader

Comment by kondro | Hacker News Reader

Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB.

I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.

View on HN

There are 61.44 TB NVMe drives (best price I've seen right now is ~6200. They were ~4800 earlier this year). You can have a 1U server with 32 E1.L slots so you should be able to fit ~1.9PB raw storage into 1U for a little over $200k. Don't know how business financing works, but at 8% interest with a 5 year amortization, that's a bit over $4k/month.

Do you have any good recommendation for such 1U server with 32 slots? Thanks

Supermicro https://www.supermicro.com/en/products/nvme?pro=formfactor%3...

Shows 2U in order to fit 32 drives, but hopefully you can find a 2U slot for this absolute monster of a storage box if you have that need.

This one is 1U and supports dual socket CPU and 6 TB RAM:

https://www.supermicro.com/en/products/system/1U/1029/SSG-10...

I've never done procurement so can't really speak to recommendations. I just see that the products apparently exist.

There's also 32 drive 1U JBOF enclosures for more expansion:

https://www.supermicro.com/en/products/system/1u/136/ssg-136...

Our ops team actually wanted to do this, but we on the project have nightmares from putting 1PB of database on a single host ><

Nimbus ExaDrive is 100TB [1].

[1] https://nimbusdata.com/products/exadrive/

At the moment they're just paying someone else to buy $5000 SSD's and run a database on them at many X markup.

There is no upper bounds to economy of scales. Maybe there is for the cents per GB of raw storage, but power usage, security, rent, and everything else scales too, and few of them have upper bounds on economy of scales.

Economies of scale generally have upper limits. Often when you approach the largest scale the existing market will supply you essentially need to become your own supplier which then runs into span of control issues. The organization needs to become competitive in that new market or their costs increase.

Keep scaling and eventually vertical integration ends up looking like a Soviet style planned economy. Your remote mining town needs some way for people to get soap etc so you open a store with it’s own supply chain etc etc.

Yes, there is an upper limit, and that is the entirety of global economy. But that assertion is as useless as it is true.

Hardly the global economy, even approaching a global monopoly for something tends to hit such limits. Few companies deal with this today because most markets are fragmented.

But even at the 30% marketshare the iPhone has been dealing with these issues for a while. They just can’t buy 200 million volume buttons or whatever off the shelf. Now imagine what happens if one of their suppliers would fail days before the phone launches. They have such tight integration not just with the manufacturer’s process but also their finances because Apple simply can’t get replacements at scale on short notice. And remember that’s at ~30% market share, it just gets worse after that.

Outsourcing doesn't mean off the shelf. Apple still works with third parties to manufacture parts as they can leverage the scale of their overall operations to do it cheaper than it would cost apple to do.

They aren’t just working with from a design or quality control standpoint that’s normal enough, they are also purchasing equipment for 3rd party companies to use which isn’t. At this point they need to be world class experts in everything from batteries, sensors, software, and processors to glass.

Tesla does some stuff in house because they can, but they just didn’t have any options when it came to scaling battery production. They hit the limits of what the market could supply without getting directly involved.

Why are these companies going so far as purchasing equipment for 3rd parties instead of doing it in house?

Several reasons, both from Apples perspective and the supplier’s.

Suppliers don’t want to purchase a bunch of equipment and scale their manufacturing capacity for a contract that could go away at any time. So when Apple shows up they are going to price that risk in to their bid or sometimes just outright say no. However, by Apple buying that equipment those risks suddenly get reduced.

Apple on the other hand has huge cash reserves and isn’t that price sensitive. Plus they want to avoid all the issues bring yet another component in house.

That is a long way of saying "economy of scale".

If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.

> or with backups take a while to restore a PB and a way to take a hot backup without impacting the service that by itself is a task or snapshots which is more disk

> 1:1 replication Depending on the amount of writes could be a ton of extra disk and a bucket for network cost

These systems support zero-downtime snapshots. You tell it to snapshot, it instantly snapshots, you can run a differential/incremental backup at great speeds. Your RAID controller is already caching the hot data, so the impact is minimal.

Except network cost there's no extra disk required. It's just broadcasted writes consumed on the other hand.

These boxes are not dumb JBODS. They support their own replication/backup subsystems, so everything is transparent.

Resilient and fast from a disk perspective, but in practice massively bottlenecked by the fact that Sqlite can only have 1 writer at a time.

StorageReview plays with 2PB flash machines all the time https://www.youtube.com/watch?v=UQMKtlIjeuk

1PB in a rack with spinning rust + flash buffer has been easy for years now.

Reply on news.ycombinator.com