80M I/O Per Second with a Standard 2U System | Hacker News Reader

80M I/O Per Second with a Standard 2U System

spdk.io

85 pointspbalcer39 comments

> The system has 2 CPUs with 64 PCIe Gen 4 lanes each, giving us plenty of PCIe lanes to connect NVMe SSDs

Imagine if they used EPYC CPUs... 160 PCIe lanes.

They could have achieved this 'milestone', 1.5 yrs ago. Feels like an Intel ad to me.

Agreed. Intel Optane NVMe has been out for quite a while, we've been using them for at least 3 years now. Still, I'm love reading articles showcasing their performance. NVMe is such a game changer for storage I/O based systems.

What is the point of that speed if the rest of your system is a bottle neck and the drives will fail in a year if you use that speed continously?

I think we need to go back and look at cycles per bit (DWPD is confusing because you need to know the number of years and sometimes it's not mentioned on the product page).

Complexity can increase some performance but there is always a cost to complexity. Optane seemed promising because they increased the speed (that I personally will never need) and preserved the resiliance.

Time will tell, but I bought 30x new 50nm SLC 64GB X25-E drives from 2011 on ebay for $100 each that I will mount on 12x SATA motherboards (only got cable/power for 8+1 per board) instead of buying one 480GB Optane drive for $1500!

Cheaper, slower and maybe more robust? Atleast it is simpler!

Atleast SATA I can use in my laptops/desktops/consoles etc. Also nobody is looking at CPU usage of NVMe vs. SATA!

I did something similar on a smaller scale: "Caching" ssd drives which are not the best but have a huge amount write bytes left are widely available on ebay and I bought 8 200GB SAS3 drives for dirt cheap (under 200$ total, drives from 2017). Not the fastest drives but good enough for my purpose (overkill even).

What drives did you get?

Sun/oracle branded HGST SS300 drives (HUSMH4020ASS210), they have the specs in the ballpark of the SS800 drives in endurance (according to the scsi data pages from the disks) but read/write performance of the SS300.

Trying to google it it seems unclear to me what the product is... SSD or hybrid? Also what tech (SLC, MLC, TLC, QLC) and how many nm? This is the real reason why I went all in on the X25-E; there is no clear winner since 10 years... either it's super expensive and might be good or cheap and guaranteed crap!

Also the interfaces are all over the place, PCIe, SAS, SATA, NVMe (which is PCIe?), M.2 with double PCIe?! with all kinds of connectors...

We have a serious problem with storage of data when magnetic tape is still the real solution!

MLC SSD, Sorry for being unclear, googling it is difficult as they were never released to non DC providers (these are custom for oracle) so data sheets are hard to get (I only have partial data, rest is from the sas inquiry and mode pages). You can deduce some of the specs from the name as they are the same for all hgst drives:

HUS - hgst uktrastar standard

MH - MLC high endurance (4PB is claimed for the 200GB drives, verified this by looking at the lifetime counter and the bytes written)

4020 - max spec 400gb, device is 200gb

A - generation

S - 2.5" formfactor

S2 - 12GB/s (sas3), dual port

1 - probably means custom spec all my hgst former netapp/sun/oracle drives have this set to 1 both SSD and hdd

0 - crypto sanetize

Haven't seen them a lot on eBay last month but caching drives with low percentage writes do show up from time to time in batches.

$25 per 200GB MLC used... I'd rather pay $100 per 64GB SLC unused.

For reference, you can buy a 2 TB SSD for $200 today. Same capacity as the 30 old drives, for much less space and cables.

I'm lost on your comment.

What specific problem die you see?

It's much cheaper and easier to pay and use NVM than ram. You get best latency for databases across your data.

The prices are already super cheap.

I also not find it easier to buy a lot of small devices instead of one. Your NVM is doing the same thing you do but directly on the controller

Check out the latency difference between sata and NVM.

And yes CPU is idling often enough. Good thing if they have a little bit more io overhead but when you look at PS5 and Xbox, Sony solved this issue and Microsoft is bringing their new storage API to PC.

>I think we need to go back and look at cycles per bit

I've started playing around with new storage architectures for databases and with flash storage in mind. One thing I encountered that seems incredibly powerful, when combined with a log-based store, is micro batching of operations.

When you aggregate a few hundred/thousand transactions together and then append them as modified subtrees to the end of a log, you start to be able to say crazy shit like "Transactions per I/O operation or block"

When working with objects in the 512b range, I can easily hit 2 million inserts per second on a single Samsung 960pro using this technique.

We use a similar technique for TigerBeetle [1], a financial accounting database, to do a million financial transactions a second with full distributed persistence and reasonably low stable latency (< 20ms).

We have around 10,000 financial transactions in a batch and amortize away the distributed consensus (Viewstamped Replication), the many assertions, syscalls and O_DSYNC so that they are almost free.

Direct I/O helps to avoid flushing the CPU cache through memcpy's to the kernel's page cache for a roughly 7% improvement, and io_uring helps to amortize the syscalls themselves into a single io_uring_enter() for a doubling of throughput for small 4 KiB read/write units, but having a first-class batching interface from end to end is the biggest win by far.

[1] https://github.com/coilhq/tigerbeetle

This rack could be used for storage, not compute. You could add a Mellanox 400Gb NIC to the machine (assuming you have the required 32x PCI lanes for it), and then route that to your compute resource. This is a typical scenario.

Does this setup do 80M IOPS? I'm not sure what the point of your post is, everyone can make slow storage cheaply. The article is about fast storage which is also cheap (compared to similar solutions from flash storage array vendors).

>What is the point of that speed if the rest of your system is a bottle neck and the drives will fail in a year if you use that speed continously?

>I think we need to go back and look at cycles per bit (DWPD is confusing because you need to know the number of years and sometimes it's not mentioned on the product page).

This work is for people developing dedicated storage arrays, not your laptop. Datacenter optane is good for 60 DWPD. Given that it is most frequently used as a caching layer, it's extremely unlikely you'd exceed that in a well designed system.

https://www.intel.com/content/www/us/en/architecture-and-tec...

>Atleast SATA I can use in my laptops/desktops/consoles etc. Also nobody is looking at CPU usage of NVMe vs. SATA!

Literally every storage vendor in the market is looking at CPU usage of NVMe vs. SATA. There's a reason they benchmark in "a standard 2U system" and not a Dell Laptop or Desktop.

60 DWPD was the original optane development result, but the final product that we can buy is 10 DWPD.

My issue with DWPD is that you can have the firmware complexity pretend the drive is good while losing space... I prefer to have really solid "100.000 writes per bit" 50nm SLC flash memory cells with less magic.

If everyone is measuring NVMe vs. SATA how come I can't find anything interesting when googling? It's only popular articles about the high level difference, "the plug is different"... I would like benchmarks with apples to apples CPU usage while having alot of concurrent data in flight simoultaneously!

You realize you could have bought at least 30 terabytes of SSD space for the money you spent right?

If you wanted simplicity you could have bought a single 2 TB drive.

This makes for a great database server.

Is this the death of kubernetes? Can I now use one server instead of 20?

NVMe will allow your storage to be super fast, but it doesn't add any more compute resources. So unless your K8s cluster was all about IO, i'd say no...

I daydream about running our entire company k8s infra on a single 1U with 256 Epyc threads, a few terabytes of memory, and a bunch more of tiered storage. IMO this kind of miniaturization is amazing for container orchestration. Add a couple more, overprovision by 50%, let k8s handle the high availability, and you’ve got a datacenter in a small box.

I have nightmares of someone convincing my company to do something like that and then a having a hardware, network, or power failure.

Buy 2.

4. 2 each for 2 data centers.

And, at that point, you'll probably want an easy way to orchestrate work across your multiple nodes.

Probably so.

Oh for sure. I'm happy to keep it as a daydream. Cloud providers are better at high-availability than I am.

We run single node K8s. I do it for some non-prod apps. K8s deployments are a lot more easier to organize than a bunch of docker compose deployments and its also a great way to learn K8s internals, etc.

You probably already can; how much load are your systems under?

So what is the price for such a system

2x$8k for the CPUs + 16x$2.2k for the drives + rest of the system >= $50k

We've been using these systems for about 2 years now. We can put 32TB in a 1U box with dual Xeon Gold processors for a little under $20K. Our business is storing incoming market data and this architecture has allowed us to write bursts of up to 80 Gbps with ease. NVMes rock!

80M 512 byte I/O per second. That’s 10M 4KB I/O per second. That’s not groundbreaking.

They also did 25M 4K IOPS on the same system. Anyway, it's basically an ad for Intel® Xeon® and Optane®.

That looks close to the maximum rate of an optane 5800x... 1.5 million 4k iops * 16 devices = 24 million 4k iops. It also looks close to the maximum of pcie4, 16 * 7.8GB / 4K = 32M > 25M.

> it's basically an ad for Intel® Xeon® and Optane®.

Bingo.

No.Extending your reasoning, it would just be 1 40GB I/O per second. Each io request has a fixed cost, which is better measured with small requests. Bandwidth is not really relevant here.

Reply on news.ycombinator.com