PCIe 4.0 Card Hosts 21M.2 SSDs: Up To 168TB, 31 GB/s

This is No Such Agency who would buy as many of these as could be produced...

People like to throw the innuendo around but the NSA's pathetic little datacenter is something that you would lose in a corner of a real datacenter operated by a real hyperscale system like Amazon or Google.

PCIe switches just shouldn't be so dammed expensive. A decade ago there was a lot more market competition but now there are, what, two companies with chips?

It has gotten a good bit harder to build, especially with so many of the tricks & tight timings in PCIe 5 and 6, but the lack of market competition has made getting any kind of parts at all much much more expensive.

This actually sounds like it could be a nice mid-range product. For lots of money, you can get a fancy motherboard and enclosure that routes a ton of CPU PCIe lanes direct to the NVMe drives. This ends up with a lot of performance per unit storage, which one might not want.

With a card like this, one can get a ton of high-speed (much better than SATA but not as fast as direct NVMe) storage in a regular machine.

> Almost like it's custom designed […]

https://www.microchip.com/en-us/product/PM42100

It's a standard COTS part.

Coincidentally, the PCIe 5.0 variant is in "Samples available", i.e. not full production yet, which is very likely the reason for this card only being PCIe 4.0.

https://www.microchip.com/en-us/product/PM50100

Absolutely. I keep saying to people "it doesn't matter, everything is fast now".

Gigabit fibre to the home, NVMe that is way faster than RAM was not that long ago, CPUs in phones that make old desktops look like toasters.

The disconnect is that the numbers feel huge in comparison, and what my computer can do for me really, is not hugely different.

I marvel at 5GB/s mass-storage, but then I groan at the knowledge it will be used to run a denornalized Postgres database table containing JSON blobs for queries that will do full tablescans because apparently dropping $Lots on a high-end storage system is preferable to learning how to do things properly.

Sometimes you have to move 15TB about and then nothing is fast enough.

At 5GB/s, that would take nearly an hour and 5GB/s would be the fastest part of the trip. If it has to land on tape or travel through the net, it's going to take days.

Its a shame all our data is sent/received over HTTP these days, otherwise I'd be excited about it

Got a VIC-20 for my 13th birthday in 1983. The local TV shop sold Commodore hardware and hosted a BBS that my little nerd cronies and I connected to at 300 bps. All of us knew the owner who ran the store and mentored us and hired some of us when we got old enough. On one visit he took me back, behind an actual curtain, and showed me the BBS machine. If you know the 1541 Disk Drive, you know. Well this beast had a Commodore branded 5 MB hard disk connected to a C-64. In my memory it was 6” by 8” by 24” long, with an enormous power supply, and must have cost thousands of Reagan-first-term US dollars.

Two or three years later another mentor hired me to put a 33 MB hard disk into his IBM PC. Not a clone. My memory tells me it was a DOS imposed limit, those 33 MB: the biggest drive available. I managed to plug the connector in upside down and released the magic smoke. That was a many-hundreds-of-dollars mistake. (And a good lesson in patient mentoring.)

In 1991 I obtained a used 80 MB drive (half height!) to put into my own PC XT clone, via a local Usenet group. I set the volume name to $1_PER_MB because going under that threshold was so impressive.

Those are my reference points for storage.

Yes and no. Throughputs are crazy high these days but the rate at which they increase is slower than the rate at which compute increases, by a lot. So if we're discussing in a relative sense, then there's a growing divergence and thus one can argue that I/O is getting "slower." This is actually one of the major topics in HPC discussion and why there have been so many crazy hacks. Things like flashbuffers are pretty much essential these days. Even if you're doing multi-node ML training you see pretty big differences using infiniband due to the frequency in which nodes need to communicate (there are regularization interval tricks too). In scientific computing this is a big limitation to our ability to visualize at high resolutions and is why in situ visualization is growing popular.

As far as consumer hardware and consumer usages, yeah, everything just feels fast though.

It's starting to become difficult to comprehend much of it now. My ISP is pushing me to replace my 300 Mbps service with 2Gbps, but I never even saturate what I have. Maybe if I were a gamer with huge downloads it would make sense to upgrade.

Talking size and not speed, to be honest without joking, I still marvel that I have a 64Gb USB stick. 64Gb! HUGE!

In my humble opinion some home NAS use cases are beneficially converted to Thunderbolt. It's a lot more practical than the faster varieties of ethernet. If you have two hosts that need fast access to the NAS you can do it with TB4, and one of those hosts can re-export the stored resources for applications with lesser performance requirements, over SMB or iSCSI or whatever.

u.2 NVMe uses 2.5" bays and SATA Express connectors to offer up to 4 lanes of PCI-e or two lanes of SATA. That's probably where most of the enterprisy NAS is going.

I really need to finish my YouTube video on this...

-- -----

So, the answer is you don't need a lot to get big impact.

Let's take some NICs like the Intel XL710-QDA1. These are about $250 used. This is a 40Gbps NIC using just PCIe 3.0 x8. I used DACs to drop power consumption through the floor, and to reduce latency. All of this is presented via ISCSI, and with jumbo frames to eke out a few extra percent throughput. If you're using passive DACs, figure 4W of power per port for the NIC, plus another 1.2W at the switch (assuming four SFP+ fan-out), for one end of that connection. You could also just direct connect between the server and the client.

At this point, you can basically shove older prosumer (say 980 Pro) PCIe 4.0 x4 NVMe sustained transfer over the network. Granted, if you're outside of that STR use case, you'll fundamentally be limited by IOPS. Figure for every 40Gbps, you can throw up to 1,200,000 4K IOPS worth of data over the wire.

If you hit the IOPS cap, increase your link speed. A PCIe 4.0 x16 card can handle two 100GbE ports just fine. Note that as you increase IOPS, you'll eventually hit your operating system's IO scheduler limits somewhere in the 10M-15M IOPS range.

-- -----

The question then is actually keeping the NIC fed if you're going over the network. If you're local, it's at least much easier.

First you likely have data buffered in memory for reads. So RAM will cover you there. For writes, you probably want a FUSE pass-through filesystem on NVMe in front of your real backing store (if it's disk, and you're not pure NVMe), or alternatively a writeback cache. The idea here is that pass-through filesystem is basically a storage tier that sits in front of another volume (or volumes) in an entirely transparent manner, so it still appears as if you're writing to a volume that might just be a RAID with a large amount of disks, but instead it's being written to the (presumably mirrored or striped+mirrored) NVMe first to move it to the final destination later. Alternatively, you could also double it as additional read cache, if you don't need to use is all as a write cache too.

-- -----

That's basically what I've done. I have QSFP+ NICs in the storage server and my HEDT (10GbE, 2.5GbE and 1GbE in the rest of the lab hosts), a mirrored NVMe cache/ingest tier, and a boatload of 16TB HDDs in ZFS across two ZFS volumes behind it. This is further backed by 128GB of ECC memory for ZFS ARC, and all powered by an AMD Ryzen 7 PRO 5750GE that maxes out at just under 39W. The whole system, with the NIC, HBA, two 2TB NVMe drives, eight 16TB HDDs, SATA DOM, 128GB ECC, 8-core 35W TDP CPU, and onboard BMC idles at about 43W, and loads that can primarily hit the cache, sits in the 60-65W range, with an absolute system peak under 140W. These figures can be confirmed by a metered PDU.

This gives me ~80TiB of usable redundant storage, with native prosumer PCIe 4.0 x4 NVMe performance, over the network... with an idle of 42-43W that bumps up to 60-65W for most work.

You can do a passive x16 -> 4 x4, iff your board supports pci-e bifurcation. Theoretically, you could do x16 -> 8 x2 also passively, but I haven't seen bifurcation go down to x2. PCI-e switches are probably too expensive for anything active though.

I'm gonna claim that this card isn't aimed at enterprise use-cases that need this kind of service. I'd put it along the lines of "nearline SAS" HDDs, aimed at non-critical applications where you care more about bulk capacity than reliability.

Relatedly, M.2 SSDs are inherently slower than the same pile of silicon in an U.2/2.5" form factor — the power/heat budget is noticeably lower.

> No powerloss protection advertised to OS

How can the OS (I'm interested in Linux) know about this feature?

Note that a Thunderbolt 4 Hub can be a bottleneck:

https://eclecticlight.co/2023/02/21/thunderbolt-4-hubs-can-s...

Same issue here, found anything good?

If its indeed PM42100 based then you will see separate drives.