Hacker News Reader

Comment by rjzzleep | Hacker News Reader

How long does it usually take to get consumer products of new PCIe specs? Fast PCIe Gen 4 is only just getting affordable. Like $350 for 2 TB NVMe ssds.

Also, I remember playing around with PCI implementation on FPGAs over a decade ago and timing was already not easy. What goes into creating a PCIe Gen4/5 device these days? How can you actually achieve that when you're designing it? Are people just buying the chipsets from a handful of producers because it's unachievable for normal humans?

EDIT: What's inside the spec differences between say gen 3 and 6 that allows for so many more lanes to be available?

View on HN

I love how exponential growth can be utterly terrifying or unfathomably amazing.

Just a few years ago I was trying to explain to an IT manager that 200 IOPS just doesn’t cut it for their biggest, most important OLAP database.

He asked me what would be a more realistic number.

“20,000 IOPS is a good start”

“You can’t be serious!”

“My laptop can do 200,000.”

> Just a few years ago

> “My laptop can do 200,000.”

Only now (PCIe 4 and very recent controllers etc) are the very latest top-end NVME drives hitting around 150k IOPS (which isn't stopping manufacturers from claiming ten times that; WD's NVME drive tests at around 150-200k IOPS and yet they claim 1M) and only in ideal circumstances...reads and writes coming out of the SLC cache, which typically under 30GB, often a lot smaller except in the highest-end drives.

Many drives that claim to reach that sort of performance are actually using Host Backed Cache, ie stealing RAM.

IOPS on SSDs drops precipitously once you exhaust any HBC, controller ram, SLC cache, mid-level MLC cache...and start having to hit the actual QLC/TLC. In the case of a very large database, a lot of IO would be outside cache (though certainly any index, transaction, logging, etc IO would likely be in cache.)

I would love to pick up from the 200k IOPS laptop quote and demo a RAM drive and then saturate the RAM drive into swapping - I don't know how you could do this on stock distros or Windows but it would make a great executive suite demo of the issues.

More than 500k IOPS was downright common for PCIe gen3 SSDs. 1M IOPS is totally achievable with one high-end gen4 SSD.

If you meant IOPS with a queue depth of one, or sustained write IOPS, then you need to specify those extra conditions before calling vendors liars.

No high-end drives implement the NVMe Host Memory Buffer feature, let alone rely on it for maximum performance.

You can buy an Enterprise QLC drive that is fully capable of 800k IOPS for reads. TLC drives that can do twice that are available. Those drives don't have SLC caching.

It really is something else. I used to run a scaled out Exchange system circa 2008, and we spent alot of time optimizing I/O. I think 150 iops was the target per disk performance.

There's not more lanes available. The generations are getting faster just by increasing the transfer clock rate, up to PCIe 5, and in PCIe 6 by increasing the number of bits per transfer. The way they doubled the speed every generation was pretty basic: the timing tolerances were chopped in half every time. The allowable clock phase noise in PCIe 4 is 200x less than in PCIe 1. The miracle of progress, etc.

That miracle is somewhat over. They're not going to be able to drive phase noise down below 1 femtosecond, so 6.0 changes tactics. They are now using a fancy encoding on the wire to double the number of bits per symbol. Eventually, it will look more like wifi-over-copper than like PCI. Ethernet faster than 1gbps has the same trend, for whatever it's worth.

Speaking of which, when is 10Gbit Ethernet coming to laptops? Most have even lost the port ffs.

Many laptops have a thunderbolt port which serves a similar purpose. On TB4 I get 15gbps in practice, and I can bridge it to ethernet using either a dock or a PC (I use a mac mini with a 10g port to bridge TB to 10ge).

I think the real question is when will 10G Ethernet USB4 dongles drop to a reasonable price like <$150.

A USB4 peripheral interface is $15 and a 10g ethernet IC is like $80 at least, plus a little box, SFP+ thing, other physical items. Seems hard to get the price down that low and still make a profit. Plus the buyer still needs an expensive cable on both sides.

A 10G PCIe card is only $100 retail so $150 seems fair for an external version. Newer dongles will include a captive USB4 cable. And consumers use base-T not SFP+.

10GB Ethernet requires at least CAT6 or CAT6a cabling. CAT5e is insufficient. For home use, that means that the newer standards of 2.5GB and 5GB which do work over 5e are more reasonable for personal computers.

Depends on the length.

Who cares about 5e? Maybe if it's embedded in your wall, but how many people have done that?

A 25 foot cable on monoprice costs $6.49 for cat5e and $7.99 for cat6. It doesn't matter at all.

I could get cat8 for $22.49! Compared to what you're plugging that into for 25/40Gbps that's nothing.

The last batch of cables I bought was 5x 10 foot cat7, for $22 total. (You're not 'supposed' to put normal plugs on cat7 but nobody really cares.)

Never. Power consumption for 10Gbps Ethernet on copper is huge compared to 1 Gbps, so it is not sexy for laptops. Consumer switches with 10 Gpbs ports don't exist - price per port is way too high versus 1 Gbps. Even 2.5 & 5 Gbps switches are rare and very expensive, so the 2.5Bbps Intel and Realtek Lan on Motherboard chips are there, but not useful.

At the same time, 10 Gbps on fiber is low power and great speed, but the form factor of SFP+ is simply too big for any laptop of 2022. In theory USB-C 3.2 @ 10Gbps to SFP+ fiber adapters are possible, but fiber is not popular outside server rooms, so there is no market for it.

As I said, never. It is not a matter of PCIe speeds, but a technology one: copper is power hungry and requires new cabling, fiber has no cabling.

10GbaseT power requirements are dropping every year as the MACs get more efficient. Its also of course dependent on cable length. Besides saving power it has also enabled 10GbaseT SFP+'s in recent years.

So, there probably isn't a good reason for not putting it in a laptop. Like wifi, just speed shift it based on load, etc. AKA see 802.3az

So, the old max power draw models from 15 years ago, don't really apply entirely.

> Power consumption for 10Gbps Ethernet on copper is huge compared to 1 Gbps, so it is not sexy for laptops.

How often do you have your laptop plugged into networking but not power? And if that's an important use, I could see manufacturers spicing it up with some power over ethernet to not only fix but reverse the problem.

Though it feels like ethernet ports on laptops are already unsexy at any speed. And you could have the 'balanced' power profile limit the port to 1Gbps on battery.

Ethernet/power brick combo on new macs come to mind ...

> Consumer switches with 10 Gpbs ports don't exist - price per port is way too high versus 1 Gbps

Mikrotik CRS305 ($150, 4x SFP+) & CRS309 ($250, 8x SFP+) w/ as many S+RJ10s as you want.

Mikrotik quotes 2.7W per transceiver at 10GBASE-T.

https://www.servethehome.com/mikrotik-crs309-1g-8sin-review-...

https://mikrotik.com/product/s_rj10

So yes, it's a bit hot, but definitely something you can get done well under $500 if you want copper.

Those Mikrotik SFP+ RJ 10Gbps modules get VERY hot. Idle temp is usually in the 60-70C range in my experience. They have guidance about not putting them next to each other in switch ports: https://wiki.mikrotik.com/wiki/S%2BRJ10_general_guidance#Gen...

I know about that option, but the price per port, including RJ45 transceiver, is huge compared to 1 Gbps ports. Also it is limited to 4 ports only - even for home use it is not enough, all the switches in my home (more than 3) are 8 ports and above.

Well, the idea is that you only wire trunks, wifi APs, and SANs w/ 10GbE, and light a select few wall jacks that matter.

~$120 per port (assuming 309s, the recommended interleaved spacing, and amortizing the switch cost over # ports) isn't very expensive if (a) you need the capability & (b) already have copper in the walls.

Drywall finishing along will run you that much. :) But yeah, new builds should definitely be fiber.

> They are now using a fancy encoding on the wire to double the number of bits per symbol.

Wasn't there some network standard that used that trick as well?

Yes, "modern" ethernet uses 56 Gb/s (and soon 112 Gb/s) serdes channels with PAM4 modulation.

Similarly HDR Infiniband also switched to PAM4.

they can easily extend the number of lanes. there are many examples of much higher speed serdes point to point.

> How long does it usually take to get consumer products of new PCIe specs?

Like 2 years.

When PCIe 3.0 was getting popular, 4.0 was finalized. When 4.0 was getting popular, 5.0 was finalized. Now that PCIe 5.0 is coming out (2022, this year), PCIe 6.0 is finalized.

PCIe 5 was standardized May 2019 and you could buy it at retail in late 2021. 2 years good rule of thumb.

There was a much bigger gap between 3.0 and 4.0. PCIe 3.0 was available with Sandy or Ivy Bridge, so 2011/2012. PCIe 4.0 was introduced with Zen 2 in 2019.

We seem to be back to a faster cadence now however.

The large delay between 3.0 to 4.0 was a gap between specifications (2010 to 2017) not a gap between specification to implementations (2017 to 2019).

With the rise of GPU-compute, a lot of the supercomputers are playing around with faster I/O systems. IBM pushed OpenCAPI / NVLink with Nvidia, and I think that inspired the PCIe ecosystem to innovate.

PCIe standards are including more and more coherent-memory options. It seems like PCIe is trying to become more like Infinity Fabric (AMD) / UltraPath Interconnect (Intel).

One bus to rule them all! Would make some sense and if it's fast enough, not saying we will see memory in PCIe form factor, though, size of memory progression, maybe down the line it may go that way.

With sufficiently fast memory bandwidth, does dedicated GPU memory become irrelevant? AFAIK discrete graphics on laptops still share the main memory, but at what point does that become a standard setup on desktops too?

DDR5 is far slower than GPU VRAM.

If anything, the higher-bandwidths of future PCIe-specs would allow the CPU to access the dedicated GPU VRAM at its full speed, rather than PCIe-limited speeds.

> It seems like PCIe is trying to become more like Infinity Fabric (AMD) / UltraPath Interconnect (Intel).

So far CXL is pretty strictly a leader/follower type approach, not really suitable for a more symmetric relationship like you have in a multi-socket system. But maybe one day..

>How long does it .......

It is not just about getting a product out ( i.e PCI-E 6.0 SSD ), but also the platform support. ( i.e Intel / AMD Motherboard support for PCI-e 6.0 )

Product Launch are highly dependent on Platform support. So far Intel and AMD dont have any concrete plan on PCI-E 6.0, but I believe Amazon could be ahead of the pack with their Graviton platform. Although I am eager to see Netflix's Edge Appliance serving up to 800Gbps if not 1.6Tbps per box.

I recently bought a few Zen 2 and Zen 3 HPE servers, and found out only via trial and error that HPE sells Zen 2 servers without Gen4 motherboard support!

It seems they took the original Zen motherboards with Gen3 and just swapped out the CPU. Only the Zen 3 has a refreshed motherboard. Makes me now check things more carefully to be sure.

Same, worse my vendor sent me an outdated motherboard that officially didn't support the Zen2 CPU and it randomly wouldn't boot, and then they sent me a patched BIOS so it would boot consistently.

We needed that machine yesterday, and it's seen maybe 2 minutes of downtime in the past couple years, so I couldn't send it back. My fault for not specifying I wanted the .v2 of the barebones when ordering it over the phone I guess..

I've not done PHY development personally, but these interfaces are called SerDes. SerDes is short for Serial-Deserializer. Outside of dedicated EQ hardware, everything on the chips are done in parallel so nothing needs to run at a multi-GHz clock.

I think that these days there's a lot of convergence going on - everything is essentially serdes in some form - some chips just have N serdes lanes and let you config them for PCIe/ether/data/USB/etc as you need them, much as more traditional SoCs config GPIOs between a bunch of other functions like uarts/spi/i2c/i2s/pcm/...

> How long does it usually take to get consumer products of new PCIe specs?

Personally I'm expecting this spec to drive pcie 5.0 adoption into consumer space.

Tbh consumers dont need this througjput. But given that consumer space has remained stuck around 20x lanes off the cpu (plus some for the chipset), the 5.0 and 6.0 specs will be great for those wanting to build systems with more peripherals. A 1x 16GBps link is useful for a lot.

I'd be leery of dismissing the potential consumer demand. That much throughput could be put to good use for a myriad of personal and business functions, and software trends to filing whatever hardware can provide. It's like every prediction about users not needing X amount of ram or cpu or dpi or network or storage space.

Having that much throughput suggests paging and caching across multiple disks, or using giant models (ml or others) with precomputed lookups in lieu of real-time generation. At any rate, all it takes is a minor inconvenience to overcome and the niche will be exploited to capacity.

> Having that much throughput suggests paging and caching across multiple disks, or using giant models (ml or others) with precomputed lookups in lieu of real-time generation.

Those sound to me like situations where latency will be the bottleneck, even on something like PCIe 3.0 x4.

I can currently only name one consumer use for these super high speed data transfers, and that's loading 3d assets and textures from SSD to GPU in real time based on where you're looking.

Signal integrity is becoming a serious issue for these higher speed links. So it's a compromise between how many traces you need vs. the signal speed.

For the server market, GPU's and other accelerators, high speed NIC's and SSD's have an unending appetite for bandwidth, so there I expect a lot of interest. But I think we'll see an increasing gap between this and the consumer market, for better or worse.

PCIe Gen5 is now available in the latest Intel desktop processors. There are very few lanes, so it can really only run a single GPU, but that's covers a lot of the potential market.

Look like Desktop Chipset 600 series just supports gen 3 & 4[1] and PCIe gen5 ports are only available on AlderLake Desktop[2] not Mobile[3] processors.

[1] https://ark.intel.com/content/www/us/en/ark/products/series/...

[2] https://ark.intel.com/content/www/us/en/ark/products/134598/...

[3] https://ark.intel.com/content/www/us/en/ark/products/132214/...

Reply on news.ycombinator.com