Hacker News Reader

Comment by nicktelford | Hacker News Reader

The huge number of PCIe lanes is pretty exciting to me. For consumers, the limiting factor in PCs these days tends to be I/O. With this, you could have 4 PCIe M2 SSDs (that's a bit under 16Gbps bandwidth, assuming RAID0), and still have 44 lanes (the number in the top-end Intel CPU) to spare for GPUs and other peripherals.

View on HN

Consumers tend to be limited by not having many PCIe SSDs in RAID0?

Who are these people and what do they need galactic disk bandwidth for?

Enthusiasts tend to be gamers, and once you have a high end machine, most games tend to be limited on disk bandwidth, which affects load times.

By consumers, I meant enthusiasts, since that's who the Threadripper (X399) and Intel's Extreme Edition (X299) platforms are aimed at.

That sounds frivolous versus a single high-end PCIe SSD. Wouldn't be surprised if the improvement in real-world game load times wasn't even perceptible.

>That sounds frivolous

Maybe, but that's what being an enthusiast is all about :) also assets (models & textures) need to be loaded from disk during game-play which can result in stutter when gaming. Faster IO would help here.

Our baseline of a single high-end PCIe SSD isn't slow, though. This comparison is ultra high bandwidth vs ultra ultra high bandwidth.

A lot of processing happens when games load. Once you have sufficient storage bandwidth and latency, load times flatten out. It seems that "sufficient" is SATA; above that, there is little difference in game load times:

https://techreport.com/review/29221/samsung-950-pro-512gb-ss...

I wonder if game developers are actually disincentivized from qualitatively improving load times past a certain acceptable threshold, lest their AAA game be perceived as lightweight.

I doubt it. Gamers have been complaining about long load times since almost the dawn of time, and I think one with as little as technically possible could make it a selling point.

The poor market penetration of PCIe SSDs and their high price per GB is a strong disincentive for game developers to store data in lightly-compressed formats that require minimal CPU processing to load and use. We might see some improvement in using multiple threads to load game data now that CPU core counts are on the rise at the high end.

Reminds me of the first time I played Kings Quest from a RAM drive. I first played the game from a floppy disk, and it ways really nice to have everything load instantly!

Enthusiast here - can confirm, an array of nvme's makes a huge difference over single standard SSD. I set this up last year, check the benchmark results, it's night and day.

http://imgur.com/a/N4FpD

note: be sure to note the scaling at the bottom. 4-500MB/s old vs 3000+ MB/s new

Do games actually load any faster? I went from a Samsung SSD(I think the 850 pro) to an Intel nvme ssd, and load times in games didn't seem much faster. I definitely notice it when booting up VS though.

Gamers might the most vocal high-end PC users out there but they are not the people with the most money.

The threadripper is not targeted at gamers. It is targeted at people who make money from using their machines, these people can, therefore, afford to upgrade.

Consider a user who does video work. (most modern films and tv is filmed in 8k know) if this is in Raw format as it comes off the camera for a 90 min episode that is 6000 GB. Working on this data (tone mapping, cutting, etc) needs all this data to be fed in and out of the (CPU and or GPUs) so all these PCI lanes are there so you can have very fast access to the raw data and still have space for 2 or 4 full speed GPUs to do that work for you.

Forget games no game developer optimizes their games for 4 GPUs these days even Nvida have said that they dont realy want people to do more than 2GPUs in SLI.

I do a lot of compiles of a codebase which is pretty big (Netflix version of FreeBSD). A full recompile takes me ~1 hr on my box (Xeon E5-2630 v3, 8 cores, 16 threads, 32GB DDR4). Most of this is a parallel make that is 100% CPU bound. So I'm probably going to update to thread ripper when it comes out. Doubling my core count should come close to halving my compile times.

I probably will do storage via a ZFS array of small-ish SSDs or NVME drives. I started with a ZFS array of spinning drives when I built this machine ~2 years ago, with an SSD for L2 ARC. I was occasionally IO bound, so I moved most of the build stuff onto its own SSD. That's probably enough, but I want redundancy (without spinning drives to back up to), and I suck at cabling, so I'm leaning towards just using NVME drives in PCIe slots in order to avoid the cables.

> A full recompile takes me ~1 hr

Could you explain why full recompiles on a single machine are necessary? Are you testing different compiler settings?

I work remotely, and it is easier to develop and compile on the same box. I normally compile just the kernel, but occasionally have to rebuild the entire thing.

What makefile/build system r u using ?

I am curious since i did some work on parallel builds and I always wonder how much of this is (easily) transferable to other setups...

A modified version of the standard FreeBSD build, with bsd make. Eg, a script driven 'make -j X buildworld && make -j X buildkernel'

if you had access to a hosted build system, which is x10 faster. lets even assume some sort of transparent gradual sync of your source tree, and the binaries back to your system.

would u find value in such a setup or is it not worth the hassle ?

(lets assume the output is veritably binary-identical to your local build)

I use the system more for work than games but one game it did make a very noticeable difference in was Minecraft believe it or not. Long-view distance + pre-generated map (world border fill) = move/fly fast without chunks taking friggin forever to load :-)

That's a very clean looking system! Kudos. :)

I have only one of the 512 950 PROs in mine; they are awesome. :) http://i.imgur.com/sydrBvv.jpg . Pic taken before 2nd 980 GPU was put in; 6850k CPU.

I think gaming and ARM cannot be used in the same context. The extra IO is useless to the gamer unless there is an Nvidia GPU on the other end.

I've been shopping for a future proof workstation (10Gbps Ethernet 8x + 2 16x video cards) that couldn't quite fit on a Xeon E3.

What is really nice about the full Zen line is they are supporting ECC memory. It always felt like Intel had removed this on lower end chips just to force users to buy a Xeon even if the did not need the rest of the Xeon features.

Yeah, the Xeon-E3 lines lack of PCI-e lanes is really underwhelming. I've got a LSI 9200-8e and Mellanox Connect-X 2 in my ProLiant ML10 used as my FreeNAS box, leaving a total of 4 PCIe lanes available - thankfully I don't see any need for more AIB's in this machine, but if I used it as a workstation that needed a GPU + another board or two it'd be a really awful situation to be in.

Considering how hard Intel is pushing Thunderbolt it makes little sense how hard they're gimping their consumer and workstation CPU's when it comes to PCIe support.

Realistically speaking, what's the use case for something like that? And secondly, how much a mobo like that would cost?

Intel X99 mainboards (socket 2011-3) ranged from 200-600 us dollars. AMDs offerings typically go for less than Intels, but Threadripper has an gigantic 4094 pin LGA socket, so I guess this will negate the typical AMD price advantage regarding motherboards and we will see prices in the 200-600 us dollar range as well.

EDIT - and since Threadripper will have 20 more PCIe lanes compared to Intels i9 series, motherboard makers will hopefully take advantage of this to offer more IO, which probably also increases the price.

I don't fully understand. So 1 SSD need 1 lane. How many lanes need something like NVisia 1080Ti? What use of the remaining lanes? If I want to build something like 1 CPU/1 GPU/1 SSD configuration without additional peripherals, 62 lanes will be effectively wasted?

The newer M.2 SSDs suck up 4 lanes. That's 16 lanes for four drives, burning through all available lanes on a Kaby Lake CPU.

> burning through all available lanes on a Kaby Lake CPU.

Don't you also get something like 4x via the chipset through DMI?

A GPU can use 16x 3.0 lanes, theoretically. In practice 8 lanes is enough, people have measured performance between 16x and 8x, it's almost the same.

SSDs use 4x, if you do buy an NVMe drive that is. (In my country, these things are TWICE as expensive as SATA SSDs lol)

Maybe for games, but for cuda this is certainly not the case. Double the lanes is double the host to device bandwidth.

And even for games, the only reason it's not a difference is because game developers target less bandwidth between CPU and GPU so they don't use it all.

If we move to a future where every GPU has a lot more bandwidth then games would most likely start taking advantage of it.

"CUDA" is not a particular application. You could very well be doing something with CUDA that does extremely little data transfer to/from the host, but a lot of computation on the GPU. Like mining :D

Also, host to device can be bottlenecked by memory speed on either side.

CUDA meaning GPGPU. All I was pointing out is that there is definitely a bandwidth doubling going from 8 to 16 lanes. It's used extremely often by people using the cards for compute. Mining may not have that issue, but deep learning and other applications do.

My current “gaming machine” is still using PCIe 1.0 and it doesn't look like it's proving much at all of a bottleneck, because why would it? The game loads most things into memory once and then the rest of the time it's just sending draw calls and maybe geometry updates.

How are you measuring this?

This is for high-end workstations after all. Many people are suffering with video production. New cameras this year output 8K video raw (or slightly compressed). Even the guys in RED couldn't produce hardware to properly feed the stream into a computer realtime or at least without having to wait.

> I don't fully understand. So 1 SSD need 1 lane.

M.2 SSDs supports up to PCIe 4x, though there are also PCIe card formats which support 8x and 16x: http://blog.seagate.com/intelligent/seagate-demonstrates-fas...

> How many lanes need something like NVisia 1080Ti?

16 per (technically they need 4 per — that's what TB3 gives — or possibly less, but they'll use 16 if they can)

> If I want to build something like 1 CPU/1 GPU/1 SSD configuration without additional peripherals, 62 lanes will be effectively wasted?

You're at 24 lanes for the GPU (16), SSD (4) and southbridge (4). And unused lanes are no more wasted than unused USB slots, seriously.

But remember that Threadripper is an enthusiast's platform, people who might have 2~4 SSDs (8~16 lanes) or SLI GPUs (2x16).

This is an interesting possibility:

https://www.reddit.com/r/Amd/comments/6ee5qw/thanks_to_threa...

These comments baffle me. Doesn't the majority of HN work in the industry? Clearly the driving factor regarding IO at this point is machine learning and networking for cloud applications, not gaming.

Desktop/workstation (e.g. gaming) performance hasn't been driving the industry for easily 5 years now. Intel and AMD both simply use their high performance desktop offerings as proofs for their server solutions, which typically come a few months later.

Reply on news.ycombinator.com