GPU-Oriented PCIe Expansion Cluster | Hacker News Reader

GPU-Oriented PCIe Expansion Cluster

amfeltec.com

91 pointsminxomat46 comments

For those doubting the usefulness of these rigs, I can say that for GPU rendering numerous of these are in use by small sized studios and freelancers. The bottleneck of the slow PCIe connection is nothing compared to the render times per frame in most cases. It's only when a frame takes less than a second to render that you notice. They are very popular with e.g. Octane users. The only thing to be aware of is that some motherboards do not play well with this and will not recognize all cards or will not run stable with it attached.

The right way to build this sort of thing is with a hub of 8796 PCIE switches for each group of 4 GPUs such that they could all form a continuous 16x PCIE bidirectional ring suitable for O(n) collectives like gather/reduce/allGather/allReduce.

This is more or less useless, or if one wishes to be kind, an amazing emulation of building distributed GPU code over the craptastic bandwidths brought to you by AWS, Microsoft, and Google datacenters for now.

Also PCIE Gen 2? WTF? This might not even post with current GPUs.

PCIe is all kinds of forgiving. It works all in possible combos.

Except that if you want O(n) performance, the easiest topology is a ring connecting all the GPUs without having to pass through any CPUs. Passing through 1 CPU will incur a ~20% bandwidth penalty, and passing PCIE data directly between 2 CPUs will break the ring because Intel says so.

Baidu just released a library implementing this approach between nodes. NVIDIA's NCCL is a library for passing such data between GPUs connected like this in a single node, and the deep learning framework DSSTNE has tensor collectives for implementing model parallel training and inference.

I'm not aware of any PCIe ring mechanism in silicon. Do you mean emulating a ring over a switched topology, using GPU initiated GPU-GPU DMA?

A somewhat related question:

Do PCIe switches know the address windows of their children? Meaning that when one child does an access to one of its peers (another child), the packet is sent directly to the other child rather than all the way up to the root? I'm not too familiar with how PCIe switches get enumerated, but perhaps it gets configured at that time?

Or does it just behave as a regular bus where every device "listens" for accesses to addresses within their BAR regions (implying that every packet gets broadcast to everyone)? However, I suppose that would be more like a hub rather than a switch...I'll do some more reading!

The TLP header has the destination device's address (in the PCIe tree), so switches just switch. The BARs are related to the host memory map, which has no inherent meaning to devices. nVidia GPU direct introduces a virtual memory region that maps device memory, but I'm not sure if they internally have an IOMMU. For transfers to a device on another bus on the same system the remote device can't be addressed, so it has to ping-pong through system memory, but a device in a different node can be addressed.

The device could query the root complex (the OS) to find the bus address of the device with a particular host memory address but I don't know if that is a common idiom.

(How this works with Multi-Root IO Virtualisation I'm not sure and it gives me a headache.)

Sure, if you want performance but you said it might not POST. That's not true.

Yes, it is. For example, in my own experience, Maxwell GPUs frequently do not post with SandyBridge CPUs on some X79 motherboards (Asus P9X79E-WS is one). There are BIOS fixes for this, but many of those motherboards were discontinued before NVIDIA GTX 980 in September of 2014.

Ripping out the SandyBridge CPU and replacing it with an IvyBridge CPU magically fixes the problem. I think the difference here is that SandyBridge only supports PCIE Gen 2 while IvyBridge is PCIE Gen 3. One will never know for sure though.

This is for mining or password cracking.

Both bitcoin and litecoin are both far faster on ASICs. CrytoNight (Monero) is designed to be harder for ASICs but it is early days, I would expect new chips to make a significant improvement over GPUs. Since the memory requirement is only a few MB there is no need for access to large memories or a PCIe bus, and they will continue using ethernet.

https://bytecoin.org/blog/proof-of-work-part-2/

There are many others that do not (yet) have ASIC implementations.

But does anyone use them? And do they use any schemes that are not (a) a basic hash, (b) some scrypt scheme? Any reason why you wouldn't just put 3 GPUs in a regular chassis and be done for a quarter of the price?

> Any reason why you wouldn't just put 3 GPUs in a regular chassis and be done for a quarter of the price?

Not a *coiner but I can guessL

With most motherboards even 3 GPUs will require you to have the cards right up next to each other. This is not great for cooling, which is important when you have the GPUs cranking away at full bore.

Also you can only get 4 GPUs at max in to a normal size PC case. With these you can have a single host PC running 16+ GPUs.

Sure, great, but if they could build this to support Deep Learning and renegade HPC, I suspect they could sell 10,000+ units/year on the down-low. That would be $10M in revenue, maybe twice that if it also worked with Macs.

NVIDIA will do their worst to shut this down because it's a direct threat to DGX-1 running neural networks that aren't entirely communication-limited (long story), but if they could throw this together, I think they could make a great quick buck before the axe falls.

If you take a look at their clients, it's mostly massive companies that probably order large lots of OEM solutions. I don't think amfeltec needs the scale or rather isn't already at the scale you are talking about (margin, not turnover), by enterprise tax and support for OEM builds alone.

The 4x splitters go for about $200. But if you want guaranteed compatibility (i.e. a full build), the price (and margin for them) will skyrocket.

How could Nvidia really shut them down? It seems like they're just selling PCIe switches. I could be missing something, but I don't see anything even remotely infringing about this.

PCIe is a parallel buss gone serial. As a result of its history all its routing mechanisms are tree based. Sadly no rings are possible...

Tell that to the creators of NCCL and DSSTNE, who have both achieved ~12.5 GB/s O(n) collectives on 16x PCIE Gen 3 across an arbitrary number of GPUs given they are connected across a network of PCIE switches with no intervening CPU in the ring. Sure, there are latency issues here that ultimately cause a small amount of friction, but nothing like what you are implying, and NVIDIA could address those issues if they ever wanted to do so.

Or who are you going to believe? Your #PCIEAlternativeFacts or the above lyin' code libaries? But I'll cut you some slack, back in the day (2013-2014), it took a great deal of data to convince GPU server vendors like Quantum, SuperMicro, and Cirrascale that this was the case. And to this day, they still sell multi-GPU servers with chipsets where one set of GPUs on a PCIE switch cannot directly access the other(s).

There is 1 such bidirectional ring in a tree of PCIE hubs. For bonus points, figure out the 4 bidirectional rings inside the NVLINK topology within a DGX-1.

That said, I like the idea of this expansion cluster. If they were to quickly research and offer an 8796 switch-based variant, they could make top dollar in the 99% segment of the emerging Deep Learning market. Potentially also from hedge funds and big pharma, none of whom wish to pay $5000 for $1000 GPUs with premium trim.

And they would get to do this until NVIDIA gets its knickers in a bunch (once again) about people using GeForce GPUs instead of Tesla GPUs for the sort of workloads they have arbitrarily classified to require said Tesla GPUs and then either cuts off their supply of parts or extorts them in some way to shut them down.

Unfortunately this setup would be very slow for most GPGPU applications, because the CPU-GPU and GPU-GPU bandwidth is very slow. To effectively use this the data transfer requirements have to be very low.

Even with a dedicated 16x PCIe 3 connection there is a latency overhead compared to inter-CPU buses, like HyperTransport or QPI, which is why nVidia and IBM have scaled up the NVlink inter-GPU bus to become a memory speed interconnect.

https://www.ibm.com/blogs/systems/ibm-power8-cpu-and-nvidia-...

I'm not as clued up as I used to be about this stuff, but wouldn't this have a pretty serious impact on the performance of the individual cards? Seems like splitting 4 16x cards off one 4x bus would limit the available bandwidth somewhat.

OP here. I found exactly one person using it, with positive results: https://hashcat.net/forum/thread-2622.html

Though amfeltec is sadly a pretty unknown company. Probably best known for their "squid" PCIe "split" cards for multiple M.2 drives in one x16 slot. I've used one of them.

That depends if the workload is data-transfer-bound or computing-bound.

If the former, yes you will suffer a massive performance blow even with just one GPU - but if the latter, it's an easy way to upgrade your system.

I suppose if you're just working on a data set that's already stored in memory on the GPU(s) the initial work involved in getting that data to the card would be impacted but everything after that benefits from having an absurd amount of computing power

This might be useful for password cracking on GPUs. I imagine there is negligent traffic on the bus in this scenario.

I saw a lot of this when GPUs were being used to mine bitcoin/altcoin.

They're still being used for some blockchains with ASIC-resistant mining, including zcash and ethereum.

If there are any reasonably-priced ways of doing this especially with more PCIe lanes please let me know. In my experience it's often easier to just buy more motherboards and CPUs than invest in PCIe expansion.

Depends on your bisection bandwidth requirement and software cost. If you want to do MPI it is going to be cheaper to buy some InfiniBand cards. But if you have a small cluster and tightly coupled code, or large I/O requirements, then a PCIe switch might be the go. These get used on some video on demand systems and signal processing architectures:

https://www.microsemi.com/products/drivers-interfaces-and-pc...

The Dolphin systems are tuned better for computation: http://www.dolphinics.com/products/IXS600.html

I expect small PCIe NVMe external storage systems to become quite common in the near future, because enterprise systems need multipath storage for reliability; 8GB/s FC is too slow for SSDs, let alone NVMe, same with SAS bus expanders.

https://events.linuxfoundation.org/sites/events/files/slides...

The Dolphin systems seem more expensive than just buying 4U servers.

You only buy them when you can't fit your problem into one server (i.e. not enough memory or I/O) or you need redundancy (multipath I/O, synchronised memory). In those cases they can be a lot quicker/cheaper/more reliable than trying to solve it with more computers, protocols and ethernet.

A used Tesla s1075 from eBay looks like a good bet. That's my plan, anyway.

Would require some modding to use more up-to-date GPUs (there are some mounting studs on the chassis floor that need trimming) but the forced airflow means that passive cards like the original Xeon Phis are also a possibility.

This needs a (2014) in the title.

This is interesting but would different architectures of GPUs play nicely? E.g. 3 AMD gpus and one NVIDIA gpu?

They should. I have currently AMD + NVIDIA GPU in my PC, the lanes are split on the motherboard (it can be 1 x16 or 2 x8).

There is a bit of fiddling with drivers but it is possible to have one manufacturer working in compute only mode and other as graphics and compute.

If you think regular GPU drivers are shitty, then wait until you try to install both AMD and nVidia drivers at the same time. It's not very stable, to say the least (Windows 7). Never tried it since. Not worth the hassle.

Easy to install both nvidia and AMD and works like a charm for GPGPU on Arch Linux, though.

Take a look at the hashcat link I posted above. Someone had success with a mixed AMD/NV setup using linux.

At those speeds, doesn't the cable/bus length play a huge role? What's the maximum length that can support them? I'd imagine it'd be a few centimeters at most...

Gen 3 PCIe with optical cables works up to 100 meters.

Seems strange they are using cat 6 cable, which is specified/characterized/tested to 500 MHz, at 2500 MHz for gen 2. Loss is probably not an issue because I see a redriver on the adapter card, but this doesn't seems like a robust solution. I suspect there is a lot of variation in cat 6 cable behavior (across vendors, cable, crimping) at 2.5 GHz.

shorter is obviously better, but from what I've read 30 cm PCIe riser cables work, and there are 50 cm examples.

We have problems with signal integrity on cheap risers and ribbon cables.

Huh, that's longer than I thought, thanks.

As long as the insulation is good, in theory everything is possible.

You could build your own super long PCI Extender Cable, using superconductors...

Resistive losses are only a part of the total loss on a transmission line.

Reply on news.ycombinator.com