Hacker News Reader

Comment by spullara | Hacker News Reader

SLI? Can't you put 8x or 16x Titan Xs together?

[NVIDIA Employee disclaimer]

Not in SLI, no. SLI is a technology to have two GPUs act as one, by syncing among themselves. When you have two GPUs in SLI, your CPU program will generally see them as a single GPU with (almost) twice as many cores and twice as much bandwidth, but the same amount of memory.

For this to work, the GPUs need to be very similar (i.e. you can't SLI a TitanX and a 1080).

You can, however, put many[1] different GPUs in the same system, and then the onus is on the CPU program to properly manage and send work to each one separately. This is not SLI.

Generally, SLI is useful for games where the program assumes "a GPU" and sends work to it, and you want to have more performance than your single card can provide. For compute/ML, you're much better off just using each card directly.

[1]: Don't quote me on this, but I believe up to 8x is supported, above that is uncharted territory.

Is it possible and would it make sense to use an SLI Bridge without using SLI to boost the cross-card data transfer rates when doing compute workloads?

Sorry, not sure exactly how SLI bridges work - it may or may not be possible to selectively sync only parts of video memory.

In cases where you want faster cross-card transfers and are willing to mess with HW configurations, there's NVLINK.

Wiki: https://en.wikipedia.org/wiki/NVLink

PR piece: https://blogs.nvidia.com/blog/2014/11/14/what-is-nvlink/

Amazon is offering instances with 16 Tesla K80 GPUs. I don't know how that is realized under the hood, though.

SLI is only for realtime rendering with Direct3D and OpenGL. Tesla GPU products are intended primarily for CUDA and OpenCL use, which have always required the application to be aware of the underlying hardware configuration. There's no need to pretend that the 16 GPUs are one super-powerful accelerator device.

Some of the high end Xeon systems have gobs of PCIe lanes. If you have a four CPU socket system, you can hang four x16 PCIe cards off of each processor.

Or get specialized board with PCIe switch ie. http://www.cirrascale.com/blog/index.php/welcome-big-sur/

so GPUs get better bandwidth (P2P etc).

4CPU sockets !=> 4x x16 cars. It always depends on the number of PCI lanes for each processors. Check i.e. these 2 E7 Xeons: http://www.intel.com/content/www/us/en/products/compare-prod...

Both have max 32 PCI Express lanes - so you can theoretically hang max 2 PCIx x16 cards off each CPU, but as few of the lanes is reserved for other peripherals, in 4GPU/CPU setup they will practically run at lower speeds, ie 8/8/4/4. However, using system with PCIe root complex [1] can improve the GPU<->GPU communication speed.

[1] https://www.microway.com/hpc-tech-tips/pci-express-root-comp...

They're treated as independent compute units and the host program load-balances compute work between them. You can even do P2P memory copy or access between your CUDA devices without going through the CPU. See https://www.nvidia.com/docs/IO/116711/sc11-multi-gpu.pdf for a high-level overview.

I understand there was a recent shift away from such practises at nvidia, but am I correct in saying that the 3-way sli bridges do still work with the newest cards?

No.

NVIDIA said the 10-series can only do dual-SLI at most (two cards), nothing more.

Sorry for asking, but you might have a better insight. I have two questions, which you might answer:

- Is it possible, officially supported way, to have both GTX and Quadro at the same time in the machine? I prefer GTX for rendering speed (Redshift renderer), but I need 10-bit output Quadro has which GTX doesn't have.

- Are there GTX (1080 TI in this case) cards out there which don't take double PCI space? I don't need multiple outputs and I can hook them up to the water block, if needed. I need as many cards in a single workstation case as possible for rendering. I've seen people saw/solder off stuff from cards in order to get at that, but not something I would do personally. Ideally I would prefer to have eight 1080 TIs in a single machine along with a single (modest) Quadro for 10-bit output to three monitors - are there even motherboards that could support that?

Note that I'm looking for workstation / deskside solution, not server 747-fan-sound-behemoths.

GTX cards do have 10-bit output (on Linux at least).

I would like 12-bit output, but that's not possible due to limitations in X11.

Is that a recent development? One of Quadro's feature over GTX was/is 10-bit output. No way around it if you're using a 10-bit monitor like EIZO ColorEdge or whatever.

My GTX on Windows also claims to output 10 bits. From some playing around in Photoshop, it seems like the monitor I have does not actually use the extra data. (It asks for it, but either doesn't receive it, or doesn't process it properly.)

You can use PCI risers to give yourself some more flexibility as to how your cards are plugged in. However, please make sure you have enough PCI x16 or x8 slots. If you're not running a server motherboard, I highly doubt you have enough PCI lanes to fit more than two or three cards.

"You can, however, put many[1] different GPUs in the same system, and then the onus is on the CPU program to properly manage and send work to each one separately. This is not SLI."

Interesting that you mention this as I've been searching for an answer before I build a ML rig and embark on learning it. Is there any benefit for ML performance if I put a GTX 970 and a 1080[ti|x] in the same machine? Just trying to figure out how to re-use spare parts.

Reply on news.ycombinator.com