Hacker News Reader

Comment by arielweisberg | Hacker News Reader

This is really trippy. GPUs generally need a fast interface like PCI-E and won't work over a network like a disk.

So to elastically provide GPUs over a rack how does that work? How do you not have a ton of GPUs just sitting around due to the physical constraints of PCI-E given that you can attach GPUs to some common instance types at any time? How do you not run out of capacity and just have to say no?

View on HN

You can deploy PCIe with switches over distance and keep the GPUs in an external enclosure:

http://www.dolphinics.com/products/pent-dxseries-dxs410.html

http://www.dell.com/us/business/p/poweredge-c410x/pd

*there are newer/better/faster versions of the above available, just two examples I had handy as my guess of how to make this happen. I'm confident that even if AWS is doing something similar it's being done on customized hardware.

The problem is that transferring data to/from GPUs is already problematic performance wise. I guess it depends on your application whether the data transfer rate hit is acceptable or not.

Looks like the use case that they have in mind is focused on CAD / CAM. Don't see much mention about deep learning and I would guess that isn't being targeted at all.

I'm in the process of moving our local graphics workstations to GRID-powered workstations on AWS, with Raspberry Pi 3 "heads" and 1080p 60fps streams.

The goal is to cut our capital costs, allow access to GPU-based graphics apps "anywhere", and host all of our virtual film production assets on AWS, where they are accessible for rendering, simulation, etc.

As a gamer I was initially surprised that works. It makes sense if server side use cased are more batch oriented/compute intensive.

Latency to start and stop "jobs" is critical in gaming as you are trying to hit a 60-144hz job time.

GPUs in the cloud aren't targeted at gamers. They're targeted at people doing things like running render farms and training deep learning models.

What about companies interested in real-time streaming of 3D content? That use case is extremely sensitive to latency, probably more so than normal gaming is.

Fortunately the amount of additional latency introduced is likely to be negligible (another comment cites PCI-E switches incurring <= 1µs).

The added latency of a PCIe switch and some cables is 1 us or less; it should be negligible for almost any workload.

> How do you not run out of capacity and just have to say no?

I don't know the answer to your other questions but EC2 can and often does refuse our run instance requests because they don't have the resources available in the AZ we've requested.

One thing you can do is put 4-8 GPUs in each server and just try to bin pack different customers properly.

Or you could use PCIe switches and cables to dynamically reassign GPUs to different physical servers.

Some other sort of ultra-high bandwidth interconnect between instances such as infiband (or whatever, I don't know that space well)?

(This is speculation/questioning, not implying knowledge. I noticed the placement of the ? makes that a bit ambiguous)

PCIe is really the only option; the question is just whether it's in one box or cabled.

I was thinking that you could package the commands across some other wire to load on the GPU. As long as your dataset fits on GPU (so you don't have access main memory, which would be horrifically slow) seems like it might fit the GPGPU use case.

I agree that external PCIe is the obvious good/best solution. I didn't realize that external PCIe switch boxes existed.

That's only for graphics, for gpgpu people routinely downgrade the pcie to x1, etc

I was wondering this too. The best I could think is that they're trunking PCI-E over TCP or something? Or maybe all of their new instance types have something like Infiniband?

You can buy 10M PCIe cables commercially. In fact, some equipment use them for their backplane.

Scale. AWS is huge.

Reply on news.ycombinator.com