Hacker News Reader

Comment by modeless | Hacker News Reader

There is zero chance that Intel can add "good enough" functionality to their CPUs without changing their memory architecture. A high end i7 might have 35 GB/s of memory bandwidth. Even the biggest, most ridiculously expensive ($6000) Xeons have only about 100 GB/s. Nvidia's GTX 1080 has 320 GB/s. Their P100 has 732 GB/s.

The threat to Nvidia from Intel is in the Nervana chips they recently acquired. Those are presumably using HBM2 and could potentially beat GPUs for neural net training performance.

View on HN

A modern Xeon has 10s of MBs of cache, which has drastically higher memory bandwidth. What matters is how big the matrices (the "model") is not the input data. With native 8-bit or 16-bit support, you can fit an awful lot of model in 30MB.

I'd recommend reading this paper about matrix math on GPUs from friends of mine way back in the day: https://graphics.stanford.edu/papers/gpumatrixmult/gpumatrix... . While NVIDIA has built much larger register files and L2 caches since then, a modern Xeon still is unbeatable when something fits in L2 or even L3 cache.

I believe streaming the input data in to update the weights is usually done with non Cache polluting instructions (mm_stream equivalents with the NT hint), so it's not hard to keep it fed.

In general neural nets today are so slow (due being computational resource hungry, in clock cycles, memory, and memory bandwidth) that even 4 of the fastest GPU's money can buy stacked together still take days or weeks to train a production quality model.

Trying to peg this onto one of their CPU's will have a result much like they have had in pegging the graphics processing stuff onto their CPU's. That is, very meagre results that is fitting for only the lowest resource intensive things that a consumer may want to do.

30MB to hold a model in is almost nothing. Even 12GB is insufficient to train something like imagenet.

But in the same vein as their iGPU implementation, it may be just enough to satisfy the requirements of the average consumer, and still be able to make a large dent in Nvidia's market share.

Unless you've got extremely low I/O latency, allowing you to work in smaller batches in parallel without loosing all of your parallelisation speedup to the 'tax' you pay when loading data & instructions from RAM to <insert processing unit here>. Although I have no idea what that implies for the hardware discussed above, as it's way over my head.

SOTA models are much, much larger than 30 MB. You can't use 8 bits for training (yet?), and Intel doesn't support 16-bit float (they need to catch up with Nvidia on that, it's really a travesty, we could have used it years ago for audio and imaging). Besides, the top end Xeons with the most cache are fully 10 times more expensive than a GTX 1080.

IMO the travesty is crippling FP16 on consumer GPUs for exactly the reasons you cite. And given that AMD's Vega GPUs apparently will possess full speed FP16, it appears to be yet another in a long line of arbitrary performance cripplings for the consumer space that is IMO the most annoying feature of CUDA development.

But while FP16 is useful for audio and imaging, FP16 nearly killed NVIDIA over a decade ago* when it lacked the dynamic range for DirectX 9 HDR effects in contemporary games without banding. FP32 was more than enough for the task, but power hungry, and thus NV30 could be used to figuratively fry eggs while AMD GPUs had FP24, which was just enough for these effects.

These days, it's all going in the opposite direction w/r to deep learning, but I find it ironic that INT8/INT16 is missing from the Tesla flagship P100, but present on GP102 and GP104, the consumer GPUs (and yes, I know about Tesla P40, but that lacks fast FP16).

I agree with the top poster that Intel could make quite a comeback here given how far it's currently behind. I also agree that it would be hard to dethrone NVIDIA without higher bandwidth memory, but I don't think that's necessary, a bloody nose is more than enough to turn heads IMO.

https://en.wikipedia.org/wiki/GeForce_FX_series

Correct, we may have a few MB of text that is training data. But the trained models are several GB's

Models usually don't fit the cache (the MNIST one might fit, but that's a toy model today)

Also backpropagation makes you iterate through the whole model at each step

100 GB/s is the bandwidth of six DDR4 channels. Knights Landing has 500 GB/s of local memory bandwidth, plus the same 100 GB/s of a Xeon for DDR4.

If future Xeons integrated RAM like Knights Landing does, this would make GPUs less interesting. But KNL can already be used as the main CPU, if I remember correctly.

You're ignoring the main factor: cost.

Intel's offering costs as high as €4000 per processor, which come with a meager 8 cores.

NVidia's offering sells, right now, for less than €1000 a pop.

What about ongoing electricity costs? IIRC my graphics card chews up more power than my CPU.

A delta of $3000 per component buys you a whole lot of electricity.

Furthermore, the performance of each of NVidia's GPUs falls somewhere between 5 and 20 teraFLOPs. A Haswell Xeon gets you about half a teraFLOP for around 1/3 of the power consumption of a NVidia Tesla P100 GPU.

I am not sure where you get those numbers. Both a Knights Landing Xeon Phi and a Tesla K80 are about 3 teraflops for $5000.

Comparing GPUs with Xeons makes no sense.

Many inference tasks can fit in cache or eDRAM. The SIMD units are not fasts enough to make it a big problem so long as the code is well-optimized.

On the Nervana front, outrunning a GPU for neural nets is not that hard with an ASIC. I checked your profile, I bet your employer knows something about that :)

That's just moving the data around. CPUs can't do any meaningful (like AI training) data processing at these speeds, as far as I know. GPUs probably can do something simple at these speeds, but I doubt AI training can be done even at 320GB/s on a single GPU.

moving large data around is the whole issue of deep learning.

No, it's not. You have to apply the data to the evolving network. Can you really do it at 320GB/s?

Good question. If you have enough cores - maybe? Does anyone have a precise idea?

Reply on news.ycombinator.com