Hacker News Reader

Comment by cs702 | Hacker News Reader

Yes, Nvidia dominates in deep learning and AI, but the risk of disruption by Intel is much greater than a lot of non-technical people realize. Intel has to do only two things to start taking market share away from Nvidia in DL/AI quickly:

1. Add "good enough" functionality to its high-end processors. Intel is already working on that: https://news.ycombinator.com/item?id=12709220

2. Contribute open-source code to the main branches of the most popular DL/AI frameworks (Tensorflow, Theano, Torch) so these frameworks support the new chip functionality "out of the box," without requiring any additional tweaking. This is not yet happening, but I'm hoping it will soon.

Many DL/AI developers would be content with "good enough" performance out-of-the-box from CPUs if it means not having to pay extra for Nvidia cards or deal with Nvidia's proprietary drivers.

It would be great for Nvidia to get real competition in this space.

View on HN

There is zero chance that Intel can add "good enough" functionality to their CPUs without changing their memory architecture. A high end i7 might have 35 GB/s of memory bandwidth. Even the biggest, most ridiculously expensive ($6000) Xeons have only about 100 GB/s. Nvidia's GTX 1080 has 320 GB/s. Their P100 has 732 GB/s.

The threat to Nvidia from Intel is in the Nervana chips they recently acquired. Those are presumably using HBM2 and could potentially beat GPUs for neural net training performance.

100 GB/s is the bandwidth of six DDR4 channels. Knights Landing has 500 GB/s of local memory bandwidth, plus the same 100 GB/s of a Xeon for DDR4.

If future Xeons integrated RAM like Knights Landing does, this would make GPUs less interesting. But KNL can already be used as the main CPU, if I remember correctly.

You're ignoring the main factor: cost.

Intel's offering costs as high as €4000 per processor, which come with a meager 8 cores.

NVidia's offering sells, right now, for less than €1000 a pop.

What about ongoing electricity costs? IIRC my graphics card chews up more power than my CPU.

A delta of $3000 per component buys you a whole lot of electricity.

Furthermore, the performance of each of NVidia's GPUs falls somewhere between 5 and 20 teraFLOPs. A Haswell Xeon gets you about half a teraFLOP for around 1/3 of the power consumption of a NVidia Tesla P100 GPU.

I am not sure where you get those numbers. Both a Knights Landing Xeon Phi and a Tesla K80 are about 3 teraflops for $5000.

Comparing GPUs with Xeons makes no sense.

That's just moving the data around. CPUs can't do any meaningful (like AI training) data processing at these speeds, as far as I know. GPUs probably can do something simple at these speeds, but I doubt AI training can be done even at 320GB/s on a single GPU.

moving large data around is the whole issue of deep learning.

No, it's not. You have to apply the data to the evolving network. Can you really do it at 320GB/s?

Good question. If you have enough cores - maybe? Does anyone have a precise idea?

Many inference tasks can fit in cache or eDRAM. The SIMD units are not fasts enough to make it a big problem so long as the code is well-optimized.

On the Nervana front, outrunning a GPU for neural nets is not that hard with an ASIC. I checked your profile, I bet your employer knows something about that :)

A modern Xeon has 10s of MBs of cache, which has drastically higher memory bandwidth. What matters is how big the matrices (the "model") is not the input data. With native 8-bit or 16-bit support, you can fit an awful lot of model in 30MB.

I'd recommend reading this paper about matrix math on GPUs from friends of mine way back in the day: https://graphics.stanford.edu/papers/gpumatrixmult/gpumatrix... . While NVIDIA has built much larger register files and L2 caches since then, a modern Xeon still is unbeatable when something fits in L2 or even L3 cache.

I believe streaming the input data in to update the weights is usually done with non Cache polluting instructions (mm_stream equivalents with the NT hint), so it's not hard to keep it fed.

In general neural nets today are so slow (due being computational resource hungry, in clock cycles, memory, and memory bandwidth) that even 4 of the fastest GPU's money can buy stacked together still take days or weeks to train a production quality model.

Trying to peg this onto one of their CPU's will have a result much like they have had in pegging the graphics processing stuff onto their CPU's. That is, very meagre results that is fitting for only the lowest resource intensive things that a consumer may want to do.

30MB to hold a model in is almost nothing. Even 12GB is insufficient to train something like imagenet.

But in the same vein as their iGPU implementation, it may be just enough to satisfy the requirements of the average consumer, and still be able to make a large dent in Nvidia's market share.

Unless you've got extremely low I/O latency, allowing you to work in smaller batches in parallel without loosing all of your parallelisation speedup to the 'tax' you pay when loading data & instructions from RAM to <insert processing unit here>. Although I have no idea what that implies for the hardware discussed above, as it's way over my head.

SOTA models are much, much larger than 30 MB. You can't use 8 bits for training (yet?), and Intel doesn't support 16-bit float (they need to catch up with Nvidia on that, it's really a travesty, we could have used it years ago for audio and imaging). Besides, the top end Xeons with the most cache are fully 10 times more expensive than a GTX 1080.

Correct, we may have a few MB of text that is training data. But the trained models are several GB's

IMO the travesty is crippling FP16 on consumer GPUs for exactly the reasons you cite. And given that AMD's Vega GPUs apparently will possess full speed FP16, it appears to be yet another in a long line of arbitrary performance cripplings for the consumer space that is IMO the most annoying feature of CUDA development.

But while FP16 is useful for audio and imaging, FP16 nearly killed NVIDIA over a decade ago* when it lacked the dynamic range for DirectX 9 HDR effects in contemporary games without banding. FP32 was more than enough for the task, but power hungry, and thus NV30 could be used to figuratively fry eggs while AMD GPUs had FP24, which was just enough for these effects.

These days, it's all going in the opposite direction w/r to deep learning, but I find it ironic that INT8/INT16 is missing from the Tesla flagship P100, but present on GP102 and GP104, the consumer GPUs (and yes, I know about Tesla P40, but that lacks fast FP16).

I agree with the top poster that Intel could make quite a comeback here given how far it's currently behind. I also agree that it would be hard to dethrone NVIDIA without higher bandwidth memory, but I don't think that's necessary, a bloody nose is more than enough to turn heads IMO.

https://en.wikipedia.org/wiki/GeForce_FX_series

Models usually don't fit the cache (the MNIST one might fit, but that's a toy model today)

Also backpropagation makes you iterate through the whole model at each step

I'll point out that this same argument applied to discrete GPUs when Intel tried to enter the market with Larrabee (before giving up and claiming that HPC was where it's at anyway).

"This time is different" might apply here though, because training is "just" a lot of low precision matrix math. This requires a lot less new ecosystem / software for Intel (Intel has a long history in matrix math), so much like SSE, AVX, and now AVX-512 (or as I still call it LRBni) Intel can easily make some tweaks to at least get 70% of the gains.

Also: Intel learned many things with Larabee. Larabee may have been a failure in public perception, but for Intel it seems to have been a fruitful experience which still helps them. (Somewhere, I've read that they've won the contract for one of the next super computers based on their Knights Landing architecture)

> It would be great for Nvidia to get real competition in this space.

It seems obvious AMD will be a player, since they have similar hardware (GPUs) and with a proper port (using HCC, openCL, etc. ) for cuDNN should also have excellent performance for Tensorflow, Theano, etc. 'out of the box'.

Their mixed GPU/CPU modules should be very interesting as well.

No, they have inferior performance, and all the same issues as Nvidia- in addition to requiring a port.

inferior performance because of lack of performant DL libraries, like CuDNN, not because of the hardware itself

It's a mix of both, hardware is inferior for current use cases but current use cases are driven heavily by NVIDIA.

NVIDIA knows how to do 2 things right; understanding that software matters, even more than hardware and deliver on that notion; and how manage developer relations.

While AMD makes it easy to access information without NDA and sign ups NVIDIA blows it it if the water the moment you show even the slightest interest at signing up with them.

NVIDIA at this point can probably spin off a pure consulting division for AI/ML and bring in more revenue than AMD in its entirety.

I liked some of this point except the end, which is not correct.

> NVIDIA at this point can probably spin off a pure consulting division for AI/ML and bring in more revenue than AMD in its entirety

AMD had $3.99 Billion in total revenue in 2015. Nvidia had 5 Billion.

Perhaps you mean profit? Amd was operating at a loss and nvidia was very profitable.

NVIDIA doesn't charge you for consulting services, they do expect you to buy pretty expensive hardware tho. They offer training and consulting services for "free" and paid services through their network of partners, to which NVIDIA provides services for free. If they spin off their in house knowledge and start charging for services and software instead of giving it out for "free", they might very well reach a 4bln $ a year revenue stream, that said a lot of people won't be happy about it...

How is the hardware inferior for current use cases?

The compute capabilities are pretty much the same for the latest NVidia/AMD cards in the same price range. It's a software thing - lack of optimised DL ops, particularly fast convolution kernels that is hurting the perf of AMD in deep learning

No. AMD has worse floating point performance, and has poor support for half precision floating point values. Also gets beat on power consumption per flop.

FLOPS/$ is better on AMD afaik. double speed fp16 is only on gp100 and I don't know if any framework supports fp16 training yet. But yes AMD need to add 2x fp16 too.

The main reason people don't use AMD for deep learning is lack of fast libraries, and AMD's inability to provide a CuDNN equivalent optimised for their hardware

But I don't think this is a software issue: clFFT runs faster on NVIDIA. Simple operations like SAXPY run faster on NVIDIA. Etc.

Speaking of this, AMDs new Vega cards in theory should offer more fp16 than the P100.

No, AMD doesnt do software suppport. In words of their own developer:

>>We don't happen to have the resources to pay someone else to do that for us.

Yeah, that whole ongoing saga is very unfortunate. Also very illustrative. Reading between the lines, my take is that AMD senior management are a bunch of generic MBA types optimising for higher profit in the present (Windows gaming market) by stealing from the future (HPC & server segments + growing niches like ML ---> Linux and open standards, in other words).

It makes zero sense for them to place increasingly large bets on proprietary drivers when they're clearly not the market incumbent.

It sounds what was happening when AMD, ATi were still seperate entities.

The high end Intel CPUs like Xeon Phis that have enough compute muscle to get anywhere near NVidia GPU perf for deep learning are far too expensive to be able to replace them

Precisely.

Intel is trying to compete with NVidia by offering inferior performance at a premium price.

In massively parallel applications, cost is closely tied to performance. NVidia has both aspects secured very well.

In fact, NVidia is already a prominent part in supplying components for the 3rd place in the Top500, Oak Ridge National Laboratory's Titan system, which mixes up AMD Opterons with NVIdia k20x GPUs.

Isn't some of the advantage coming from the fact that gpus are much better designed for massively parallel computations?

> Intel has to do only two things to start taking market share away from Nvidia in DL/AI quickly:

...except that Intel has been trying to do that for over a decade, and Intel has been failing repeatedly for over a decade.

If that was as simple and easy as you lead to believe, Intel would've already done it by now.

Except it didn't, because it can't.

How exactly Nvidia and ARM are competitors of Intel is an interesting question in itself. Nvidia and ARM are fabless and compete against a subset of Intel's IP and products. Intel may take multiple routes to reacting against superior competition without necessarily even making better GPGPUs.

Along these lines... For inference, do you really want to put a Tesla M4 in every server in your cloud when you're already paying $$$$ for CPUs?

Do you really want to put a Xeon Phi in every server in your cloud when you're already paying $$$$ for CPUs?

Xeon Phi can be its own cpu though. That's what Intel is counting on.

> Xeon Phi can be its own cpu though

yes and they are priced accordingly ;-)

No, there's no "Good Enough" it needs to be fast enough.

Xeon Phi is their contender, but we'll have to see enough

A high-end GPU is around 10x faster than your avg CPU https://www.nvidia.com/object/gpu-accelerated-applications-t...

>>This is not yet happening, but I'm hoping it will soon.

Intel made massive gains promoting OpenCL, when they helped to redo OpenCV's umat infrastructure.

True but Nvidia has a first mover advantage which should not be discounted.

Reply on news.ycombinator.com