The Looming Battle Over AI Chips
barrons.com
barrons.com
Or you can be like BIDU, buy 100K consumer GPUs, and put them in your datacenter. In response, Jensen altered the CUDA 9.1 licensing agreement and the EULA for Titan V such that going forward, you cannot deploy Titan V in a datacenter for anything but mining cryptocurrency, and his company reserves the right going forward to audit your use of their SW and HW at any time to force compliance with whatever rules Jensen pulled out of his butt that day after his morning weed. And that's a shame. Because there's no way any of these companies can beat the !/$ of consumer GPUs and NVDA is lying out of its a$$ to say you can't do HPC on them.
But beyond NVDA shenanigans, I think it's incredibly risky to second guess those siloed computation graphs from the outside in the hopes of anything but an acqui-hire for an internal effort. Things ended well for Nervana even if their HW didn't ship in time, but when I see a 2018 company (http://nearist.ai/k-nn-benchmarks-part-wikipedia) comparing their unavailable powerpoint processor to GPUs from 2013, and then doubling down on doing so when someone rightly points out how stupid that is, I see a beached fail whale in the making, not a threat to NVDA's Deepopoly.
GOOG: Google
AAPL: Apple
AMZN: Amazon
BIDU: Baidu
ASIC: Application specific integrated circuit
GPU: Graphics processing unit
CUDA: Compute-unified device architecture
EULA: End-user license agreement
weed: marijuana
HPC: High-performance computing
NVDA: Nvidia
HW: hardware
Now when a Tesla GPU costs $9-10K instead of <$1000, that's a somewhat different story, but even then, NVDA tapes out a brand new architecture almost annually. Good luck keeping up with that pace ASIC startups. And that's exactly what happened to Nervana. Their ASIC was better than a Pascal GP100, but it's clobbered by Volta V100. So at best you get a 6-12 month lead on them.
In contrast, if you can right-size the transistor count for expensive and very specific workloads across 100K+ machines like companies with big datacenters and big data sets can do, I see an opportunity for all of the BigCos to build custom ASICs for their siloed data sets and graphs. That's what GOOG is doing and it seems to be working for them so far. FB is now hiring to do the same thing I suspect.
A V100 costs ~300/yr in electricity. If you are buying at the scale of 100k units, but can price per operation, even by just 10% (for example, by dropping features you don't care about), that's a million dollars of electricity over the lifetime of your hardware.
GPUs (and DSPs) historically are way on the compute side. You get kilobytes of on chip memory, and really fat parallel buses to off chip RAM.
On the other end, you have some chips that put more memory near the compute. It means you have less compute, but way better power efficiency. Each hop from a register to cache to off chip to between boards to between nodes is roughly 2-10x power hit, so you get orders of magnitudes here.
In terms of training, it's really hard to fit your training data on chip. In which case you end up with an architecture very similar to a GPU or DSP/TPU. NVIDIA is no slouch here. A couple years ago the big trick was reducing precision - you dont need single or double floats during training so an architecture more specialized at fp16 or int8 would get some big savings for power and throughput. NVIDIA is playing this game too. At Google/FB scale tweaks to improve cost may make sense but it doesnt seem like architecturally speaking there is some major design decision being left on the table anymore (I'd love to hear a specific counterpoint though!).
In terms of inference, there can be some big power savings putting the weights next to compute. This isn't rocket science and folks have been doing this for a while. In a strange way it can sometimes be more efficient to use a CPU, with fat caches.
I'd expect if I knew any, they'd be under NDA. I'll just point out that a GPU, even one with specific "ML cores" as NVIDIA calls them, is going to have a bunch of silicon that is being used inefficiently (for more "conventional" GPU uses). There's room for cost saving there. Perhaps NVIDIA eventually moves into that space and produces ML-only chips, but they don't appear to be heading in that direction yet.
Secondly, GPUs don't really have very much hardware specialized for graphics anymore. If we called it a TPU I'm not sure you'd be making the same point :P At a company like MS/FB/Google where you don't need to leverage selling the same chip for gaming/vr/ML/mining for economy of scale, like you said you can reduce your transistor count and have the same compute fabric. This would reduce your idle power consumption through leakage but you wouldn't expect a huge drop in power during active compute. Because the smaller precision compute ends up increasing the compute per byte, you either need to find more parallelism to RAM, get faster RAM, reduce the clockrate of compute, or reduce the number of compute elements to find a balanced architecture with lower precision. If you just shrink the number of compute elements - voila! - you're close to what NVIDIA is doing with ML cores.
I think this is semantics. Modern GPUs do have a lot of hardware that isn't specialized for machine learning. My limited knowledge says that very recent NVIDIA GPUs have some things that vaguely resemble TPU cores ("Tensor cores"), but they also have a lot of silicon for classic CUDA cores. Which I was calling "graphics" hardware, but might better be described as "silicon that isn't optimized for the necessary memory bandwidth for DL". So its still used non-optimally.
To be clear, you can still use the CUDA cores for DL. We did that just fine for a long time, they're just decidedly less efficiently than Tensor cores.
Developing a new high performance microarchitecture for GPU or CPU is complex task. A new clean sheet design architecture takes 5-7 years even for teams that have been doing it constantly for decades in Intel, AMD, ARM or Nvidia. This includes optimizing the design into process technology, yield, etc. and integrating memory architectures. Then there is economies of scale and price points.
Nvidia's Volta microarchitecture design started 2013, launch was December 2017
AMD's zen CPU architecture design started 2012 and CPU was out 2017.
But also Google is already doing the 2nd generation of the TPUs and would expect a third.
The data of chips coming from 3rd parties are probably numbered.
The dynamics of the industry changed where now the companies that are buying the chips have skin in the game with the cost of running the chips. Versus the old days Intel sold a chip to Dell who then sold it to someone else.
It just makes sense for Google to do their own silicon at this point as saves them a ton of money.
But also Google has the data to improve the chip that Nvidia just does not have.
But without the data and not doing higher layers of the stack how can Nvidia create a product that is competitive with the TPUs for inference?
EDIT: I’m trying to draw a comparison between the availability dates (and where the companies are now), not the start of production (and their development velocity). Including the announcement date was probably a red herring.
Making a chip and making competitive chip are two different things.
When Nvidia enters the market with specialized chip it's likely on completely another level in bandwidth, energy consumption and price per flop performance. They have so much more experience with this.
* https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk...
* https://blogs.nvidia.com/blog/2017/04/10/ai-drives-rise-acce...
Yeah, nVidia throwing it's weight around in terms of requiring that data centers pay more to use cheap consumer gaming chips may turn out to backfire and certainly has an abusive-monopoly flavor to it.
As I've researched the field, Cuda really seems provides considerable value to the individual programmer. But making maneuvers of this sort may show the limits of that sort of advantage.
https://www.theregister.co.uk/2018/01/03/nvidia_server_gpus/
OpenCL could have been killer on mobile, and it could have delivered low-power machine learning and computation there all the way back in 2012, but both AAPL and GOOG went out of their way to cripple its use despite many of the mobile GPUs having hardware support for it. We all lost there IMO.
The first is the ecosystem. Nvidia went to great lengths to provide well optimized libraries built on top of CUDA that supply things people care about - deep learning stuff, dense as well as sparse linear algebra, etc. There's nothing meaningfully competitive on the OpenCL side of things.
The second is user friendliness of the API and the implementations. OpenCL is basically analogous to OpenGL in terms of design, it's a verbose annoying C API with huge amounts of trivial boilerplate. By contrast, CUDA supports most of the C++ convenience features relevant in this problem space, has decent tools, IDE and debugger integration, etc.
Neither of these issues is necessary a dealbreaker if you're willing to invest the effort, but choosing OpenCL over CUDA requires prioritizing portability over user friendliness, available libraries and tooling. As a consequence, not many people choose OpenCL and the dominance of CUDA continues to grow. Unfortunately, I don't see that changing in the near future.
The reason NVidia is used in deep learning is because the frameworks give the best performance on it.
The reason they give good performance on it is because CUDA an CuDNN (which is even more important but I don’t see mention of) work really well and give better performance than anything else.
There are no MBA, just lots of grad students trying anything to get better speed.
1. Volta GPUs already have little matmul cores, basically a bunch of little TPUs.
2. The graphics dedicated silicon is an extremely tiny portion of the die, a trivial component (source: Bill Dally, nVidia chief scientist).
3. Memory access power and performance is the bottleneck (even in the TPU paper), and will only continue to get worse.
All that said, Bill Daly rocks, and NVDA is a hardened target. But the DL frameworks have enormous performance holes once one stops running Resnet-152 and other popular benchmark graphs in the same way that 3DMark performance is not necessarily representative of actual gaming performance unless NVDA took it upon themselves to make it so.
And since DL is such a dynamic field (just like game engines), I expect this situation to persist for a very, very long time.
See Google and anything chat related after 2010
It's not about them being ignorant about it. It's about them making decisions in spite of that knowledge - decisions that Make Sense™ for the advancement and increased profitability of the incumbent cash cow, but they are often contradictory or have a negative impact on new tech investment.
Here's one main reason why Nvidia will not go "full TPU" with its chips - it wants "scalability". That means it wants an architecture that can be "flexible and server different markets".
The companies that specialize in AI chips will likely beat them in performance because they only care about winning one market at a time (and the AI market is a pretty big one).
Intel's AI strategy is even more of a mess, because it has no clue what it can use to beat Nvdia, so its investments and developer ecosystems are all over the place.
I do hope the AI specific TPUs that come out on the future will follow the ARM model instead of being silod into proprietary architectures. Fucking hate the vendor lock in of NVIDIA with CUDA.
As the areas of expertise and manufacturing grow closer, the advantages of paying someone to do it for you decrease.
I know much too little to have an opinion on who is likely correct, but I can understand the two sides each having positions that don't assume the other side is an idiot.
I'm thinking back to strategies like the 486SX https://en.wikipedia.org/wiki/Intel_80486SX
So unless they cripple CUDA altogether, there will always be efficient workarounds (arguably DirectX or OpenGL programmable shaders in the very worst-case scenario). They even gave up on doing so for GTX Titan Black and then resumed with Maxwell. Currently, I would not be surprised that the lack of a true consumer Volta GPU is their only play at crippling consumer Volta by making it effectively nonexistent or $3000 for the GTX Titan V.
What they could do across the board is hamfistedly disable the deep learning frameworks on GeForce. That would probably stop 90% of amateur hour data science on GeForce. But the remaining 10% would just recompile them without the cripple code in violation of some sort of scary EULA clause against doing so and requiring such cripple code in all HPC/AI applications. I would love to see them try this - they'll pry my FP32 MADs (which is the core operation of AI/ML as well as vertex and pixel shaders) from my cold dead consumer GPU desktop.
I don't think they'll do that though. They know the low-end is the entry point to their ecosystem. They just want to force people to graduate into the high-end after hooking them. Not that you have to: multiplication and addition want to be free.
This sounds like a dumb idea, and it probably is. But consider a few things:
* NVIDIA GPUs have exceptional memory bandwidth, and memory can be a slow resource on CPU based systems (perhaps limited by latency more than bandwidth)
* The clock speed isn't that slow, it's in the GHz. Still one's clocks per emulated instruction may not be great.
* You can still do pipelining, maybe enough to get the clocks-per-instruction down.
* Branch prediction can be done with ample resources. RNN based predictors are a shoe-in.
* communication between "cores" should be fast
* a many-core emulated CPU might not do too bad for some workloads.
* It would have good SIMD support.
Food for thought.
That also means that you can really only emulate as many cores as the NVidia card has streaming multiprocessors, not as many as it has shared processors or "cores".
Also, it's true that GPUs have huge memory bandwidth they achieve that by trading off against memory latency. You can actually think of GPUs as throughput optimized compute devices and CPUs as latency optimized compute devices and not be very mislead.
So I expect that the single threaded performance of a NVidia general purpose computer to be very low in cases where the memory and branch patterns aren't obvious enough to be predictable to the compiler. Not unusably slow but something like the original Raspberry Pi.
Each emulated core would certainly have very good SIMD support but at the same time pretending that they're just SIMD would sacrifice the extra flexibility that NVidia's SIMT model gives you.
There are clever ways around this limitation, see links in my post this thread.
Not totally dissimilar to the thinking behind NetBurst which seemed to be all about having a deep pipeline and keeping it fed with quality predictions.
But take a look at the MOG code and see what you can do.
Check out H. Dietz' stuff. Links above.
However, it is quite feasible to emulate, on a GPU, a networked group of a general purpose CPUs( ie, run MIMD[1] programs on SIMD[2] architecture). This, MOG[3], has been a project of Henry G. Dietz of the University of Kentucky. Unfortunately, the project seems to have stalled at a "rough" level. He claims that he can run MIMD programs at 1/4 efficiency while also running SIMD programs at near full efficiency. His video is instructive [4].
Edit: Note that this isn't intended for deep learning applications as such but rather for traditional supercomputing applications (weather prediction, other physics simulations ,etc).
[1] https://en.wikipedia.org/wiki/MIMD [2] https://en.wikipedia.org/wiki/SIMD [3] http://aggregate.org/MOG/ [4] https://www.youtube.com/watch?v=FZ6efZFlzRQ
A perfect example is the Google new speech synthesis. Doing 16k samples a second through a NN is not going to be possible without your own silicon.
https://cloudplatform.googleblog.com/2018/03/introducing-Clo...
Listen to the samples. Then think the joules required to do it this way versus the old way and trying to create a price competitive product with the improved results.
>"LeCun and other scholars of machine learning know that if you were starting with a blank sheet of paper, an Nvidia GPU would not be the ideal chip to build. Because of the way machine-learning algorithms work, they are bumping up against limitations in the way a GPU is designed. GPUs can actually degrade the machine learning’s neural network, LeCun observed.
“The solution is a different architecture, one more specialized for neural networks,” said LeCun."
Could someone explain to me what exactly are the limitations of current GPGUs such as those sold by Nvidia when used in machine learning/AI contexts? Are these limitation only experienced at scale? Ff someone has resources or links they could share regarding these limitations and better designs I would greatly appreciate it.
The key challenge he pointed to was power - both in terms of getting energy in (modern CPUs/GPUs take similar current to your car starter motor!) and also getting the heat out. Logic gates take a lot more power than RAM, so he argued that collocating small chunks of RAM right next to your processing core was much better from a power perspective (meaning you could then pack yet more into your chip) as well as obviously being better from a performance perspective.
https://www.youtube.com/watch?v=Gh-Tff7DdzU isn't quite the presentation I saw, but it has quite a lot of overlap.
Hope that helps!
* For inference, how do you efficiently move your data to the chip? In general most of the time is spent in matmul, and there are lots of exciting DSPs, mobile GPUs, etc. that require a fair amount of jumping through hoops to get your data to the ML coprocessor. If you're doing anything low latency, good luck because you need tight control (or bypassing entirely) of the OS. Will this lead to a battle between chip makers? Seems more likely to be a battle between end to end platforms.
* For training, do you have an efficient data flow with distributed compute? For the foreseeable future any large model (or small model with lots of data) needs to be distributed. The bottlenecks that come from this limit the improvements from your new specialized architecture without good distributed computing. Again better chips don't really solve this, and comes from a platform. I've noticed many training loops have terrible GPU utilization, particularly with Tensorflow and V100s. Why does this happen? The GPU is so fast, but things like summary ops add to CPU time limiting perf. Bad data pipelines not actually pipelining transformations. Slow disks bottlenecking transfers. Not staging/pipelining transfers to the GPU. And then there is a bit of an open question of how to best pipeline transfers from the GPU. Is there a simulator feeding data? Then you have a whole new can of worms to train fast.
* For your chip architecture, do you have the right abstractions to train the next architecture efficiently? Backprop trains some wonderful nets but for the cost of a new chip (50-100M), and the time it takes to build (18 months min), how confident are you that the chip will still be relevant to the needs of your teams? This generally points you towards something more general purpose, which may leave some efficiency on the table. Eventually you end up at a low precision matmul core, which is the same thing everyone is moving towards or already doing whether you call yourself a GPU, DSP, or TPU (which is quite similar to DSPs).
Coming from an HPC/Graphics turned deep learning engineer, I've worked with gpus since 2006 and neural net chips since 2010 (before even AlexNet!!), so I'm a bit of an outlier here having seen so many perspectives. From my point of view the computational fabric exists we're just not using it well :)
A simple example is the Google New Speech synthesis service. It is done using NN on their TPUs but nobody needs to know any of that.
https://cloudplatform.googleblog.com/2018/03/introducing-Clo...
What the buyer knows is the cost and the quality of the service.
Now Google had to do their own silicon to offer this as otherwise the cost would have been astronomical. The compute to do 16k samples a second with a NN are astronomical.
If I could not see it myself I would say what Google did was not possible.
Just hope they share the details in a paper. If we can get to 16k cycles through a NN at a reasonable cost that opens up a lot of interesting applications.
https://arxiv.org/ftp/arxiv/papers/1704/1704.04760.pdf
But also this is a great example of why these new "AI" chip matters. This would NOT be possible with the Google chips.
https://cloudplatform.googleblog.com/2018/03/introducing-Clo...
I do wish Google shares more detail. Specifically how they are doing 16k samples a second through a NN. That has a lot of application beyond speech.
Some things Google will not share and this might be one of them but we can hope.
Then came a 486 and it got integrated again.
But during that time, the GPU split off. Companies like ATI and S3 began to dominate, and anyone wanting a computer with decent graphics had one of these chips in their computer.
Fast forward several years, and Intel again would bring specialized circuitry back into their main chips, although this time for video.
Now we are just seeing the same thing again, but this time it’s an offshoot of the GPU instead of the CPU. Seems like the early 1990’s again, but the acronyms are different.
Should be fun to watch.
I would tell a younger version of myself to focus your education on some aspect of the semiconductors industry.
Doing speech with 16k samples a second through a NN and keep at a reasonable cost is really, really difficult.
The old way was far more power efficient and if you are going to use this new technique which gets you a far better result and do it at a reasonable cost you have to go all the way down into the silicon.
Here listen to the results.
https://cloudplatform.googleblog.com/2018/03/introducing-Clo...
Now I am curious on the cost difference Google as able to achieve. It is still going to be more then the old way but how close did Google come?
But my favorite new thing with these chips is the Jeff Dean paper.
https://www.arxiv-vanity.com/papers/1712.01208v1/
Can't wait to see the cost difference using Google TPUs and this technique versus traditional approaches.
Plus this approach support multi-core inherently. How would you ever do a tree search with multiple cores?
Ultimately to get the new applications we need Google and others doing the silicon. We are getting to extremes where the entire stack has to be tuned together.
I think Google vision for Lens is going to be a similar situation.
I remember over a decade ago, they even had mood analysis they could apply to listening to people. Far from new. Is it truly more effective or efficient nowadays? Or just getting marketed by companies you've heard of?
I question whether it is just 100x compute. Feels like more, since naturally speaking and friends didn't hog the machine. Again, over a full decade ago.
More, the resources that Google has to throw at training are ridiculous. Well over 100x what was used to build the old models.
None of this is to say we should pack up and go back to a decade ago. I just worry that we do the opposite; where we ignore progress that was made a decade ago in favor of the new tricks alone.
Hope Google will do a paper like they did with the Gen 1 TPUs. Would love to see the difference in terms of joules per word spoke.
Surprised. Curious if you can compare the inference per joules of Google 1st gen TPUs compared? Google shared a paper and the numbers are pretty impressive and was not aware of anyone else close to the gen 1 TPUs?
Here is the paper that you can use for the TPU side. Love so see someone else in the ball park? We really do not want just one company but competition.
Not to say that the likes of the echo and others aren't impressive. Just that the speech recognition is the least of those products. Fully transcribed voice mail was available for years with Google voice (even before it was Google voice), yet that seems to happen less now than it did when I first for the product.
So what changed? And why?
That is, I am interested in comparing them on speed of transcription, speech synthesises, error rates, etc. Not on speed of network execution.
It is hard to believe they are able to do 16k samples a second through a NN even with the TPUs.
So be curious to see if reduced and how much?
If they really do have the ability to do 16k a second at scale that opens the door for all kinds of other applications.
So the company that made the chip had no skin in the game with running the chip or the cost of the electricity to run it.
Today we have massive cloud with Google and Amazon and lowering the cost of running their operations goes a long way unlike the days of the past.
This is why we will see more and more companies like Google create their own silicon which has already started and well on it's way.
Not only the TPUs but Google has created their own network processors as they quietly hired away the Lanai team years ago.
https://www.informationweek.com/data-centers/google-runs-cus...?
Also this article helps explain why Google built the TPUs.
https://www.wired.com/2017/04/building-ai-chip-saved-google-...
The premise from the title seems plausible, although NVIDIA seems to be catching up again fast.
First time on HN I can read a paywalled article, as I have a Barron’s print subscription.