Nvidia Is Building Its Own TensorFlow Processor Unit | Hacker News Reader

Nvidia Is Building Its Own TensorFlow Processor Unit

forbes.com

140 pointsmchugbl45 comments

Finally! One of the big boys is talking about embedded deep learning at the consumer level. (low cost)

Open Source? Wow did not see that coming. It’s good to see Nvidia is not trying to lock up the low end IOT Deep Learning chips. It makes sense at they want to sell more GPUs for Training.

What can you do with a $5 Pi Zero like SoC with a TPU/DLA/DSP that runs on 200 milliwatt that can infer deep learning models as well as a desktop CPU? Yes onboard training would be nice, but models don’t always need to be updated in real time. Also you can't always rely on the cloud...

$5? You still need to store the weights, and with 100's of millions of parameters@32 bits each that still won't be cheap. It'll be a lot cheaper than a GPU though and it will use far less power.

You can quantize it to get the parameters down to 8 bits.

Actually even 1 bit might be enough.

Yes, $5. Not every thing needs to be the standard 224x224 AlexNet/VGG16/ResNet. My work horse is LeNet @ 1.6MB with 32bit float weights and no compression or other tricks.

Oh that's cool if your problem is that small. But if your weights are that tiny then do you even need an accelerator? Or is it still too slow on the CPU?

The application NVIDIA targets their TPU at is clearly vision/robotics and while there are probably many vision/robotics tasks that can be done with very small nets I think the majority of them will require something a bit more beefy.

You can take a trained model (TPU, GPU accelerated) and replicate it to an embedded board. It would be possible to use the Cortex M4F FPU to accelerate the deep learning inference functions. Using a TPU-like core on a microcontroller/SoC would be an improvement.

Deep Learning inference processing is not reliant on the cloud, and once trained can work fine without any connectivity.

Strategy is, use expensive GPUs to train.

When your model is ready to deploy at scale, take the matrices and deploy them in efficient and cheap ASICs.

Nvidia wins if the product of the GPU is as easy to deploy at scale as possible for everyone, not just Google.

A lot of people who don't have the scale are going to use GPU/CPU for the whole pipeline, and the people who have the scale weren't going to use GPUs in the long run, this just helps them get there faster and realize return on their GPUs.

Well,

The broad paradigm of deep learning (and machine-learning generally) is a "train-test-deploy" cycle and this fits with it. The thing is that however successful this approach has been so far, it has rather clear limitations. Unlike a scientific discovery, what's being discovered isn't a universal rule but a heuristic between massive, real-world data and labels/qualities. As the world changes, the actual correlation changes but the deployed solution doesn't (and sure you can re-teach, re-test and so-forth but what if the required model changes, what if your experts leave, etc).

So it seems like to make a deep-learning solution sustainable, you'd want a method of including learning in your deployed solution (how you'd do that may not yet be discovered but that doesn't mean it won't be discovered).

But this approach seems to do the opposite. It bakes the basic learn-test-deploy process into silicone, making this approach more obligatory.

> As the world changes, the actual correlation changes but the deployed solution doesn't

Are you saying that self-driving cars might start crashing when pedestrians wear different fashion in the future, or when cars have different designs?

That's roughly the gist of it. If the validation set is a subset of the training set then you are making a whole pile of assumptions that need not necessarily be true.

Since the network can't explain what those assumptions exactly are it may make conclusions that work empirically at training time and that pass validation but that have no solid theoretical underpinning as soon as those assumptions fail the systems will fail.

See also: data leakage

Continuous learning is hopefully going to address some aspects of this.

https://deepmind.com/blog/enabling-continual-learning-in-neu...

And many more future tricks like it.

Or essentially that deep learning is unable to discriminate between beliefs and facts. And we can't read the models well enough to tell which is which.

So it would be like being handed a list of high level bridge building rules with no context, then being asked to build a bridge in a geological environment where one has never been built before.

Will it work? Who knows! How would we say?

Well, it's early days. We already know it is possible to generate pathological inputs to elicit a false response from a network much like optical illusions can fool humans.

For instance, in the context of adaptability of self driving cars:

The retraining could theoretically be fixed using an OTA update that periodically pools new training data from in-service vehicles to further improve and generalize the models. You'd have to build that in from day one but presumably anybody busy with self driving cars would have this at the forefront of their thinking.

Just so that when and if black clothing with white stripes becomes fashionable we don't end up with a large amount of roadkill because self driving cars interpret these as empty lanes.

Whenever the sensed input changes.

To use the car analogy: when additional or fewer sensors are used, when a different part number sensor is used, when driving laws are changed, when road signs or traffic lights are updated, when other cars begin behaving differently, etc.

Other people adapt to self-driving cars. They walk in front of them expecting them to stop. They find and exploit artifacts in the way cars behave to make them do funny things. They take it beyond a tipping point, they create situations outside the training set. Maybe it crashes, or maybe it doesn't crash but gets stuck or doesn't go where it's supposed to.

You act like the train-test-deploy cycle only happens once. But in reality wouldn't teams do this continuously? Training/improving models would happen regularly on the side.

I'm not sure I'd call an "online" learning algorithm more reliable. They have plenty of tradeoffs.

> "You act like the train-test-deploy cycle only happens once"

My impression is currently the train-test-deploy cycle is very much not an automatic process but rather a tricky, difficult process which experts working and scratching their heads to get working. The phrase for getting a deep learning system working is often "graduate student descent". Sure, you could just blindly run the process again but there's a lot to know to get it right - why we don't just have "deep learning in a box" right now (though people are working it).

So yeah, I am sure the process is re-done periodically but redoing may not be a slam dunk as is (though people are working on that too, people are working everything...).

And I'm not advocating any particular approach, just observing current approaches.

You're ignoring completely Reinforcement Learning and the broader area of Unsupervised learning which is built exactly to counter the point of a "static" NN.

You can't do online optimization continuously but you can gather input and batch it and update the ASICs on a cycle.

Like Google has guys with carts patching failed hardware... every week you reprogram a new set of ASICS and go around swapping them out.

tbh I'm not sure you need to replace ASICs except when you change the architecture of the model...presumably an ASIC can load parameters at startup/runtime?

There is very little about design and manufacture of ASICs that is cheap, unless you're selling them by the millions. Two previous employers evaporated because they couldn't sell enough units to recoup VC investment in building semi-custom ASICs. The kicker is that each employer was already selling a previous generation of their product using more plug-and-play, less custom, ASIC libraries. The customized next-gen products were intended to be more performant and power efficient, which they were, but they still sunk both companies.

I think that might be a mistake. It should simply be Tensor Processing Unit. Tensors, not tensorflow (which is an application). Am I wrong?

It's possible they are building hardware specifically for TensorFlow's API, but it's also possible the journalists are just not tech-savvy and don't know the difference.

That seems more likely. They've working hard on creating their own complete deep learning ecosystem with TensorRT built on top of CUDA and cuDNN, DIGITS, etc.. They do support all the popular deep learning frameworks, and I don't see that stopping any time soon.

You are not wrong. Forbes only expanded TPU to "Tensorflow processing unit" in the context of the Google chip.

No, you are definitely right

Vaporware announcement 2 days before Google I/O...I wonder what Nvidia expects Google to announce.

That they're selling their TPUs to the public?

Volta isn't vaporware.

The article says the title here, but doesn't TPU stand for "Tensor Processing Unit", not "TensorFlow Processing Unit"?

I remember early releases saying that and Wiki[0] seems to agree.

I suspect that this is just lazy tech journalism.

[0] https://en.m.wikipedia.org/wiki/Tensor_processing_unit

There's a big gap between the price of NVidia link servers and servers with a bunch of 1080ti's or Titans in them. I think that this must be to do with commodity production and pricing. GPU's are sold to gamers. Gamers want 60fps 4k gaming and there is a large market for cards that support this. I want GPU's to train neural networks - but I think that very few businesses are buying 20 GPU's and sticking them in servers. Even fewer are buying 20 NVidia link systems. There is a market here, but it's small and hard for a gaming company to address. I think NVidia know that there is a lot here for them, but they must look at the numbers and wonder.

> but I think that very few businesses are buying 20 GPU's and sticking them in servers.

No, they're not buying 20, they're buying 100's.

And not just for deep learning either, CFD and all kinds of other computationally expensive algorithms are more and more found on clusters of servers with GPUs in them instead of mainframes. There is definitely a huge shift happening there.

So correct me if I am wrong but when Google created their TPU the only advantage was inference with high power efficiency . NVIDIAs response on making their own took a year because Xeons are nearly everywhere and can do inference but less efficiently. I imagine since a TPU is a specialized hardware it makes more sense in mobile and then a data center looking at how slow AWS deploys hardware .

This is targeted straight at self driving cars and robotics applications and presumably inference only.

Intel needs to double or quadruple their processor's AVX width, add 8 bit instructions and make it slightly more power efficient (for AVX) and it will easily beat many of these TPUs. Of course this is easier said than done but I am really surprised they haven't done it so far and instead bought Nervana and other hardware chip startups.

None of the above is easy, as far I know AVX is one of the most thermally-constrained parts of the CPU. Xeon Phi is the best Intel managed to come with.

Have you read about the Google TPU? It's got 65536 ALUs running in parallel turning out a result every clock:

https://www.nextplatform.com/2017/04/05/first-depth-look-goo...

They claim faster memory (GDDR5?) could easily triple the performance which should bring it to 270TOPS or so. I don't think extending AVX is going to get there any time soon.

All this raw compute/parallelism is great but it does not really help the algorithm output in terms of efficiency. The CPU/GPU difference is much smaller than nvidia would like you to believe especially for more complex networks which are becoming more and more common now. Of course if you want to just do convolutions (which the TPU paper claims is only 5% of their google workload), building a hardware around it may work well for specific algorithms.

GPUs are already SIMD monsters, and even if you make them 8-bit then they won't be anywhere near what TPUs/DPUs can achieve because the TPUs/DPUs are in a specific arrangement for deep learning. Besides, there's tons of bloat in an OoO processor being used for deep learning...

In addition, people seem to disregard the fact that the biggest factor in power consumption is data movement, not the cost of computation, see here (slide 29 is the important one): https://www.ssken.gr.jp/MAINSITE/event/2013/20130827-sci-1/l...

Yea.. it's not a "deep learning processing unit", but a "tensorflow processing unit". And it certainly is not just a normal modern graphics card (GPGPU) that can do other-then-32bit-float computations without dying (as AMD's could do for a while now).

* Written using a FFPU (firefox processing unit).

How do I read forbes articles?

I only get a blurred page with a "Quote of the Day" that I can't close, navigating to the original URL of course redirects me back to this nonsense page, and trying to delete the covering elements with the inspector does not yield results. WTF?

If they don't want people to read their articles, they could just make it "HTTP 403 Forbidden, Status Code of the Day".

Try just refreshing the page, that works with my config (FF w/ Ghostery)

uBlock Origin seems to bypass the quote page.

Doesn't for me, I have almost all default lists enabled.

However, whitelisting forbesimg.com works.

See if this helps: https://chrome.google.com/webstore/detail/xray/dgkdfehohjdbm...

Reply on news.ycombinator.com