Hacker News Reader

Google has always been strategic about announcing what it was doing, even since the early days (I used to work there too). Think about the impact the first MapReduce and GFS/BigTable papers had.

My guess as to why they're announcing the TPU is that they are feeling the pressure from Facebook and other AI labs, and want to reinforce their reputation as being the best place to do AI research. By revealing that AlphaGo was based on this hardware, they indicate to researchers around the world that if you want to build the most advanced ML models you need to be at Google. Same reason they talked about MapReduce/GFS back in the day.

> By revealing that AlphaGo was based on this hardware

Interesting, as the nature/science paper made no mention of this, it was exclusively trained on GPUs.

Training and inference are not the same thing.

Sure... I think you are missing the point. In Nov., the Nature paper which contains the AlphaGo algorithm, the hardware detailed was exclusively CPU+GPUs.

Between that time and the Lee Sedol match, the hardware running AlphaGo was switched, to these TPU.

From the paper: "The final version of AlphaGo used 40 search threads, 48 CPUs, and 8 GPUs. We also implemented a distributed version of AlphaGo that exploited multiple machines, 40 search threads, 1,202 CPUs and 176 GPUs."

Or they consider the TPU to be something similar to a math co-processor, it just offloads work from the CPU. They also didn't discuss the custom built power modules, or the custom built network switches used for interconnects.

Note that the version of AlphaGo that beat Fan Hui and was presented in Nature is significantly different from the Version that played Lee Sedol. Unless you believe that they didn't work on it for half a year.

This explains a lot. Going into the match, Sedol thought he could beat DeepMind but thought it might only be a couple of years until the technology outpaced him. We knew Fan Hui and other Go professionals were helping the team, but a massive speedup is always nice too.

It's a bit underhanded, however. IMHO, the player should be able to study recent games before the match. But this is pretty typical, there were similar late-stage improvements with Chinook (checkers) and Deep Blue (chess).

It's possible it was trained on GPUs but ran on TPUs.

It really boosts Google Cloud Platform with image and prestige, they're coming out with what is pretty much tensorflow as a service with their machine learning product.

Coming out saying they can give you a service no one else can right down to a custom chip may sway a few buyers in the market.

https://cloud.google.com/ml/

For me this is true. GPUs on Amazon are not cheap, and if google passes the savings on from performance power consumption I'd certainly offload some work to Google cloud.

This is true - I've heard rumors that GCE has been trying hard to poach cloud consumers from other providers, this could be another stroke in that effort.

Well this also shows their commitment to Tensorflow if they willing to pay to fab custom chips for it. Then again, in the Google scheme of things that's probably not a huge cost.

They've probably carefully modeled the energy savings over GPU/FPGA and found it to be substantial enough, even taking into account costs of design changes.

If you need enough chips it doesn't take particularly careful modelling to say that you'll beat an FPGA quite substantially in most cases.

It might also give Nvidia some sorely needed worries.

"We have an FPGA, oh but need to go faster, ok, build an ASIC then" is a natural thing to come up with. That's kind of what bitcoin farms did.

Obviously details and plans how it was done is where all the good stuff is, so that being hidden is understandable.

I'm not even a hardware or AI person, and even I could have told you ASICs would make way more sense than GPUs or FPGAs for machine learning. It's all about data locality. Fetching memory is the most costly thing a GPU does, and for ML (DNNs) there no big need for global access memory 99% of the time.

Anyone the casually follows AI knows that people have been talking about making DNN ASICS for some time. It was all a matter of time and $$$$$$

There is no doubt FB is working on them too. Which is why Google is finally publicaly saying that "we did it first ;)"

> Google is finally publicaly saying that "we did it first ;)"

Makes sense. In that respect yeah, they probably wanted to keep it under wraps to avoid Facebook/others from getting a timeline estimate out of it and jump ahead.

> It's all about data locality.

That's kind of what Movidius says, too, about its Myriad 2 VPU, which is kind of a GPU (SIMD-VLIW) with larger amounts of local memory combined with hardware accelerators.

It makes a lot more sense now why they open sourced Tensor Flow -- they have an ASIC design that supports it, and if they either want to sell the chips, or sell access to them via Google Cloud Platform, they'll need customers. And it's not like HP is about to roll out a server line sporting these any time soon, so Google will continue to have their advantage over the outside world.

> it's impossible to deploy GPUs at scale, let alone ASICs

A bit old but cf. DE Shaw and Anton https://en.m.wikipedia.org/wiki/Anton_(computer)

I think the jury is truly still out on whether the anton designs are worth it (compared to commodity clusters/GPUs, etc).

and Anton wasn't really "scale" in this sense. It was a vertically scaled single (or several) machines.

OK, so what is it? The announcement neither says what a TPU actually is nor what it can do. It's a magic black box. No specs. No price.

Well, it's a first announcement on a blog. They say it accelerates TensorFlow by 10x. They say it fits in an HDD slot. And the whole announcement must stay within a page or two.

It's a "more details to follow" type of thing. Pretty standard actually.

The cool thing here is that it's already widely deployed in production, so we know it actually works and that it's not just vaporware. The "more details to follow" is thus a lot more convincing than for a lot of similar-seeming announcements of things that don't actually exist yet (and often don't ever).

> They say it accelerates TensorFlow by 10x

They say 10x performance / watt, nothing about performance per unit time.

You can make some assumptions though. If the power consumption was equal, the performance is 10x.

The speed at which an ASIC will run is constrained by temperature (power dissipation) and and logic timing, which itself has a dependency on temperature.

So we could call that vertical scaling, to some power ceiling which may not take us all the way to 10x, but it's not impossible.

Then there is horizontal, which I assume is applicable to these problems... running more in parallel.

In both cases, I think it's safe to assume they are getting a performance increase in the instantaneous sense.

> You can make some assumptions though. If the power consumption was equal, the performance is 10x.

While I agree some performance per unit increase is likely, how does a direct 10x increased based on power savings follow? Less power usage does not mean that the chip can run through more flops in the same amount of time, right?

It does if power was the limiting factor in clock speed.

The relationship between clockspeed and power consumption is nonlinear.

http://electronics.stackexchange.com/questions/122050/what-l...

(see graph in the first answer)

Also, it's not known that the TPU have a way to allow to increase the clockspeed arbitrarily, nor is it known whether their architecture is capable of ensuring correctness at arbitrary clock frequencies. Some architectures make assumptions like "The time for this gate to reach saturation is very small compared to the clock frequency, so we'll pretend that it's instantaneous."

> They say 10x performance / watt

Well, that's the kind of metric you'd expect from a cloud provider. That's what's important to them.

If you're a tinkerer dabbling in TPU acceleration on your gaming/coding PC alongside with GPU acceleration, then the metric that would be interesting for you is speed increase per unit.

In March at the GCP NEXT keynote [1], Jeff Dean demos Cloud ML on the GCP. He casually mentions passing in the argument "replicas=20" to get "20 way parallelism in optimizing this particular model". GCE does not currently offer GPU instances. I've never heard the term replicas in the GPU ML discourse. These devices may enable a type of parallelism that we have not seen before. Furthermore, his experiments are apparently using the Criteo dataset, which is a 10GB dataset. Now, I haven't looked into the complexity of the model or to what extent they train it to, but right now that sounds really impressive to me.

1: https://youtu.be/HgWHeT_OwHc?t=2h13m6s

Since nobody has pointed this out, although he may be able to tell people "I was working on that thing", he likely can't say much more. Many large companies don't allow employees to disclose details that haven't been released publicly.

It seems it is a GPU-like device but tuned towards neural net applications, and with a more convenient HDD-like formfactor.

The article doesn't actually imply (though people might wish so) that the described TPU is intended to ever be a product available to others.

It's a post saying "that's how we do it" that serves a bunch of political and PR goals, but it's quite likely that this will stay an internal technology; maybe available indirectly as a cloud computing offering - in which case they'll give the specs and price of the whole solution, not of a particular model of TPU chip.

Hold up, you mean teams at Microsoft are "mailing zip files around!?"

How different are TPUs from GPUs? From the article, it sounds like TPUs use lower-precision arithmetic: are there any other differences?

Not just lower precision but probably less error correction in the lower bits. -Or- ... they convert the floating point values to analog values and do all the math in the analog domain and convert it back, just like old analog computers. But by going fully analog, I think the gains would be over 10x better, so probably not.

See 'low-power (inexact|approximate) computing'

They likely use half precision (i.e. 16 bit) floats in some of the computing stages. They've also discussed using even fewer bits of precision in some of their research output. When training and following an error gradient you kind of need lots of precision, but when running a learned model you often don't need much at all, 8 or even 4 bit numbers are sufficient in some calcs where rounding error propagation isn't an issue.

Now that you are free to talk about it, could you explain how ASICs stack up against GPUs, in practical terms?

ASIC stands for "application specific integrated circuit" and the name conveys a special purpose design of an integrated circuit, like for example bit coin mining.

But that's just terminology/convention. One could argue a GPU is an ASIC, and a CPU is an ASIC. The only thing to argue is how specific does an application have to be to call it an ASIC instead of some other made up name.

The prime differentiators are the process, which is the term used for the many steps of fabrication of an IC (processes are often referred to as nodes, distinguished by the smallest feature size of a transistor they create), and the ability of the design engineers to create an optimal design, in terms of boolean logic and semiconductor physics.

Given the right team of engineers, and a top notch foundry, and a great deal of experience in the problem domain (machine learning in this case), a custom IC could very likely trounce a GPU.

GPUs were designed for the domain of graphics processing, which happens to have some commonality with the processing in machine learning. But, at least until recently, GPUs weren't focused on machine learning. Just graphics.

Now the GPU vendors are trying to leverage their knowledge of graphics processing and building of graphics processors to create machine learning processors, but the thing they are leveraging could also be what handcuffs them. Which gives opportunity for a company like Google to do a fresh take on the problem domain without the baggage of the knowledge of graphics processing.

Wouldn't you typically prototype the design on an FPGA and then manufacture an ASIC once you'd worked out the kinks?

I'm suspecting that the design contains a lot of repetitive units, each of which can easily be tested in software. So perhaps testing on an FPGA isn't even needed.

I think zymhan meant prototyping on an FPGA to see if you could actually see an improvement by mapping parts of your algorithm to custom hardware. This would be a sensible first step before committing to a full ASIC run. I'm sure Google did do this initially.

As for testing for correctness, ASIC designs are almost always tested in software simulation instead of on FPGAs. Mainly because RTL written for FPGAs can look quite different from RTL written for ASIC synthesis. There's also a lot incidental engineering work needed to get something working on an FPGA.

> ASIC designs are almost always tested in software simulation instead of on FPGAs. Mainly because RTL written for FPGAs can look quite different from RTL written for ASIC synthesis. There's also a lot incidental engineering work needed to get something working on an FPGA.

It's actually very common to test ASIC RTL on FPGA. You are right that if you are targeting an FPGA you may write things differently, but this doesn't mean RTL written for an ASIC won't work it's just it might not clock as fast as it could and may not make the most efficient use of resources.

There is a lot of engineering work to take an ASIC targeted IP and run it in an FPGA, but thorough verification of an ASIC design is extremely important so it's usually worth the time to bring up an FPGA that can run it (sometimes as it's too large you have to split it across multiple FPGAs).

Ah I'd state the opposite. Your design will have new problems showing up when going from FPGA to ASIC

Especially if you try to make it go faster

Chip designs are usually simultaneously tested in software simulation, FPGA and hardware emulators (basically special purpose super computers that run Verilog or HDL, not quite FPGA speeds but far better debug capabilities, approaching that of software RTL simulation, downside is they cost a few million a piece).

You'll also break a design down into units or blocks. These can be tested separately. E.g. you might see an L1 cache as a single unit and have a testbench (piece of HDL to stimulate and check a design) to test only that. This is useful as it's far quicker than a full design test however extensive full design testing is still required as many bugs you see as the consequence of the interactions between units.

Once you've done a lot of testing you'll tape out a chip. This may well have some issues so you'll do a new 'spin'.

I've only dabbled in this area, so it's entirely possible my question doesn't make sense. Machine learning algos in general seems to have a bunch of iterative components, but within each iteration there is massive parallelism. Did you consider using a whole bunch of DSPs? They can also do parallel processing but might be cheaper in bulk than GPUs.

My vague impression is that there had started to be more and more research on use of ASICs for ML (though I can't say I follow it that closely). Enough of a changing tide that the competitive advantage was going?

What was your design flow? Did you use the standard cadence, Synopsys toolset to build this? How did you achieve basic design and JIT(or whatever it is you call the machine code part of the Python VM ) together?

Rrally interested to know how this was managed...

any idea which foundry google have used for the fabrication?

I wonder if this is a response as well to Nvidia's announced Tesla P100 chip?

There were no references to this TPU in the TensorFlow source code?

No but if you look at the code it's clearly designed to support pluggable hardware.

Very interesting observation -- this seems so much more obvious in retrospect given the meticulous mapping of graph to devices outlined in the paper.