Quantifying the performance of the TPU, our first machine learning chip

Memory controllers are not so simple to do, and fast MCs also eat quite a bit of power. So a simple reason that they didn't do it could be either

a) they did not want to license a more expensive, faster design

b) while it would be faster, it would decrease efficiency to a point that did not meet their goals (for data centers, efficiency > absolute performance, within reasonable boundaries)

c) like b) just with cost of memory

d) GDDR5 and DDR3/4 have different design trade-offs. The former is optimized for sequential bandwidth (and low capacity; GDDR always was a point-to-point memory bus just to achieve the clock speeds), while DDR3/4 takes random read/write workloads into account (eh... to the amount possible with DRAM...)

--

HBM requires a silicon interposer, which is basically like another complete chip (just without the FEOL parts, "just" metallization), that has to be significantly larger than the size of all chips combined. So unless you really need that performance or have a volume product it's unlikely to be a good deal.

GDDR5: Getting the controller and PHY is difficult, not many companies have them and the ones that do (NVIDIA etc) don't have a strong interest in licensing them. The PHY is a mixed signal design so requires specialized expertise and likely some debugging on the target process so not only would it take a long time but it'd be expensive and very possibly flaky in rev 1.

HBM: The interposer means you get a complex, and risky manufacturing process.

Well HBM requires integrating the memory into the chip, presumably they didn't have the manufacturing capability or didn't want the spend required on a first version.

No idea on GDDR5 vs DDR3, maybe they didn't like the latency of the first.

If "forward passes" means inference (as opposed to training), then the post says the first generation targets that. I don't think they say anything about any future generations (other than they're working on them).

Some parts of stats (esp. classically) do use "inference" for the whole process of going from data -> result, especially when doing descriptive rather than predictive statistics. In most of ML the process is split into two phases, training a model on a data set, followed by using the trained model to predict labels (or whatever else it's predicting) for new data. Training is an implementation of statistical induction, from data to model, while model "use" or "evaluation" is an implementation of (probabilistic) deduction, from model + query to result. In ML, "inference" is usually a synonym for the "deduction" or "evaluation" portion. It makes sense to me intuitively if you think of it as learning things vs. inferring things from learned knowledge.

But you do find constructions using it as a synonym for training too, as in phrases like "model inference" (which means inferring models from data, aka model induction or training). Inferring things from other things is a pretty general concept, so it can be slippery without context...

Inference in this context means getting the output of a neural network on some input. Training meaning adjusting a neural network to so that it's outputs are more like the training data.

Training involves finding derivatives and requires higher precision calculations which is why they don't train using TPUs.

Training is inference, except over model space instead of possible outputs. Depending on what models you're looking at, the two can have very different computational properties, so treating them separately is not surprising even if what they're doing is conceptually similar.

That's like saying "so basically it's a chip". Well yes, it is a chip.

Most accelerator-esque chips are often considered some kind of ASIC. CPUs are not ASICs because ??? ... never heard a good reason there. It's a fuzzy (but not fluffy) term.