TPU transformation: A look back at 10 years of our AI-specialized chips

That wouldn't work. Even TPUv4 was on a 7nm node and you don't just build a 7nm fab just like that. If it were that easy NVIDIA would already be building their own fabs, as they have basically raised as much capital as there is in the known universe (bigger market cap than the entire London stock exchange), but they seem to prefer to let the fab experts get on with it rather than compete with them.

LLM AI is largely HBM bottlenecked anyway i.e. Samsung, SK Hynix and Micron are where the supply chain limits enter the picture.

There seems to be this idea that the people who design and operate fabs are infinite, when it's actually a technically demanding job.

We don't even have enough McDonald's employees, how the hell are we going to just suddenly have multiple companies creating fabs left and right? TSMC cannot even build their Arizona plant without a shortage of workers.

Intel has been trying to make cutting edge fabs...and we all know how that is going.

There is good reason nobody wants to be in the fab business.

Do you think that NVidia is happy to not have an online ecosystem to tie to its GPUs, for added (sales) value? They are more than happy to entangle the GPUs with their proprietary CUDA language.

For a large, established, quasi-monopoly company it's always more attractive to keep things inside their walled gardens. Suggesting that Google should start supporting TPUs outside Google Cloud is like suggesting that Apple should start supporting iOS on non-Apple hardware.

That is correct, and I should have been more, clear: when I say "Buy them" I mean direct sales of the hardware from seller to buyer. I am not referring to cloud compute style sales. Yes, they have been offering TPUs through Google Cloud for a long while now, but this still falls under all the stuff I said earlier: they don't need to have sales pipelines or channels (outside GCloud's existing ones), they don't need to design the hardware/software for arbitrary environments, they have one set of staff and machines to train and support, etc. All of that stuff costs money and ultimately it results in an entirely different sales and financial model.

Google could spin the TPU division out of Google, but 99% of the time people refer to moves like that they omit the implied follow up sentence which is "I can then buy a TPU with my credit card off the shelf from a website that uses Stripe." It is just not that simple or easy.

Isnt Groq pivoting to the IaaS/SaaS model because hardware channel sales is hard and its easier for everyone to just use an API?

Yes, and now after years of doing that Groq is pivoting to being a cloud compute company, renting their hardware through an API exactly the same way Google does.

Building out your own vertically integrated offering with APIs is comparatively a lot simpler and significantly less risky in the grand scheme. For one thing, cloud APIs naturally benefit from the opex vs capex distinction that is often brought up here -- this is a big sales barrier, and thus a big risk. This is important because you can flush mid-8-figures down the toilet overnight for a single set of photomasks, so you are burning significant capital way before your foot is ever close to the proverbial door, much less inside it. You aren't going to make that money back selling single PCIe cards to enthusiastic nerds on Hacker News; you need big fish. Despite allusions to the contrary (people beating down your door to throw you bathtubs of money with no question), this isn't easy.

Another good example of verticality is the software. The difference in scope and scale between "Tools that we run" and "Tools you can run" is actually huge. Think about things like model choice -- it can be much easier to support things like new models when you are taking care of the whole pipeline and a complete offering, versus needing to support compiler and runtime tools that can compile arbitrary models for arbitrary setups. You can call it cutting corners, but there's a huge amount of tricky problems in this space and the time spent on procedural stuff ("I need to run your SDK on a 15 year old CentOS install!") is time not spent on the core product.

There are other architectural reasons for them to go this route that make sense. But I really need to stress here that a big and important one is that hardware is, in fact, a very difficult business even with a great product.

(Disclosure: I used to work at Groq back in 2022 before the Cloud Compute offering was available and LLMs were all the rage.)

Only because of Nvidia's margins. "Worse but cheaper" is actually great for a company of Googles scale, but it doesn't make for a particularly compelling press release or paper.