Amidst the noise and haste, Google has successfully pulled a SpaceX | Hacker News Reader

Amidst the noise and haste, Google has successfully pulled a SpaceX

markmaunder.com

69 pointsmmaunder63 comments

If they have 10x cheaper compute, and a lot of cash to spare, why isn't Google dominating AI, like SpaceX is dominating the launch of satellites?

Does Google not want to, or is there another reason they cannot?

There seems to be a consensus that we haven't yet seen the killer AI app that produces ROI that justifies the level of investment we're seeing. If that's true, then it's all still up for grabs.

My own view is that what AI is going after i.e. the size of the pie, is the $50 trillion per year portion of global GDP that makes up wages. Putting aside the risks - which are enormous - when you see it that way you begin to understand openai's valuation late last year, and that that's just the tip of the iceberg of what's coming.

The risk of disrupting a sizable chunk of the world's wages is fuel for a separate post and conversation. IMHO it's the real risk of AI, and the Skynet scenario around ASI/AGI that has been popularized is a distraction.

None of that matters if you are selling shovels. Cash is cash. Google could be selling TPUs and directly competing with Nvidia. You might think Nvidia’s valuation is inflated, but the fact of the matter is they sold $60B in hardware last year.

Nvidia is selling shovels. Google will be selling gold if successful. And they make their own shovels.

When I look at current ChatGPT I honestly think that the gold has been found, and a significant amount of it is undermining Google’s home market, the presentation and aggregation of information on the web. Their product has become so much worse over the years while ChatGPT has improved many fold. It’s a better search engine among many other things.

We've yet to see how enshittification will work it's magic on AI's. Maybe you're too young to remember AltaVista. Their search was OK'ish, and Google was slightly better - but AltaVista was the incumbent. It should have counted for something, but it was wiped out overnight. What killed AltaVista was them promoting paid search results without telling you. Ham fisted enshittification in other words.

Google resisted it for a while. But slowly ads crept to the top, and then became near indistinguishable from search results. Worse, SEO triggered by their ad business undermined their search, and they famously didn't fix that because the ad business is their cash cow. It is an indirect form enshittification, probably not even a deliberate move, but enshittification nonetheless. Enshittification is killing Google search.

Right now AI's look wonderful. The free ones as you say often replace search. But we are in the early phases. They are all surviving on VC money. That can't continue. Enshittification will hit the free ones, and when it does it seems like it will damage them far more than search. You can work around promoted search results by just looking at more results. How can you work around a hallucinating AI that's been paid to lie to you?

Oh absolutely - and I‘m not too young to remember AltaVista at all. Eventually everything new on the internet turns to shit. But that doesn‘t change what I wrote: Right now, ChatGPT is gold. It‘s the Windows 3.0 with Google being OS/2. Does that put a better age on me? ;-)

> My own view is that what AI is going after i.e. the size of the pie, is the $50 trillion per year portion of global GDP that makes up wages.

This is why AI is the smart investment play in the near term, but those ahead of the curve are long on canned goods and firearms.

Not only isn't there a killer app, LLMs are getting to be commoditized.

That will soon stop. When training is relatively cheap, many actors can do it.

When training requires a $100B data center, there are only a few companies in the world that can afford that.

I believe the killer feature would be sensible embedding tools with respective functions that support generating facts about internal data.

You generally cannot just push your data tables with millions of entries to your AI, but you can reduce them to statements that the AI then can work with.

It would be quite nice for training if a company can embed their product catalogue or their process instructions. Or technical data and common faults for customer support.

Also production monitoring would benefit from AI, but you also need to reduce the large amount of data into congestible statements. Usually the same work you do when you build some kind of dashboard.

In most cases that would mean you either run an AI locally, which I would heavily prefer, or you serialize the result of said embeddings and have them be used with your cloud AI of choice.

If you want to use an Nvidia GPU you can buy them and put them in whatever you want.

If you want to use a TPU you have to use Google cloud.

Why would someone want a TPU?

Not just that but, why would someone (not working at Google) want a TPU so much to put up with all the hurdles, including learning a completely new and non-portable programming stack?

Most people don’t care what hardware it is running on as long as it can do fast matrix multiplications. Right now CUDA is the lowest friction way to do that.

XLA is the compatibility layer. You train using Torch/Jax/whatever you want, and XLA ensures you can use TPU. There aren't many hurdles, most people train in the cloud anyway.

>or is there another reason they cannot?

I'll never get tired of saying this, extremely incompetent management starting from Shundar (or however that is spelled).

I'm quite sure they still have the best talent in the world, but they also have the kind of PM that does those dumb "my day as a PM in San Francisco" stories at the helm, this is the result, maybe a trillion USD worth of value is unrealized because of this.

> Shundar (or however that is spelled).

You could perhaps google it? And then you would learn that his name is spelled Sundar.

Also, “that” is not how one refers to human beings or their names. But at least we can know how much to trust your opinion. If you got tired postulating about their management but couldn’t be bothered to ever check the name of their CEO.

Agreed he could have searched for the spelling, but "that" is probably referring to the word, not the human the word refers to, so "that" seems appropriate here. I would have used "it" ("however it is spelled"). You wouldn't say "however he is spelled". I suppose you could say "however he spells it", but then you're using it to refer to the word in question, putting us back at the start.

> Agreed he could have searched for the spelling

Or just not qualify it. People make typos, nothing wrong with that. But the qualification means they felt they don’t know the spelling but couldn’t be bothered to do the 1 second thing and look it up.

It is one of those cases where being confidently wrong is less insulting than being wrong, suspecting that you are wrong but not caring about it.

> You wouldn't say "however he is spelled".

No. As discussed above I would search for it. But if one has to express this already bad thing one can write “however he spells his name”.

I guess the reason i’m sensitive to this is that I’m bored of seeing people performatively mispronounce names. Not saying that is what happened here. In fact it is probably not what happened. Just that it reminded me of that particular kind of uglyness.

The classic answer to any question like this at Google was "but does it have the same level of profitability as ads?" This was commonly cited as a reason Google shouldn't get into several different areas (cloud being one). I kid you not, one of the most common arguments I had when I joined google was that google was late to the cloud, the response was always "we have a cloud, it's called borg" and "we have a cloud, it's called appengine" while I'd have to explain to people that there was an entire market of people who wanted to "import numpy" on 128-core machines.

Another answer is that Google has quietly dominated industrial AI for some time; few people talk about SmartASS, or Rephil or SETI , and most modern ML folks would barely recognize Sibyl, yet those products help Google grow Ads into a monster business. YouTube (Watch Next) and Android (Play Store) both have Sibyl to thank for rapid growth at a critical time. Google played a big role in bringing around modern deep learning (voice recognition, language modelling) and is one of the largest, if the not the largest, industrial deep learning research publishers.

Another answer is that while google vends access to TPU thru cloud, it's a tricky product to maintain and sell for external users, and to keep the product affordable, they are making some very serious decisions about how much gets used internally vs. deployed (an Ads job running on a TPU is likely more valuable than a customer job running on a TPU) with massive capital expenditures on hardware and facilities that affect their earnings and profits.

Yet another answer is, Google is one of the largest resellers of access to nvidia GPUs but they don't dominate because they entered this business a bit late and their competitors (amazon and microsoft) are excellent at selling products to enterprise and at buying large piles of GPUs and running facilities.

I just don't think it's in Google's DNA to be highly profitable outside of a few businesses- a search page for ads, ads in videos, ads in applications, ads around the internet, etc, and it's unlikely to change any time soon, because their top leadership doesn't know how to cultivate new products (stadia and google+ being two particularly egregious examples).

> I just don't think it's in Google's DNA to be highly profitable outside of a few businesses

I've often wondered if this was the reason why the entity Alphabet was created, with X projects to "graduating" into standalone companies. Because otherwise most of them would die an unceremonious death from not being as profitable as Ads.

Pretty sure you're correct.

Google was the Alpha Bet and the other projects where the Other Bets.

The long game isn’t done yet, and “dominating AI” doesn’t mean “most popular chatbot in January 2025”

Contrary to the title, fhe post is comparing Starlink's access to cheap launches, not SpaceX launch capability vs the rest of the market.

> What Google has done by vertically integrating the hardware is strategically similar to SpaceX’s Starlink, with vertically integrated launch capability. It’s impossible for any other space based ISP to compete with Starlink because they will always be able to deploy their infrastructure cheaper. Want to launch a satellite based ISP? SpaceX launched the majority of the global space payload last year, so guess who you’re going to be paying? Your competition.

If people aren’t getting paid, then goods aren’t being bought and GDP will crash.

I don’t know what the end goal here is. Any sizable impact on wages will directly impact every other business sector.

Apple will sell phones to the few remaining Tesla employees, Tesla will sell some cars to the few remaining Apple employees, and the rest us will just eat each other?

There is no end goal. It's just a bunch of independent actors who are all trying to get ahead of their competitors a step at a time. Two steps if they're lucky.

The collective system steers into whichever direction was determined by the previous set of steps, with very little potential for any individual actor to turn it around.

does google know how to sell hardware to hyperscalers?

The 10x figure is made up. We don't know if or by how much the TPU ecosystem is cheaper than Nvidia.

Firstly, the h100 margin number appears to be looking purely at the manufacturing cost vs sale price. Where are amortised driver, software stack and design costs in that? Nvidia has won because of the strength of the CUDA stack, including their uniform driver codebase which isn't free to develop.

The cost of a tpu or GPU isn't simply in the hardware costs. Google has to maintain an entire parallel software ecosystem as well. Then we can assume Nvidia is spreading design costs over many clients including gamers and the console industry as well, whereas Google isn't.

It might well still be a saving for Google because Nvidia is making bank right now, but if it was actually easy to replace these chips with in house designs and software stacks then all the big cloud companies would be doing it. They're trying, but so far those projects haven't worked out.

The author is right about the vertical integration. It’s just that none of the stages are great. I don’t use Google cloud and have no interest in learning its idiosyncrasies. And I have PTSD from Tensorflow. (It’s only a bit of an exaggeration to say no one outside of Google uses it anymore.) Using TPU in PyTorch is possible though.

I wish Google would compete with Nvidia directly and let me buy a TPU (the big boy ones, not the flash drive sized Corals). Because as it stands now, Google could have Nvidia’s business and OpenAI’s.

> It’s just that none of the stages are great.

Really? Nobody uses TF but plenty of people use XLA, which is what you allude to saying that TPU in Pytorch is possible. That's arguably the most important piece; the sledgehammer that makes good performance and good devex compatible.

> I wish Google would compete with Nvidia directly and let me buy a TPU

I don't see the utility of buying _any_ GPUs. It's mainly a cost + availability optimization to own them yourself if you're doing a lot of continual training. Outside of the foundation model companies, most of us just use cloud services -- and I want those to be cheap and always use the latest thing.

Google has a highly optimized infrastructure to make sure unused resources (CPU, TPU, RAM, disk) are properly allocated. This means preemptible instances, highly co-located compute + data, etc. In theory everyone should win when you use ML hw through hyperscalers, because they collect on their structural advantages and your TCO is lower.

> Google could have Nvidia’s business and OpenAI’s

They arguably have a better _business_ on their hands but a worse _speculative outlook_ in the eyes of Mr. Market. Those are different things. There's an argument to be made that Google's valuation could be as high if they ran their public relations strategy as well as OpenAI / Nvidia / Tesla, which are all riding monumental hype.

> most of us just use cloud services

Yes, but because cloud services buy GPUs almost exclusively Nvidia, it would likely drive cloud prices down as well.

Cloud prices are already driven down by the fact that you can deploy with Google's cloud offerings.

Also, if you weren't going to use Google's offerings you probably wouldn't be down to buy their HW either :)

FWIW your PTSD with Tensorflow is shared by everyone at Google, it was just unavoidable because Google needed ML to be performant and they started by sacrificing devex for performance whereas Pytorch went ergo-first, performance later. In retrospect a prescient move by Meta, but now making the high performance stuff ergonomic is proving out for Google via XLA and JAX.

OK but vertical integration and low costs is only half of the problem. You also have to make a product that people want, otherwise the whole enterprise is meaningless. SpaceX is selling launches every week. What is Google selling that uses all those TPUs, and to whom?

Behavioral advertising.

TPUs are used internally for products and research and also sold as a service thru Google Cloud. The list of TPU customers is pretty long. But Google could the product entirely internal and still reap plenty of benefit.

Gemini 2 was entirely trained on Trillium, Googles 6th generation TPU. About 100,000 of them. And all inference is run on TPUs.

https://venturebeat.com/ai/google-new-trillium-ai-chip-deliv...

Right. But are people paying for Gemini? Does all that vertical integration make it cheaper than its competitors? (Which is what works for SpaceX)

Gemini Flash is my go-to for many client projects, so there's at least a few people paying for it. If it fits your use case, it's easily the best value for money, and often makes something commercially viable that wouldn't otherwise be.

Sample size of 1, obviously, so YMMV.

I also pay for Claude Pro for personal coding use, so I'm aware that Flash isn't exactly a drop-in replacement for a more powerful model. From my limited testing, though, Pro 2.0 is almost indistinguishable from 3.5 Sonnet.

Even if the LLM hype dies, and nobody at all wants to rent TPUs from GCP, then Google can use the TPUs to train and run large advertising models and improve the quality of their search.

They themselves are a big enough customer to make this worth it. Any business from GCP is just icing on the cake. Keep in mind that TPUs can train non-LLM models too.

All the ad tech in the world won’t matter if people have stopped using the medium those ads are displayed on top of by then. I think the way people use the web is about to change drastically.

Very much this. I think the reason Google products are more and more infested with ads is to keep up revenue against people moving to better alternatives.

Google has been charging people actual money to access features running on TPUs for years. I helped launch TPU inference features for Gmail in 2015. There may have been earlier systems running on TPU that I wasn't aware of.

Targeted ads use a lot of behavioral models. Google sells ads.

Has this improved their ad targeting? In a way that shows up in the bottom line, or even the top line?

Yes

Just killing the competition is enough with their integrated products?, like google search with ai features,(perplexity.ai) Ai on android now gemini replacing google voice (chat gpt and others).

Many of those launches are sold to itself (starlink), just like google uses a lot of its TPU capacity for its own purposes.

Minor nitpick:

> NVidia’s margin on the H100 is 1000%. That means they’re selling it for 10X what it costs to produce

If the margin is 1000%, they're selling it at 11x what it costs to produce. Were they selling it at 10x, their margin would be 900%.

Don’t you just love percentages? Thanks.

One of the big things this article misses is that Google pays Broadcom a significant amount for the actual chip design, also around a 70% margin.

Google certainly has infra/cost advantages, but it's nowhere near 10x.

Mind sharing your source on that? I’ve been trying to find one.

Edit: Specifically the nature and current status of the Broadcom/Google relationship as it relates to TPUs.

https://www.theinformation.com/articles/to-reduce-ai-costs-g...

Which takes it from

> Broadcom generates a 70% profit margin from its work on TPUs, said a person with direct knowledge of the internal analysis. SemiAnalysis, a chip research firm, earlier reported that figure.

https://semianalysis.com/2023/08/30/broadcoms-google-tpu-rev...

Thanks!

Interesting allusion to Desiderata, “Go placidly amid the noise and the haste, and remember what peace there may be in silence.”

if AI is a weapon, google wins, but if AI is a product, apple wins. google can build compute capacity, but solving cringe at their products isn't a compute problem.

products are hit songs, who here is going to write them.

Since it's both, both companies will win big.

the "why not both?" worldview is kind of muddying and neutralizing to conversation tho. really, one model will prevail.

Did one Cloud prevail?

And yet Amazon and Azure have more market share in Cloud. Whatsapp in chat. Microsoft in Office. Zoom in vid conf. The Telcos in fibre, Netflix and Disney more than Youtube etc etc. Google's enemies are internal ad tech managers, who aren't interested in any other team overshadowing them. If Android is better than iOS it wont be used to take out Apple. It will be used to sell more Ads. Their Adtech guys run the show.

Reply on news.ycombinator.com