Also worth noting - TPUv4 uses a 6-way 3D torus interconnect vs the 3-way "multi ToR" NVLINK topology; the total bisection bandwidth of the TPUv4 pod is over 1PB/s!
Can't wait to see what TPUv5 looks like. As you say, it's probably already chugging away with v6 on track to tape out in a year.
That said, I think NVidia has nailed bringing the ecosystem along, and I think making the whole setup look more like "one huge GPU" could simplify a lot of ML programming.
I am actually disappointed I haven't seen more of that style in CPU programming. Where's my 20,000 core 100TB RAM VM instance?
You could simulate this with a bunch of regular machines and a networked hypervisor.
You could do some kind of smart caching so that processes rarely need to wait to access RAM stored on a remote machine.
Combined that with a big lock eliding/speculation scheme (ie. When a process reads memory that might have been written by a remote CPU, you continue as if it hadn't, and if you later find out that data was written then you rollback). These rollbacks 'undo' all work done in however many microseconds it takes for data to travel from one side of the machine cluster to another.
Reads of RAM that aren't cached yet on the local node can also be speculated - you just assume that RAM contained null bytes and continue execution, rolling back and replaying when the actual data arrives.
So if you can make sure that processes are contending for locks and writing conflicting data less often than once per system-roundtrip-latency, then you should get a high performance system.
But upon further thought, a lot of things such a system would need are actually rather inefficient to implement in software (ie. rollbackable RAM), yet quite cheap in hardware (for example rollbackable RAM can be implemented with regular RAM plus either a buffer of 'overwritten data' or a write queue)
The biggest issue it seems is bandwidth and humans' patience for a response...
So 1/100th of the CPU and 40% of the RAM. (I suspect the RAM comparison is reasonable - I'm not sure about how to compare the CPU's).
The nvidia device uses a fabric with 900 GBps switched fabric between any of the 256 nodes in the system. The TPUv4 3d torus network is basically a ring network of 56 GBps connections creating separate rings. From a raw perspective, the nvidia solution is the overwhelming winner. There is absolutely no contest.
Machines with terabytes of RAM do exist and get used - working well on such setups is a goal of modern JVM GCs for instance - but making a single machine that large which acts like a single machine isn't easy, nor especially desirable. One machine is a unified failure domains outside of mainframe-land, so if you had a 20k core machine with 100TB of RAM you could never reboot it to apply OS updates and it'd die all the time from failed parts.
Even once you get beyond that most software stacks use locking and stop scaling beyond a few hundred cores at best and that's assuming very heavily optimized stacks. AI workloads are easier because they're designed from scratch to be inherently parallel without lots of little locks and custom data structures all over the place like a regular computer has.
Disk storage is one of the places where you can parallelize and scale out relatively easily and you do see datacenter sized disks there.
edit: actually now i can't find the claim, maybe i misremember what the papers said.
Software for TPU is still in its early stages. CUDA is well established. You can test on a gaming GPU that you can find (locally!) in many markets. XLA is meant to solve this, but first impressions matter and my first impression was that it has not yet "solved" this issue.
TPU is only available via Google Cloud - as far as I know they don't have NVIDIA's widespread distribution to various HPC/supercomputer systems. This also has implications on scaling up more than a few pods, as they will need to be colocated with speedy interconnect (which is provided by the various existing HPC systems that use NVIDIA's chips).
Finally, I think many people are discovering that the supposed benefits of TPU are marginal at best in the face of the types of natural scaling issues that both GPU's and TPU's suffer from when scaling out to e.g. hundreds of pods.
I'm certain that someone with more experience than I could give a better answer though - and again, all speculation. I refuse to use TPU because Google Cloud's system for getting access to said TPU's was horrible for me when I tried it. I believe John Carmack has a nice tweet thread specifying the same issues I ran into.
In general, Google has a habit of developing tech for other Googlers first, and as such winds up ignoring a lot of real-world scenarios faced by researchers/practitioners. NVIDIA on the other hand has been working directly with a ton of institutions and businesses ever since the inception of CUDA.
That their TPU's have seen any adoption at all is mostly due to their research program which granted very cheap access to TPU's to tons of people.
(this is not meant to be reassuring).
Coupled with a magazine or a show presenting new product categories for those interested, customers will eventually visit a physical or online shop and check out the goods. And then word of mouth will do the rest.
Aggressive advertising will mostly just help you get ahead of your competitors and perhaps speed up the adoption rate at the cost of increased volatility of the market and to the detriment of people's mental health.
We would be better off regulating aggressive ads away.
That's a hell of a load-bearing "just" you managed to insert there. Getting ahead of your competitors in market share can be the difference between having a company succeed or fail.
Or: all advertisers of all brands with a same or similar product must collaborate. Only voluntary input counts as collaboration; if a brand simply doesn't care about presentation of itself in the advertisement, they have trivially collaborated. Easiest way to implement this is giving every owner of all relevant brands a right to veto every entire final advertisement product (this right could also be surrendered, for all or some possible vetoed advertisements, in exchange for something in a contract).
Ignoring flaws of this proposition itself, what could be society's reasons for rejecting it? Does society perhaps want havers of more money to gain further advantage over havers of less money?
Maybe 50 years ago that would have worked. Today, not so much. Go to Amazon and look, well, just about anything. What is BEHENO, what is DINGEE, what is Etoolia, what is Romedia, what are the over 300 different 6/7 letter companies that show up when I search up some random product.
Unfortunately your consideration causes its own parasite effect of countless companies forming up to feed of the big advertisers budget.
And, you're also crating a regulatory nightmare. Say I put up an add for XXYZXX company, and it includes ZZXYZZ and YYXZYY information (I mean totally random picks), and I just happen to have a stake in those companies too. Now you're going to have to track hundreds of thousands of these entities to ensure no fraud is occurring, and in most cases the fines for this kind of behavior are well under the cost of doing business.
Everything you've said so far just creates bigger messes and solves nothing.
As a result, spending ad money on Google is ridiculously expensive, but companies accept this because there is no alternative hoping to "build long lasting relations" with the people who make them pay upwards of 1 dollar per click
Also, if your established it probably a good idea not to let new competitors get a foot hold in the market with an easy google win.
It's also pretty effective for local businesses because not a lot of local businesses are tech savvy enough use it effectively.
FWIW, the cheapest (quality) clicks I've seen, at least in the B2B space, is closer to $3/click, and it can quickly balloon to upwards of $10/click especially on company brand names where competitors are bidding on another company's brand name.
Knowing this, I cringe every time I'm screensharing with someone and they search "[B2B Company] login" to login to a tool they use every day. Each login = $2-$10
It's not uncommon for companies to spend $100k+/year JUST bidding on their own company name.
I just don't get where all the marketing money is coming from. Bootstrapping is clearly not an option these days
That bag of chips did not cost $4 to make, not even a little close.
Conversely, in Communist systems they could never get this right. Factories were just told to produce 5 or 10% more than last year, didn’t matter if the product quality was worse or if people didn’t want it.
Marketing is already difficult to tell apart from other company communications, product documentation, etc. What about a company blog showing how to use their products? Is that marketing or product documentation?
Tax ad companies, and spend that money on education. The better ad companies are doing, the more we spend on education, the fewer gullible marks we produce, the worse ad companies will do.
Informative ads make the market more efficient. Persuasive ads actively make the market less efficient.
Most ads in the US in 2023 seem to be persuasive.
Perhaps the ad industry would become more useful (and smaller) if we managed to effectively regulate it to significantly reduce the persuasive bits.
I think that most people would support this if you explained it right - from the free-market perspective, this would give you a better market.
So we can have internet anywhere. To click on ads.
Think about all the stuff ChatGPT and GPT-4 can do with even minimal prompting. Even when they hallucinate, the text is still ostensibly coherent and natural sounding. Now imagine a similarly powerful model, but its input is a ton of metadata about your behavior and its output is ads.
Now consider that adtech has had substantially more funding for substantially longer than research into LLMs, so ad serving models are probably way more powerful and optimized than even GPT-4.
It's freaky to think about indeed.
More competition would be fantastic. Better pricing at scale would be fantastic. But there is absolutely no doubt that nvidia is far ahead of Google right now. Tesla made some believably pushing claims about their own efforts with their own hardware, so who knows maybe they're the real challenger.
I don’t think they will exist in 10 years.
gmail is a freebie! the core product is how they index your messages to create an anonymous profile that they will then offer on reverse bid to advertisers when you do a search or visits an AdWords site.
Businesses fail. google will likely still exist, but alphabet, I don’t see a future for - just a gradual withering followed by a collapse and disintegration into myriad properties in a fire sale. They are brittle, overburdened by unity of disparity, culturally adrift, and they aren’t taking risks any more. Inertia will keep it all going for a while, but not forever.
Sure, I may be wrong, but I do put my money where my mouth is, and I am right more often than not.
I see a lot of hate for alphabet on HN. It seems very emotional. I think people feel personally betrayed by thier bad behaviours because they were 'supposed to be better'.
The thing is, there are a lot of companies you can hate. Exon, mcdonalds, blackrock, even Microsoft, there are people who are very mad at these companies.
That's not an argument that the company is doomed. If you are really putting your money where your mouth is (what shorting google?) Then I hope you have a better reasoning as to why they will fail not just eventually but this year.
None of the companies you list are likely to collapse soon, as they remain focussed on their various missions, and have a unity of purpose. Out of all of them, I think Microsoft is the most likely to fail, as they are likely to be blindsided when the user-focussed desktop OS era ends. Their diversification efforts have been a mixed bag, and without windows, they are far, far less significant.
What I do look at is sentiment analysis - what other people feel and think about businesses, as that drives the market.
No, I don’t short, as just buying equities which are beginning significant growth is just as effective and doesn’t drive demise - I held goog for nearly 20 years, and sold off late ‘21, as I think they’ve peaked, and anything from here on is speculative froth.
You’ll note I keep saying “I think”, rather than making statements of fact - because this is purely what I think - I am not a Sybil.
You seem to have missed this:
>> They are brittle, overburdened by unity of disparity, culturally adrift, and they aren’t taking risks any more.
My perception is that Google split into a number of focussed business units when they became Alphabet, with the Google component being execution focussed and the more speculative stuff spun out into other group companies like deepmind, waymo, etc. That's why the Google unit stopped doing nice incubator projects that we were all excited about.
From what I've seen, this cash cow execution business unit has been fairly effective - in particular they've done a good job of entering the cloud market space producing a differentiated product that is penetrating their target customers. They have not been able to compete with Microsofts excelent and deeply embedded IT sales capability, so they've done well to go after people with big problems that other vendors more civillian offerings are not so great for. They are currently the first choice platform for AI training for instance.
I'd contrast this to Facebook who seem to be trying to become a deep tech VR hardware vendor in the same business unit as their cash cow entertainemet and advertising business which has confused investors and probably distracted their focus.
We can see that Google has innovated. For instance, a lot of Tela's stock price is based on the idea that they are going to run autonomous taxis, and instead of owning cars we will just hail a Tesla when we need one. Telsa does not run autonomous taxis, but you can ride a Google Waymo taxi today in Pheonix, and they are running autonmous trucks which is a big industry Tesla aren't even attempting yet. They are doing a lot in medicine and medical devices. This seems a lot more diversified and innovative than other companies - it's just not as visible to the HN community as an RSS reader or some other internet thing we care about.
We can also say that... on the AI thing, I think it's very early days. Microsoft have a shakey looking deal with the first mover, but Alphabet and Facebook have the advantage of actually using AI extensively in their real buisnesses and may be able to deliver product market fit better. Time will tell.
On the stocks front, I agree with your overall thesis - I think it's harder for these conglomerates to grow than a new company just because they are already giants in their niche and even adding a new niche generates less growth in percentage terms than for a smaller company starting from a lower number. I just wouldn't actually bet against google as much as I would some of the others.
That might have been a reasonable assessment back in Balmer’s era. But what you saying has already happened years ago…
They have mostly reinvented themselves since then. Enterprise/office isn’t going anywhere. Xbox if fine too. And there is a lot of growth in their cloud/etc. business.
IMHO out of Google, Amazon & Facebook, Microsoft seems to be the least dysfunctional and and general best positioned one to be successful in the future.
We can both give the money to a mutually trusted third party now.
Their search products have actually gotten worse with AI. Google Images running just off basic image recognition (as in is this the same image) and the context of where they found it was far superior at identifying what an image is than ML Google Image.
The OG version could identify a frame from a movie and provide higher res versions. The ML version goes “errr looks like a woman on a street, here are random photos of unrelated women on unrelated streets with maybe a similar color scheme” close to useless why would anyone want that. Yandex Image search blows it out of the water simply by being Google Image Search from a decade ago
The overall theme is that product is no longer the focus, but rather navel-gazing - that’s to say, their internal world no longer aligns with the external world, and that is a fundamentally dangerous place for a business.
Do you just have more email now?
E.g. what’s possible with embedding where the query terms are matched with similar messages that mean the same thing.