How the rest of the industry can respond is such a mystery. And will it be lone competitors, or will a new PC era be able to start, with an ecosystem of capabilities?
Also worth noting - TPUv4 uses a 6-way 3D torus interconnect vs the 3-way "multi ToR" NVLINK topology; the total bisection bandwidth of the TPUv4 pod is over 1PB/s!
Can't wait to see what TPUv5 looks like. As you say, it's probably already chugging away with v6 on track to tape out in a year.
That said, I think NVidia has nailed bringing the ecosystem along, and I think making the whole setup look more like "one huge GPU" could simplify a lot of ML programming.
I am actually disappointed I haven't seen more of that style in CPU programming. Where's my 20,000 core 100TB RAM VM instance?
The nvidia device uses a fabric with 900 GBps switched fabric between any of the 256 nodes in the system. The TPUv4 3d torus network is basically a ring network of 56 GBps connections creating separate rings. From a raw perspective, the nvidia solution is the overwhelming winner. There is absolutely no contest.
You could simulate this with a bunch of regular machines and a networked hypervisor.
You could do some kind of smart caching so that processes rarely need to wait to access RAM stored on a remote machine.
Combined that with a big lock eliding/speculation scheme (ie. When a process reads memory that might have been written by a remote CPU, you continue as if it hadn't, and if you later find out that data was written then you rollback). These rollbacks 'undo' all work done in however many microseconds it takes for data to travel from one side of the machine cluster to another.
Reads of RAM that aren't cached yet on the local node can also be speculated - you just assume that RAM contained null bytes and continue execution, rolling back and replaying when the actual data arrives.
So if you can make sure that processes are contending for locks and writing conflicting data less often than once per system-roundtrip-latency, then you should get a high performance system.
But upon further thought, a lot of things such a system would need are actually rather inefficient to implement in software (ie. rollbackable RAM), yet quite cheap in hardware (for example rollbackable RAM can be implemented with regular RAM plus either a buffer of 'overwritten data' or a write queue)
The biggest issue it seems is bandwidth and humans' patience for a response...
Machines with terabytes of RAM do exist and get used - working well on such setups is a goal of modern JVM GCs for instance - but making a single machine that large which acts like a single machine isn't easy, nor especially desirable. One machine is a unified failure domains outside of mainframe-land, so if you had a 20k core machine with 100TB of RAM you could never reboot it to apply OS updates and it'd die all the time from failed parts.
Even once you get beyond that most software stacks use locking and stop scaling beyond a few hundred cores at best and that's assuming very heavily optimized stacks. AI workloads are easier because they're designed from scratch to be inherently parallel without lots of little locks and custom data structures all over the place like a regular computer has.
Disk storage is one of the places where you can parallelize and scale out relatively easily and you do see datacenter sized disks there.
So 1/100th of the CPU and 40% of the RAM. (I suspect the RAM comparison is reasonable - I'm not sure about how to compare the CPU's).
Software for TPU is still in its early stages. CUDA is well established. You can test on a gaming GPU that you can find (locally!) in many markets. XLA is meant to solve this, but first impressions matter and my first impression was that it has not yet "solved" this issue.
TPU is only available via Google Cloud - as far as I know they don't have NVIDIA's widespread distribution to various HPC/supercomputer systems. This also has implications on scaling up more than a few pods, as they will need to be colocated with speedy interconnect (which is provided by the various existing HPC systems that use NVIDIA's chips).
Finally, I think many people are discovering that the supposed benefits of TPU are marginal at best in the face of the types of natural scaling issues that both GPU's and TPU's suffer from when scaling out to e.g. hundreds of pods.
I'm certain that someone with more experience than I could give a better answer though - and again, all speculation. I refuse to use TPU because Google Cloud's system for getting access to said TPU's was horrible for me when I tried it. I believe John Carmack has a nice tweet thread specifying the same issues I ran into.
In general, Google has a habit of developing tech for other Googlers first, and as such winds up ignoring a lot of real-world scenarios faced by researchers/practitioners. NVIDIA on the other hand has been working directly with a ton of institutions and businesses ever since the inception of CUDA.
That their TPU's have seen any adoption at all is mostly due to their research program which granted very cheap access to TPU's to tons of people.
(this is not meant to be reassuring).
As a result, spending ad money on Google is ridiculously expensive, but companies accept this because there is no alternative hoping to "build long lasting relations" with the people who make them pay upwards of 1 dollar per click
Also, if your established it probably a good idea not to let new competitors get a foot hold in the market with an easy google win.
It's also pretty effective for local businesses because not a lot of local businesses are tech savvy enough use it effectively.
FWIW, the cheapest (quality) clicks I've seen, at least in the B2B space, is closer to $3/click, and it can quickly balloon to upwards of $10/click especially on company brand names where competitors are bidding on another company's brand name.
Knowing this, I cringe every time I'm screensharing with someone and they search "[B2B Company] login" to login to a tool they use every day. Each login = $2-$10
It's not uncommon for companies to spend $100k+/year JUST bidding on their own company name.
I just don't get where all the marketing money is coming from. Bootstrapping is clearly not an option these days
That bag of chips did not cost $4 to make, not even a little close.
Coupled with a magazine or a show presenting new product categories for those interested, customers will eventually visit a physical or online shop and check out the goods. And then word of mouth will do the rest.
Aggressive advertising will mostly just help you get ahead of your competitors and perhaps speed up the adoption rate at the cost of increased volatility of the market and to the detriment of people's mental health.
We would be better off regulating aggressive ads away.
That's a hell of a load-bearing "just" you managed to insert there. Getting ahead of your competitors in market share can be the difference between having a company succeed or fail.
Or: all advertisers of all brands with a same or similar product must collaborate. Only voluntary input counts as collaboration; if a brand simply doesn't care about presentation of itself in the advertisement, they have trivially collaborated. Easiest way to implement this is giving every owner of all relevant brands a right to veto every entire final advertisement product (this right could also be surrendered, for all or some possible vetoed advertisements, in exchange for something in a contract).
Ignoring flaws of this proposition itself, what could be society's reasons for rejecting it? Does society perhaps want havers of more money to gain further advantage over havers of less money?
Maybe 50 years ago that would have worked. Today, not so much. Go to Amazon and look, well, just about anything. What is BEHENO, what is DINGEE, what is Etoolia, what is Romedia, what are the over 300 different 6/7 letter companies that show up when I search up some random product.
Unfortunately your consideration causes its own parasite effect of countless companies forming up to feed of the big advertisers budget.
And, you're also crating a regulatory nightmare. Say I put up an add for XXYZXX company, and it includes ZZXYZZ and YYXZYY information (I mean totally random picks), and I just happen to have a stake in those companies too. Now you're going to have to track hundreds of thousands of these entities to ensure no fraud is occurring, and in most cases the fines for this kind of behavior are well under the cost of doing business.
Everything you've said so far just creates bigger messes and solves nothing.
Marketing is already difficult to tell apart from other company communications, product documentation, etc. What about a company blog showing how to use their products? Is that marketing or product documentation?
Tax ad companies, and spend that money on education. The better ad companies are doing, the more we spend on education, the fewer gullible marks we produce, the worse ad companies will do.
Conversely, in Communist systems they could never get this right. Factories were just told to produce 5 or 10% more than last year, didn’t matter if the product quality was worse or if people didn’t want it.
Think about all the stuff ChatGPT and GPT-4 can do with even minimal prompting. Even when they hallucinate, the text is still ostensibly coherent and natural sounding. Now imagine a similarly powerful model, but its input is a ton of metadata about your behavior and its output is ads.
Now consider that adtech has had substantially more funding for substantially longer than research into LLMs, so ad serving models are probably way more powerful and optimized than even GPT-4.
It's freaky to think about indeed.
So we can have internet anywhere. To click on ads.
Informative ads make the market more efficient. Persuasive ads actively make the market less efficient.
Most ads in the US in 2023 seem to be persuasive.
Perhaps the ad industry would become more useful (and smaller) if we managed to effectively regulate it to significantly reduce the persuasive bits.
I think that most people would support this if you explained it right - from the free-market perspective, this would give you a better market.
More competition would be fantastic. Better pricing at scale would be fantastic. But there is absolutely no doubt that nvidia is far ahead of Google right now. Tesla made some believably pushing claims about their own efforts with their own hardware, so who knows maybe they're the real challenger.
edit: actually now i can't find the claim, maybe i misremember what the papers said.
I don’t think they will exist in 10 years.
Their search products have actually gotten worse with AI. Google Images running just off basic image recognition (as in is this the same image) and the context of where they found it was far superior at identifying what an image is than ML Google Image.
The OG version could identify a frame from a movie and provide higher res versions. The ML version goes “errr looks like a woman on a street, here are random photos of unrelated women on unrelated streets with maybe a similar color scheme” close to useless why would anyone want that. Yandex Image search blows it out of the water simply by being Google Image Search from a decade ago
The overall theme is that product is no longer the focus, but rather navel-gazing - that’s to say, their internal world no longer aligns with the external world, and that is a fundamentally dangerous place for a business.
Businesses fail. google will likely still exist, but alphabet, I don’t see a future for - just a gradual withering followed by a collapse and disintegration into myriad properties in a fire sale. They are brittle, overburdened by unity of disparity, culturally adrift, and they aren’t taking risks any more. Inertia will keep it all going for a while, but not forever.
Sure, I may be wrong, but I do put my money where my mouth is, and I am right more often than not.
I see a lot of hate for alphabet on HN. It seems very emotional. I think people feel personally betrayed by thier bad behaviours because they were 'supposed to be better'.
The thing is, there are a lot of companies you can hate. Exon, mcdonalds, blackrock, even Microsoft, there are people who are very mad at these companies.
That's not an argument that the company is doomed. If you are really putting your money where your mouth is (what shorting google?) Then I hope you have a better reasoning as to why they will fail not just eventually but this year.
None of the companies you list are likely to collapse soon, as they remain focussed on their various missions, and have a unity of purpose. Out of all of them, I think Microsoft is the most likely to fail, as they are likely to be blindsided when the user-focussed desktop OS era ends. Their diversification efforts have been a mixed bag, and without windows, they are far, far less significant.
What I do look at is sentiment analysis - what other people feel and think about businesses, as that drives the market.
No, I don’t short, as just buying equities which are beginning significant growth is just as effective and doesn’t drive demise - I held goog for nearly 20 years, and sold off late ‘21, as I think they’ve peaked, and anything from here on is speculative froth.
You’ll note I keep saying “I think”, rather than making statements of fact - because this is purely what I think - I am not a Sybil.
You seem to have missed this:
>> They are brittle, overburdened by unity of disparity, culturally adrift, and they aren’t taking risks any more.
That might have been a reasonable assessment back in Balmer’s era. But what you saying has already happened years ago…
They have mostly reinvented themselves since then. Enterprise/office isn’t going anywhere. Xbox if fine too. And there is a lot of growth in their cloud/etc. business.
IMHO out of Google, Amazon & Facebook, Microsoft seems to be the least dysfunctional and and general best positioned one to be successful in the future.
My perception is that Google split into a number of focussed business units when they became Alphabet, with the Google component being execution focussed and the more speculative stuff spun out into other group companies like deepmind, waymo, etc. That's why the Google unit stopped doing nice incubator projects that we were all excited about.
From what I've seen, this cash cow execution business unit has been fairly effective - in particular they've done a good job of entering the cloud market space producing a differentiated product that is penetrating their target customers. They have not been able to compete with Microsofts excelent and deeply embedded IT sales capability, so they've done well to go after people with big problems that other vendors more civillian offerings are not so great for. They are currently the first choice platform for AI training for instance.
I'd contrast this to Facebook who seem to be trying to become a deep tech VR hardware vendor in the same business unit as their cash cow entertainemet and advertising business which has confused investors and probably distracted their focus.
We can see that Google has innovated. For instance, a lot of Tela's stock price is based on the idea that they are going to run autonomous taxis, and instead of owning cars we will just hail a Tesla when we need one. Telsa does not run autonomous taxis, but you can ride a Google Waymo taxi today in Pheonix, and they are running autonmous trucks which is a big industry Tesla aren't even attempting yet. They are doing a lot in medicine and medical devices. This seems a lot more diversified and innovative than other companies - it's just not as visible to the HN community as an RSS reader or some other internet thing we care about.
We can also say that... on the AI thing, I think it's very early days. Microsoft have a shakey looking deal with the first mover, but Alphabet and Facebook have the advantage of actually using AI extensively in their real buisnesses and may be able to deliver product market fit better. Time will tell.
On the stocks front, I agree with your overall thesis - I think it's harder for these conglomerates to grow than a new company just because they are already giants in their niche and even adding a new niche generates less growth in percentage terms than for a smaller company starting from a lower number. I just wouldn't actually bet against google as much as I would some of the others.
We can both give the money to a mutually trusted third party now.
gmail is a freebie! the core product is how they index your messages to create an anonymous profile that they will then offer on reverse bid to advertisers when you do a search or visits an AdWords site.
Do you just have more email now?
E.g. what’s possible with embedding where the query terms are matched with similar messages that mean the same thing.
If the competitors (mainly AMD, Intel and to some extent ARM) will keep seeing growing volumes and insane margins they will be attracted to bring to invest and take part of that market.
Till now gaming GPU market did not bring to AMD the necessary margins to really push them to bring a better competition to Nvidia. Even 10/15 years ago when ATI was way ahead of Nvidia technologically for 2/3 years (the HD 4000 and HD 5000 generations vs the Nvidia flops of the 9000, 200 and 400 series) Nvidia was posting billions of profits and ATI posted a whole...19 millions of profits across 3 years.
But today's GPU market thanks to it's non-gaming sales is much bigger to ignore (which is why Intel entered it as well) and those players will likely react.
You don't need to have the best premier product, you need to have your products good and priced well enough that they will be chosen over the competitor's.
In hindsight the 40GB of SRAM feels kind of quaint, but nevertheless their very fat nodes let them get away with more than Nvidia could with A100s, as you can see in the slides.
CS2 is a little old now. I bet an update is just around the corner.
And that's because it is very high skill work, otherwise, it would have been 100 engineers paid tens of thousands.
So I guess my question is - what use case is there for a huge truck that goes 200mph and take 4 trips, when you could just buy 16 regular trucks, and move your apartment in the same amount of time at half the cost.
'The street finds it's own uses for things' is the well known Gibson adage, I and typically it's a comment aimed low. But our entire era of amazing computing began with the Gang of Nine enabling lowness in a degree such that it quickly became the highest tech, the best. Sure you can still buy a mainframe & they have amazing feats but it's not where the value is, but and the value is where it is because possibility was unchained, I unleashed from corporate dominion, and spread wide. I think we can find amazing new futures with CXL & mad bandwidth connectivity.
The other approach, which you do when models themselves are massive, is model parallelism. You split it into multiple parts that run on different nodes.
In both cases, you need to distribute weight updates through the network although the traffic patterns can be different.
To maximize the performance in both scenarios, systems designers optimize for all-reduce and bisection bandwidth.
There are also other tricks, for example the TPUv4 ICI network is optically switched, and it is configured when a workload starts to maximize bandwidth for the requested topology ("twisting the torus" in the published paper).