Hacker News Reader

I guess if you are a multibillion dollar company, you can keep that in the bank (stock buybacks or whatever) or invest. So the question isn't really if AI is overhyped or not at the moment. The question is really do you know anything better to do with all that money to make your company future-proof.

The price of these computers is high now, due to supply constraints, so buying now isnt necessarily the best time for this outcome. You can buy some later...

The things these can do is (a) memory bandwidth, Nvidia GPUs have been described as "the worlds most expensive memory controller". Generally very useful anywhere. (b) fast low precision floating point. Historically most of the uses of supercomputers wanted high precision, eg for physics etc but people are now experimenting with using lower precision given availability (c) fast networking. generally useful.

We were seeing a spread of GPU into other applications pre AI, like GPU driven databases, I think this stopped a bit just because of pricing and availability.

Lots of memory bandwidth, low latency high bandwidth connections between nodes, lots of floating point. The MI300 looking a bit like HPC being repurposed for AI is probably a win in terms of using the hardware for other things in the future.

I've had to double check the low precision point qualifier. I expected MI300X to run f32 and f64 at the same speed (and it does) and nvidia to have slower f64 (which it does). It looks like H100 is 250TF at half, 60TF at float, 30TF at double. MI300X is 650TF at half, 80TF at float or double. Very fast F16 indeed.

(sourced from https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300x... and https://www.techpowerup.com/gpu-specs/h100-sxm5-96-gb.c3974, I might have the wrong SKUs for comparison)

Or they invested in GPU’s to mine bitcoin and the ASIC’s are just starting to arrive. At least for inference, GPU’s are not the end game. And who knows if training will continue to need them.

If so there could be another round of investment at a better performance per watt, and specialised hardware will be the key. They’ll all be building custom chips.

The specialised ASIC approach is less likely to be useful for other things though, whereas GPUs are starting to threaten to run arbitrary software well.

Sure. My base case is that LLM’s find uses and that compute gets cheaper and more efficient, which helps make more business cases possible. Investing in current state GPU’s doesn’t seem that useful in that world. Nvidia is priced for a monopoly and I’m not sure it is one.

Imagine if LLMs didn't exist, tech companies hadn't massively invested in compute and money was put elsewhere, perhaps dividends.

In that world, a tech company could arbitrarily invest in multibillion dollar high performance computers for no particular reason. Following the argument, with all that compute on site with nothing to do engineers would would find something interesting and all other companies that didn't invest would be missing out on that revolution.

So by the argument, any tech company could gain a competitive advantage by any non-strategic investment on the assumption that it will always work out. But of course, this only works if you have infinite money and infinite opportunity to speculate. As soon as your resources are constrained, then strategic choice becomes the dominant factor and your CEO has a lot of explaining to do.

What we have here though is the fear of being the company that didn't make that investment while your peers did. That has a completely different risk schedule on it.

If it's a mistake, your excuse is that your competition did it too and you wanted to guard against being left behind. If it works, everyone is happy.

If you're the only company making that speculative investment, it's great if it works and you might be fired by the board if it doesn't.

But the risk/reward is the same regardless of the frame of reference. It's the difference in the perceived risk between gaining $10 and not loosing $10, even when the odds are the same. And we know that while rationally its the same, _loss aversion_ is a cognitive bias that makes people behave differently in these two cases. But FOMO is something we should be mitigating against, since our fear alone doesn't change the actual odds.

But I will take your point that you have more cover for making a bad decision if you know all your competitors are making the same bad decision. But you still missed the opportunity to not make a bad decision and therefore get ahead. The actual risk remains the same.

Funnily enough it kind of isn't. From the perspective of the company itself, it's performance relative to the competition that matters. Not buying the machine when others do is a win if the machines don't work out.

From the perspective of the people running a successful company, it's much more important to make easily defensible reasonable decisions than to make ambitious ones. You protect the capital in preference to maximising returns. Major gains are somewhat rewarded and major failures severely punished, with a comfortable baseline if you maintain the current positioning.

If you're not yet a successful megacorp, all the dials are turned in favour of risk because you need the reward. Lots of incomers doing riskier things seeking to overthrow the incumbent is roughly how we get a turnover of companies and a degree of overall progress.

I think this round is interesting because the incumbents have seen substantial competitive risk which could otherthrow them on an alarmingly short timescale (i.e. while the current leadership are still there), and that has induced otherwise fairly unlikely massive capex spend.

LLMs are a subset of AI/ML. They are not the totality of the sphere, nor are they what is being worked on exclusively.