Hacker News Reader

"We've realized for some time that not only are language model technologies going to dominate our future"

Close, but not quite. What we realize right now is that AI has pumped, pumped, pumped a whole lot of stock valuations sky high, and the first faint glimmers of the market asking that some actual money be delivered in proportion to the sky high valuations are starting to appear. (Note carefully the "in proportion to"; it is not enough to make a bit of cash. These valuations are enormous.) The stocks so pumped need results more-or-less right now, since propagating anything they have right now out into the market to actually gather revenue is going to take some time.

I expect a couple of other Hail Mary plays like this to show up too.

And again, let me underline, emphasize, and highlight, this isn't about whether AI can make "any" money. It's about whether it can justify those staggeringly high stock valuations in, say, the next year or so. This is a fairly uphill battle. Even if you are a True Believer than AI will be an economic revolution in literally the next 12 months or so, it's hard not to look at the market and come to the conclusion that right now every AI player is being valuated as the inevitable victor and the inevitable beneficiary of 100% (or more) of the AI gain, e.g., nVidia is being valuated as if there will never, ever be any competition for AI hardware despite all the incentives, each AI user valuated as if they are the inevitable winner in their respective domains, etc. They can't all win simultaneously like that, it's just impossible.

(On the plus side, it's really just stock valuations. I don't expect an AI winter this time, the tools are much more valuable than they were last time. I mean, I would still very firmly say that what we have right now is grotesquely over hyped, but there's still quite a bit of "there" there. The previous AI hype cycles didn't have anywhere near as much "there" there.)

NVIDIA has fundamental hardware and software advantages that will take years and years to catch up on. The general purpose part of GPGPU is no joke, just ask the people trying to build competing hardware and an alternative to the CUDA stack. But...

NVIDIA is massively investing in AI companies with the expectation (explicit or otherwise) that they use the money on NVIDIA hardware. And they are pouring in ridiculous amounts of money. I think this is severely impacting the ecosystem, creating overvalued and overfunded AI startups that lack viable businesses and driving essentially fake revenue to NVIDIA that is unsustainable. The money is basically free for NVIDIA to spend, maybe even net positive between revenue numbers and progress towards monopolizing the AI stack. Time will tell if the bubble will pop or if one of the plays will hit and turn a fake flywheel into a real one, but I definitely agree that the stock valuations are too high.

It's not that clear Apple/local-AI is necessarily going to directly compete that much with large models running in datacentres (outside of some relatively limited use cases) and there don't seem to be many signs that Nvidia has any serious plans about entering the consumer market and compromise their insane margins.

Spot on. The current wave of GenAI has created a bunch of new markets, but those markets are, at least currently, the unsavory types: LLM-based virtual girlfriends, AI hentai games running off stable diffusion, SEO and social media spam with ChatGPT, etc.

The best we've got in the "regular" enterprise world is excellent parsing of unstructured documents. Which is, to be sure, hugely valuable and will make a big difference in a lot of industries. But isn't nearly enough to justify the insane evaluations and investment companies are making into AI right now.

Two years into the AI boom, and it's starting to look closer to blockchain and the metaverse than the iPhone.

The costs are also largely being ignored at the moment. Both the obvious marginal costs (electricity, hardware for compute, dev time) and the hidden costs (flood of spam and misinformation).

I was really surprised to read the other day that Dell isn't making money on AI servers. I played around with the server build tool on their website and was surprised to see that a H100 adds ~$50K to a server! How the heck are these companies spending $100K per server going to get back their ROI? Are all these costs going to get written off as R&D? This really seems like the dot-com days but worse somehow.

https://www.cnbc.com/2024/05/31/dell-shares-fall-ai-servers-...

Also, the environmental cost. It's hidden, but won't be forever.

Would be awesome to have nuclear power stations dedicated to powering these data centers for inferencing.

I believe Amazon just purchased a nuclear power plant in Pennsylvania for this.

Agree on the Apple strengths, disagree that MS is scared. PCs just aren't where they make most of their money anymore.

Enterprise is extremely lucrative for them, and their headstart in the AI space may be just what they need to close the gap with AWS.

What would you do if you had several billion devices that you had root access to, used directly by humans and isn't making you any money in sales these days?

The 2024 answer is to use it to capture data to train AI.

Microsoft could lose 9/10 of their market share (which they won't) and still have millions of devices harvesting training data for them.

If GDPR is at all questionable, Microsoft is not able to push this to EU countries. The risk for a business to get in GDPR trouble is to high.

Azure is bad. Google is the one that's going to leverage AI to catch up with AWS, GCP is actually good and they've already partnered with Huggingface.

Features aside, I simply don't trust Google to keep GCP running in the long term. Their propensity to kill projects (even ones which are successful), plus their gratuitous price hikes (aka Google Maps) has made me Goggle negative.

Sure, they might run GCP forever. Sure they may offer some sort of customer service at some point. Sure they may decide to keep pricing at sane levels. The key work in all that is "might" - and that requires my _trust_.

And for better, or worse, my trust simply isn't there.

For us Google wasn't even on the table, due to what you mentioned.

It was AWS or Azure. AWS didn't have a data center in our country and our customers are required by law to keep their data in the country, so easy win for Azure.

I 100% feel you on the google trust. I think it'll end up being central to a lot of their other efforts so I'm expecting it to stick around, but hybrid cloud solutions are just a smart bet anyhow.

Why is azure ‘bad’? Is it because you dislike Microsoft?

In my experience Azure is really good and getting better. Even if AWS is more popular, Azure is gaining traction, is more intuitive and easier to use, and in my experience: cheaper

Had to do multi cloud with Azure, worked on network infrastructure, vnet, peering, firewall, vpn gateway, etc.

Documentation was almost worthless, leaving a lot of room for interpretation. Encountered and filed more bugs trying to get it up and running than I did for years of using Aws. Support was mediocre. Non existent ipv6 support. This was a few years ago and in comparison to Aws.

they've had long running security issues and performance edge cases that make relying on them a bit of a time bomb.

You realize that Azure is much more than compute right?

Of course, and why is azure ‘bad’?

It's not, it great if you have a large org.

> We've realized for some time that... language model technologies going to dominate our future

Citation Needed.

> Apple has a huge advantage because they get to pay nothing for a superior compute infrastructure that comes with a better privacy and latency story.

Right now, AI is a pay-$B-to-play and you need the massive datacenter compute for training and skilled researchers to develop. Apple silicon is neat, but let's not pretend like they have any credible story to tell about AI, and their recent deal with OpenAI suggests that they've been forced to punt.

> As soon as they release something, it's going to be good, and it's going to be free, because it runs fast on consumer hardware and consumer electricity.

Apple recently released their first public LLM, OpenELM. The largest released model is OpenELM-3B. The Instruct version of this model scores a 24.8 on the MMLU benchmark, which is a pick-one-of-four multiple choice test.

Apple's biggest and best model somehow manages to get less than 25% on a 4 option multiple choice exam. For reference, MS released a similar size model Phi3-mini-4k which scored a 69 on that same exam.

Yes, Apple has some nice local hardware to run models, but they don't have the models. They don't have the models because they don't have the people and they don't have the compute.

> language model technologies going to dominate our future

I know many people who are _far from convinced_ of that. But that's not the relevant point: Microsoft, Google et all clearly are.

And watching them collectively, desperately stampede after the mirage, at great cost, is certainly something.

> Apple - their silence, but also their hardware capabilities to run LLMs > billion dollar datacenter investment that still provides a worse experience

It's not at all clear to me that those very small local models would be competing directly with large LLMs run in datacentres for the same use cases. If that's going to be the case MS/etc. would also require significantly less compute power that's is currently expected, so Nvidia would be the real losers in that case (unless they shift to consumer HW).

Also does Apple have such a huge advantage compared to the newer AMD/Intel/Google/Qualcomm(?) chips with NPUs? Yes Apple started investing sooner but all the phones/tablets and the overwhelming majority of laptops are crippled by extremely low memory capacity (in the context of running LLMs). So are they really that much ahead? e.g. what real advantages does the latest iPhone have compared to a Pixel 8 Pro (sure it's still a niche device, with a seemingly inferior an chip (but more memory) but based on what we know Google seems to be ahead in AI development and will make their solution available on all high end Android devices).

> As soon as they release something, it's

I doubt that's a given, their software has long ceased to be highly particularly exceptional and on the hardware side the only actually innovative product they have released for years has been a flop with a very unclear future.

The realized value of LLMs is going to be output from a data to AI pipeline. From raw data to actions or insights. This is Microsoft's play to control the entire process. They are attempting to abstract away all the "tooling" needed to manage and process data. It falls down because there is no discretion left to the consumer. Objectively, the computer is already processing all the data you are interacting with. Subjectively it is assumed its ephemeral. Computer forensics proves this wrong. So I think to refine my opening statement, the realized value of LLMs is going to be the output from a "curated" data to AI pipeline. Which Microsoft is not providing with this solution.

It seems like a somewhat far fetched explanation, but then I can’t think of a better one. They come up with this farcical application that nobody wants, to justify the hardware requirement, so that at least the hardware will be out there for when Apple shows them what the actual application is.

Apple doesn't have as much of an advantage as it appears. Their Neural Engine are fast and getting better, but modern machine learning models need another resource: fast RAM. Unfortunately for Apple, their whole product stack is segmented based on RAM. There are millions and millions of Apple devices with lightning-fast processors and 8GB or even 4GB RAM. These won't be able to do anything like Copilot+, unless they make calls to a datacenter like Microsoft is doing.

Their existing local models are hit-or-miss already. FaceTime's audio transcription is laughable. Whether this is due to memory requirements, I don't know, but it doesn't bode well for further models.

Suddenly, distributed private inference, compute, storage and bandwidth mediated by Holochain at global scale will become … valuable.

Just wait until TSMC asks for 30% of Apple's revenue or they'll kick Apple out of their Fabstore.

Fabrication is where the tech advantage lies, not hardware design IP. Anyone nowadays can program VHDL, see open source hardware. The challenge is actually fabricating it.

> As soon as they release something, it's going to be good

Huh?

How is that a given?

It is absolutely not true, they already released a model (OpenELM) and it compares extremely poorly versus similar size models from the likes of MS.

This chart is correct and to scale: https://i.imgur.com/5mjlesU.png

I think Microsoft software teams are already working behind their legal teams. Their business model is more about licensing than making products. Likely they’ll force some kind of regulatory situation that’s advantage to them. For example no AI capable OSs for government work, or something like that.

Apple is in terrible shape on software, AI is driving this home

the only tier-one, network-effects app they control is iMessage

search? someone else

maps? someone else

video? someone else

audio? someone else

and now they will go to someone else for AI

what did they do in the meantime? VR that bellyflopped

Not sure why this is relevant. Apple's forte was always the hardware (and lately the greater focus on privacy). And you could easily say the same about Microsoft:

search? someone else

maps? someone else

video? someone else

audio? someone else

Arguably Apple is at least better positioned than MS for some of these because people actually use Apple Maps and Apple Music. When was the last time someone whipped out Bing Maps?

You forgot iOS. App APIs are also network effect things.