Hacker News Reader

Comment by afavour | Hacker News Reader

I hope price increases will mean we see serious moves towards open source and/or on device models. Feels like that's where we should all be headed but the current subsidy is a distraction.

View on HN

There is no subsidy in interference. Serving existing models is a highly profitable business. The proofs are many independent companies serving models for profit.

What is expensive is model development and training, but again, that's nothing inherent to technology. It's just that Anthropic and other western vendors made a business decision to use cheap investors money to brute force a new model development and market grab instead of investing in cost optimizations for model training and inference like Chinese model developers and providers.

As I understand it, those cost optimizations relied on someone else brute forcing new model development. If everyone was trying to do it cheaper, we wouldn't have the models we have today.

Many of the chinese developers are just drafting off of Anthropic, OpenAI, and Gemini though. Distillation is leveraging those models to achieve their leaps in training. It's not obvious to me that Anthropic and do the same. Someone has to build the advanced model to draft off of.

Of course Anthropic is doing the same. Just ask Claude in Chinese to introduce himself and tell you what he does and how it can help you. There's a good chance that it will tell you its name is DeepSeek ;)

A hallucination is not an indicator of distillation...

I would self-host if it didn't cost so much for the hardware. But right now that's cost prohibitive for most individuals and small or medium businesses.

Funny how that happened, isn’t it?

How what happened? How everyone stopped having cutting edge graphics cards in all their devices? Most people have never been in a position to be able to do local inference on cutting edge models.

Maybe that's the reason why AI firms tried to monopolize ram, chips like AMD AI MAX 395 with AMD not being beholden or useful to the AI firms coupled with 128gb+ ram would probably have been capable of making home-hosted open models competitive enough that the AI firms valuations would've sunk like stone before their IPO's.

The question is if the MoE advances and compression (TurboQuant) makes common computers fast enough to run local models that are "Good Enough".

I agree but with the current hardware pricing it is a big problem

Just be patient for a few years. Even with the end of Moore's Law, compute hardware pricing always declines over the long run.

Yeah I am waiting but:

If anything breaks on my pc, it's going to be really rough.

My kids grow fast,a few years is a lot for them too.

And i have a bunch of games I want to play with my wife that require 2 computers sadly

On device is still meh.

Realistically Apple needs to put a few teams on it, and have built in support for running LLMs on OSX.

I assume if you integrate this on an OS level you might be able to pull off some additional tricks. So far most of the tools for running LLMs locally have been driven by volunteers or very small startups.

Microsoft could also do this, but they're too busy trying to upsell everyone. Microsoft wants to sell you on paying for each token.

Apple has the edge here since they fully control the hardware and can baseline their implementation. It feels like Microsoft's task is much harder because there's a much broader range of hardware and capabilities that they need to support.

Microsoft sells hardware. Surface laptops.

Microsoft has spent a lot of time telling people to buy AI ready PCs.

I just don't think it's in their business model to offer on device LLM. They want you to subscribe to Copilot. Which encapsulates the entire enshitication of Windows. Microsoft demands more money. Subscribe to gamepass, subscribe to OneDrive!

I'm aware of the Surface line, but 1) they don't own the hardware stack nearly to the same degree as Apple does, 2) it makes up only a small slice of the ecosystem and is a more premium product as far as PCs go. Windows ecosystem has far more variance in terms of CPU + GPU capabilities.

I can see Microsoft pushing the subscription angle, but I do think there is a place for on-device capabilities since those will always be more responsive for small tasks.

I agree with your points. While I'm just thinking out loud, I can imagine either anthro or open AI releasing hardware for on-prem solutions in the near future.

I believe a few small companies have already specialized in this. But imagine a situation where maybe you pay a small subscription and Open AI allows you to download non open source models guaranteed to work to a degree.

It's going to be an interesting decade.

MS is more like assembling the surfaces, it's not the same as having a full vertical integration like Apple is trying to achieve with their contracts with Intel, in-house modem, etc.

Reply on news.ycombinator.com