What may be around the corner is running great models on a box at home. The AI lives at home. Your thin client talks to it, maybe runs a smaller AI on device to balance latency and quality. (This would be a natural extension for Apple to go into with its Mac Pro line. $10 to 20k for a home LLM device isn't ridiculous.)
You can also string two 512GB Mac Studios together using MLX to load even larger models - here's 671B 8-bit DeepSeek R1 doing that: https://twitter.com/alexocheema/status/1899735281781411907
Apple silicon is still just a single consumer grade chip. It might be able to run certain end user software well, but it cannot replace a server rack of GPUs.
And of course Nvidia and AMD are coming out with options for massive amounts of high bandwidth GPU memory in desktop form factors.
I like the idea of having basically a local LLM server that your laptop or other devices can connect to. Then your laptop doesn’t have to burn its battery on LLM work and it’s still local.
Oh wow, a maxed out Studio could run a 600B parameter model entirely in memory. Not bad for $12k.
There may be a business in creating the software that links that box to an app on your phone.
The amount of data transferred is tiny and the latency costs are typically going to be dominated by the LLM inference anyway. Not much advantage to doing LAN only except that you don’t need a server.
Though the amount of people who care enough to buy a $3k - $10k server and set this up compared to just using ChatGPT is probably very small.
So I maxed that out, and it’s with Apple’s margins. I suspect you could do it for $5k.
I’d also note that for heavy users of ChatGPT, the difference in energy costs for a home setup and the price for ChatGPT tokens may make this financially compelling for heavy users.
And of course you’d be getting a worse model, since no open source model currently is as good as the best proprietary ones.
Though that gap should narrow as the open models improve and the proprietary ones seemingly plateau.
Any number of AI apps allow you to specify a custom endpoint. As long as your AI server accepts connections to the internet, you're gravy.
You and I could write it. Most folks couldn’t. If AI plateaus, this would be a good hill to have occupied.
The person that is willing to buy that appliance is likely heavily overlapped with the person that is more than capable of pointing one of the dozens of existing apps at a custom domain.
Everyone else will continue to just use app based subscriptions.
Streaming platforms have plateaued (at best), but self hosted media appliances are still vanishingly rare.
Why would AI buck the trend that every other computing service has followed?
Integrated solution. You buy the box. You download the app. It works like the ChatGPT app, except it's tunneling to the box you have at home which has been preconfigured to work with the app. Maybe you have a subscription to keep everything up to date. Maybe you have an open-source model 'store'.
I think there is a market here, solely based on actual data privacy. Not sure how big it is but I can see quite some companies have use for it.
No, but my email provider has a de-facto repository of incredibly sensitive documents. When you put convenience and cost up against privacy, the market has proven over and over that no one gives a shit.
Marketing it though? Not doable.
Apple is pretty much the only company I see attempting this with some kind of AppleTV Pro.
I’m running docker containers with different apps and it works well enough for a lot of my use cases.
I mostly use Qwen Code and GPT OSS 120b right now.
When the next generation of this tech comes through I will probably upgrade despite the price, the value is worth it to me.
A 70b dense model is slower
Qwen coder 30b Q4 runs 40+.
At that point you are almost paying more than the datacenter does for inference hardware.
Of course. You and I don't have their economies of scale.
It’s about the real price of early microcomputers.
Until the frontier stabilizes, this will be the cost of competitive local inference. Not pretending what we can run on a laptop will compete with a data centre.
The same can't be said for local inference. It is always interior in experience and quality.
A reasonable home theater pays for itself over time if you watch a lot of movies. Plus you get to watch shows as well, which the limited theater program doesn't allow.
I can buy over 8 years of the Claude max $100 plan for the price of the 512GB M3 Ultra. And I can't imagine the M3 being great at this after 5 years of hardware advancement.
Not really. I do it because it offers me more control. That's higher quality in my book.
Try building a F1 car at home. I guarantee your unit cost will be several orders of magnitude higher than the companies who make several a year.
That price is ridiculous for most people. Silicon Valley payscales can afford that much, but see how few Apple Vision Pros got sold for far less.