And of course Nvidia and AMD are coming out with options for massive amounts of high bandwidth GPU memory in desktop form factors.
I like the idea of having basically a local LLM server that your laptop or other devices can connect to. Then your laptop doesn’t have to burn its battery on LLM work and it’s still local.
Oh wow, a maxed out Studio could run a 600B parameter model entirely in memory. Not bad for $12k.
There may be a business in creating the software that links that box to an app on your phone.
The amount of data transferred is tiny and the latency costs are typically going to be dominated by the LLM inference anyway. Not much advantage to doing LAN only except that you don’t need a server.
Though the amount of people who care enough to buy a $3k - $10k server and set this up compared to just using ChatGPT is probably very small.
So I maxed that out, and it’s with Apple’s margins. I suspect you could do it for $5k.
I’d also note that for heavy users of ChatGPT, the difference in energy costs for a home setup and the price for ChatGPT tokens may make this financially compelling for heavy users.
And of course you’d be getting a worse model, since no open source model currently is as good as the best proprietary ones.
Though that gap should narrow as the open models improve and the proprietary ones seemingly plateau.
Any number of AI apps allow you to specify a custom endpoint. As long as your AI server accepts connections to the internet, you're gravy.
You and I could write it. Most folks couldn’t. If AI plateaus, this would be a good hill to have occupied.
The person that is willing to buy that appliance is likely heavily overlapped with the person that is more than capable of pointing one of the dozens of existing apps at a custom domain.
Everyone else will continue to just use app based subscriptions.
Streaming platforms have plateaued (at best), but self hosted media appliances are still vanishingly rare.
Why would AI buck the trend that every other computing service has followed?
Integrated solution. You buy the box. You download the app. It works like the ChatGPT app, except it's tunneling to the box you have at home which has been preconfigured to work with the app. Maybe you have a subscription to keep everything up to date. Maybe you have an open-source model 'store'.
I think there is a market here, solely based on actual data privacy. Not sure how big it is but I can see quite some companies have use for it.
No, but my email provider has a de-facto repository of incredibly sensitive documents. When you put convenience and cost up against privacy, the market has proven over and over that no one gives a shit.
Marketing it though? Not doable.
Apple is pretty much the only company I see attempting this with some kind of AppleTV Pro.