Why do we see no companies offering to deploy local LLMs? | Hacker News Reader

Why do we see no companies offering to deploy local LLMs?

20 pointsoogwayy14 comments

It seems an obvious idea but we cannot find anyone offering b2b local LLMs setup to corporations that doesn't want to use external ones & keep their data in-house.. ie airlines, governments, attorneys, accountants..

View on HN

Shameless plug, but super relevant maybe-

I’m a product marketing guy for Nutanix. We have a new LLM deployment solution that can run on any CNCF Kubernetes where you can deploy LLMs from a validated list from Hugging Face or NVIDIA NIM, or upload your own.

You can then create RBAC-controlled endpoints to connect your GenAI apps up to all with a point and click interface.

Check it out with basic walkthrough on the product page: https://www.nutanix.com/products/nutanix-enterprise-ai

I see large companies leveraging their large technical teams and external consulancies and technology providers to build their own e.g. banks on IBM. And I think for small companies it's impossible to beat the value for money a Saas type solution provides and the big APIs like ChatGPT and Claude. For the companies inbetween I'd imageine the risk to reward of picking a stack might be too high right now given how fast everything is moving. There is probably room for someone to do it next year but I don't get the sense that the market is ready for it just yet.

My understanding, and I could be wrong here so I’m happy to be corrected, is that AI companies like OpenAI are perfectly capable of operating an instance of their product that doesn’t send data back to the source.

They discuss this on the Enterprise page:

Protect your data with enterprise-grade privacy, security, and deployment tools You own and control your business data in ChatGPT Enterprise. We do not train on your business data or conversations, and our models don’t learn from your usage. ChatGPT Enterprise is also SOC 2 compliant and all conversations are encrypted in transit and at rest. Our new admin console lets you manage team members easily and offers domain verification, SSO, and usage insights, allowing for large-scale deployment into enterprise. See our privacy page and our Trust Portal (opens in a new window) for more details on how we treat your data.

https://openai.com/index/introducing-chatgpt-enterprise/

So it’s very likely that most companies find this reassuring enough and therefore don’t necessarily care too much about running models locally. Anyone that needs security greater than this probably has the resources to develop AI capabilities in-house.

  > ie airlines, governments, attorneys, accountants..

Because it's easier and more productive to use private cloud LLMs, like Azure OpenAI and ChatGPT enterprise.

IMHO, local LLMs still can't compete on quality and speed.

https://azure.microsoft.com/en-us/products/ai-services/opena...

https://openai.com/index/introducing-chatgpt-enterprise/

It comes with fewer risks. You have somebody to blame if anything goes wrong.

More productive is debatable. A service that is constantly updated might be less preferable to one that only changes if you want to, for many use cases.

We do that for ourselves since some customers require it. Happy to help on a contract basis. https://groundedai.company/services/

We haven't put a lot of marketing into the service, maybe we should do that.

When I last looked it seemed like cloud costs were much higher.

Thanks guys. I agree on market timing, probably next year. I doubt a lot of corps will trust those privacy policies by openAI, especially outside the USA

I think this is a the value prop for AWS Bedrock (no affiliation)

AFAIK You get a managed instance where the model data doesn’t get sent to the model provider, but you pay PAYG rates.