Hacker News Reader

Comment by acidburnNSA | Hacker News Reader

* "Self-hosted: Runs entirely on your infrastructure. No data leaves your network."

* "Bring Your Own LLM: Anthropic, OpenAI, Gemini, or open-weight models via vLLM."

With so many newbies wanting these kinds of services it might be worth adjusting the first bullet to say: "No data leaves your network, at least as long as you don't use any Anthropic, OpenAI, or Gemini models via the network of course"

View on HN

That's a good point, it might make sense to clarify that for individuals who want to self-host. I'll make the change, thanks!

Most organizations are going to be self hosting on aws, gcp or azure... So as long as you use their inference services as your LLM then you can keep it all within the private network

Exactly, enterprise customers almost always use private model endpoints on their cloud provider for any serious deployments. Data stays within the customer's VPC, data security and privacy is guaranteed by the cloud providers.

Even self-hosting on AWS, GCP, or Azure isn't local enough for certain application, such as people doing export-controlled work where any sysadmin or person with physical access to the server/data is required to be a US Person (or equivalent in other countries). This is the niche that the govcloud solutions are aimed at serving. But some people just want to build big actually-private, actually self-hosted systems and do their own physical and network security.

AWS Bedrock seems to say the inference code is only scanned for CASM and no one trains on your data.

Are all people with physical access to the servers or network access to the hosts guaranteed to be US persons? Are all physical and network accesses logged for audits? That's the kind of thing govcloud promises that export control auditors want to see.

I felt like "Confidential Compute" tech could solve this issue once and for all but I'm not so sure after seeing some of the attacks people can do with physical access.

Another option of course is to not use cloud at all and have your own rack in a locked room with a good security system and/or armed US person guards.

Reply on news.ycombinator.com