Ask HN: Are companies self hosting LLMs?
Background: Below, I lay out exactly why this question comes up.
This is mostly a sanity checking question because the situation came my way and I'm surprised I don't know with much/any confidence.
I can think of a few categories of LLM usage patterns:
[1] FULLY HOSTED APIs: For LLM usage, most companies I talk to and read about are hitting the big hosted APIs elsewhere (from OpenAI, Google, Microsoft). Obviously this has benefits: not managing the core model, not buying hardware that may be expensive/unavailable/soon-obsolete, etc.
[2] MIDDLE GROUND: Then, maybe some people "rent hardware" in a datacenter (whether highly managed or not), that already has the hardware pre-purchased, and running on fast internet, with redundant power, maybe even firewalls, replication and other features. There are many such clouds, and these could be from the Big 3 (Amazon, Google, Microsoft) or in fact from many other smaller players. Again, varying levels of DIY vs. managed-service are available, sometimes through intermediaries.
[3] FULLY SELF HOSTED: And I know about "hobbyist" self-hosters who may have bought up one or more amateur to semi pro grade GPUs or Apple machines to be able to run models basically for fun.
Comment: Both the "FULLY HOSTED APIs" and the "MIDDLE GROUND" are not suitable if a company wants to assure the "highest" levels of trued data protection to their clients, by saying the data does not leave their premises. (I understand there is a required capability level to even be able to effect and maintain such a security posture oneself, of course.) (Yes, once upon a time the "cloud" (aka, someone else) was untrusted, then became trusted, and so on an so forth; I think this is not a static trust relationship.)
What I DO NOT hear a much about is "we need to protect the data, so we will host it ourselves". This seems possible, if you have a lot of money to spend on hardware and people-time, of course.
But I am hearing "we need to protect the data, so we will host/run it (the AI) ourselves" from a startup with no capital yet, no investment, no signed customers. I don't know what they are thinking. Can you sanity check me my above thinking please? Is this something people pull off on a shoestring budget pretty frequently, beyond hobby projects? Or do people pull it off on small/middling budgets all the time?
Any comments are most welcome. If I am missing something major, please let me know. 'Appreciated.