Hacker News Reader

Comment by porridgeraisin | Hacker News Reader

Yep, this is the fundamental issue. It's a 35BA3B model and they probably finetuned it and evalled it in one bursty week on an 8xH100 rental just fine. But long term inference is always going to be easier in an API.

Unfortunately for reuters tho, they dont really have a choice. A lot of their data moat is not necessary live data as in linkedin, and the only way they can keep that moat is by doing this. I guess that justifies any cost.

View on HN

I don't see why you couldn't see improvements in self-hosted or hosting-as-a-service model throughput? Basically API-style support for a company's internal LLM system. Why not? Or secure infra offered by AWS to self-host your own models that get served to the company just like any other company internal service can be hosted on AWS or similar?

It won't match Anthropic or OpenAI, but it could be economic?

> Or secure infra offered by AWS to self-host your own models

There is no such thing as "secure infra" hosted by someone else.

This might still be fine, depending on your threat model, of course, but if your weights absolutely must never leave the confines of your org, you cannot use any shared hosting provider, because they just offer legal coverage of incidents. But if your moat is your knowledge, legal doesn't matter as much as the knowledge being suddenly unmoated.

It's possible, but it depends a lot on the nature of the data sovereignty guarantees. Today for many companies, the kind of guarantees they have with AWS amounts to basically legal coverage. And that is fine when the company is itself protecting data only due to legal or regulatory reasons. But if the company is protecting it for commercial reasons... then a different model is needed. It could happen, but it has to be worked out.

Reply on news.ycombinator.com