Hacker News Reader

Comment by x313 | Hacker News Reader

Pretty cool someone is still doing this. Training in house LLMs was extremely popular in 2023-2024, back when domain-specific LLMs could easily top GPT in their field. In my field alone (tax/HR tech) I remember that Intuit, Workday, Indeed, LinkedIn were all training internal models.

It eventually stopped making sense because of inference costs. Running something internal with 30% GPU utilization is just too cost inefficient compared to using an API. Idk how Reuters will manage to solve this fundamental problem.

View on HN

> Running something internal with 30% GPU utilization is just too cost inefficient compared to using an API. Idk how Reuters will manage to solve this fundamental problem.

That's actually easy, because you can solve it through doing nothing and simply declaring that optimizing for lowest cost is not the main goal.

The fundamental-ness of that problem is entirely man-made and thus can easily be declared void as long as you have the cash to back that up.

Which might be a winning strategy in a world where everyone else is not doing that. Plus that your knowledge stays in-house, etc.

What makes sense also depends on one's business model: TRI charges premium dollars for access to their systems, so there is no need to optimize for cost; trust in the answers is the currency of knowledge workers in today's complex domains.

At the Thomson Reuters family of companies (technically then: Refinitiv Ltd. sold to LSEG), the first foundational model (in the sense of "trained entirely from scratch") was trained already in 2018 (i.e., pre-ChatGPT); it would even have been earlier, but the electricity wires and fuses in the rented 5 Canada Sq, Canary Wharf office had to be replaced first at the time to deal with the current needed to serve the GPUs.

Yep, this is the fundamental issue. It's a 35BA3B model and they probably finetuned it and evalled it in one bursty week on an 8xH100 rental just fine. But long term inference is always going to be easier in an API.

Unfortunately for reuters tho, they dont really have a choice. A lot of their data moat is not necessary live data as in linkedin, and the only way they can keep that moat is by doing this. I guess that justifies any cost.

I don't see why you couldn't see improvements in self-hosted or hosting-as-a-service model throughput? Basically API-style support for a company's internal LLM system. Why not? Or secure infra offered by AWS to self-host your own models that get served to the company just like any other company internal service can be hosted on AWS or similar?

It won't match Anthropic or OpenAI, but it could be economic?

> Or secure infra offered by AWS to self-host your own models

There is no such thing as "secure infra" hosted by someone else.

This might still be fine, depending on your threat model, of course, but if your weights absolutely must never leave the confines of your org, you cannot use any shared hosting provider, because they just offer legal coverage of incidents. But if your moat is your knowledge, legal doesn't matter as much as the knowledge being suddenly unmoated.

It's possible, but it depends a lot on the nature of the data sovereignty guarantees. Today for many companies, the kind of guarantees they have with AWS amounts to basically legal coverage. And that is fine when the company is itself protecting data only due to legal or regulatory reasons. But if the company is protecting it for commercial reasons... then a different model is needed. It could happen, but it has to be worked out.

It’s based on qwen, not fully trained internally. I expect we will way more of this in the future, it’s pretty cheap to fine tune an open weight model for your specialized niche

Reply on news.ycombinator.com