Hacker News Reader

Comment by lkois | Hacker News Reader

That's a narrow take. Non-deterministic doesn't mean random; workflows can be reasonably validated and consistent to some known degree.

I work for an education department that serves a chatbot for students, and model changes go through painstaking content safety reviews. I initially assumed it's just a bunch of bureaucratic paranoia. But every other model upgrade has a measurably different adherence to the existing system prompts about not talking to the kids about sex and drugs and mental health issues.

View on HN

I work in the same field for one of my company, in europe, and if you're not self hosting sorry but your worries are not something I can accept because models are very much not reliable on that front, let alone when you let the host decide HOW to serve a model (ressources allocated, different version of the same model, etc ...).

I'm not being a d**, just saying, the problem you have is something that I have faced EXACTLY, and at least here it's not working until you host in house or remote but on raw hardware. Otherwise it keeps having subtle changes, and you will notice no LLM API providers has guarantees about these.

It's not that you're a d*, it's just that you lack any kind of nuance

There's a whole spectrum between self-hosting open weight models and having a cloud provider swap models from under you

Should you self host a model if want to maximize predictability to the limit? Yes. Does that mean it's wrong for someone hitting a model on API to expect that it won't switch to a completely different model under the hood from one day to another? Probably not.

Sure, but if you're working in a field where that limitation is not "because you like it" but "because you have to" it doesn't matter. If you cannot assume it to be true, then you have to assume it isn't.

If self-hosting is the only solution you can think of to maintain model stability, it's no wonder you missed the point. We already do all those things you mentioned, even with cloud providers. Model versioning exists for exactly this purpose, just like it does with any software dependency. We can specify sonnet:v1.23 and decide if and when to upgrade.

But even if we only asked for sonnet:latest, the last thing we'd expect is opus. Model names should be indicative of breaking changes, and change management doesn't just go in the bin because of non-deterministic tools.

Reply on news.ycombinator.com