Hacker News Reader

Comment by aftbit | Hacker News Reader

>In keeping with our commitment to user responsibility, following the official launch of V4.1 Flash and prior to the release of V4.1 Pro, all requests to the Pro model will be routed to V4.1 Flash and billed at Flash's price

Please don't do this kind of thing. If a user has validated a workflow on V4 Pro, they might not want to suddenly start testing it in production on V4.1 Flash. Instead, keep V4 Pro around but deprecated for a defined period of time, then remove it.

At least as open weights models, it's possible to use something like Together.ai or OpenRouter to run the V4 Pro model as long as other providers keep it up.

View on HN

Usually I would very much agree with you, but those things are not deterministic so if that's an issue for you you're probably not making the right choices.

Nope, strong disagree. The model is one small part of the process harness; behaviors are usually routable with expected propensities. Unexpected model changes avoiding change management messes with monitoring and observability thresholds. Stochastic controls are a real thing when you have your distributions defined; your workflows on a new model will throw that expected prior out the window.

I think it depends on the workload, which put you in the right (in some case, it's dangerous to do that), but for me even for very clean always correct path I always assume a model can, at time, have a vector brain fart (because it keeps happening).

Also in principle it's similar to Anthropic downgrading.

Personally I use the basis that if I don't self host (I include remote host, but that I pay per hosting nor per model or api), it can change behavior without me asking. But they shouldn't, but it doesn't matter that's what they do.

They’re nondeterministic at a fine level, but can be “deterministic” at a more general level: e.g. you might know that one model will always return properly formatted json when asked. That might not be true of the replacement, even if it is in general “better” and cheaper.

Just the risk of such a thing means regression testing every time you update the model, and you want to be able to run that testing on your schedule rather than having it forced on you.

> They’re nondeterministic at a fine level, but can be “deterministic” at a more general level: e.g. you might know that one model will always return properly formatted json when asked. That might not be true of the replacement, even if it is in general “better” and cheaper.

This isn't true. Even Sol messes up JSON formatting for me on occasion.

Do not delude yourself into thinking these things are reliable. They are not.

Is nobody using structured outputs? They use constrained decoding at the generation stage to ensure the probability of tokens that would break the format are set to 0. I kinda figured everyone was doing this at this point.

If you use a large enough volume, you will know this isn't fully reliable. You might get json, and it might not match what the model actually sent because the last layer cut it up to match what you want. At the end, not json, or json but not really matching what the model wanted, it's sort of the same issue: when you use them you HAVE to assume they can have a brain fart. That's fine, just code around it.

Most serious providers are now supporting structured outputs in a reasonable way for all model configs. But for example on ollama structured outputs are still incompatible with tool calling and with reasoning

Friends Don't Let Friends Use Ollama. https://sleepingrobots.com/dreams/stop-using-ollama/

We use structured output and To my knowledge it has never failed (millions of data points). There seem to be two classes of people: those doing productive work with LLMs, and those who only get replies insulting their mothers…

Models are not deterministic, but they do have a flavor. When that flavor changes it can change the nature of output in a way that is undesirable.

Sort of like shooting a rifle - where the bullets hit is (to some order of magnitude, no philosophizing please) non-deterministic, but different very similar rifles will group differently and need to be appropriately adjusted to hit anything.

This is a great way to think of it, idiotsecant.

That flavor profile is known -- it's typical behavioral distribution is somewhat understood (and, often, common failure modes addressed). If JSON breaks about 20% of the time, and that drops for 2% or blows up to 90%, it can drive all sorts of issues (not the least, costs for retries).

Yeah but then it doesn't matter if it's "the model changed to another one" or "the model changed but it's the same name". Point is, you're not hosting it, as far as you know it can change at any moment, build around that idea. Is that great no, is that ideal no, that's why I self host (I include actually renting online the capacity and hosting the model myself on it).

You've added a similar response like 5 times, but I don't see you adding more information.

Yes - model hosts can do nasty things to you aside from changing the underlying model. That doesn't mean it's cool to have them change the model automatically.

Yes, it would be preferable to have complete control over your model serving, and no - not everyone is in a position to do that themselves.

Often via api you can pin to the specific snapshot. Yes the model provider can screw you over - it's physically possible for many vendors to screw you over, that doesn't mean that's not a dick move and that you can't expect/push for better behavior

That's a narrow take. Non-deterministic doesn't mean random; workflows can be reasonably validated and consistent to some known degree.

I work for an education department that serves a chatbot for students, and model changes go through painstaking content safety reviews. I initially assumed it's just a bunch of bureaucratic paranoia. But every other model upgrade has a measurably different adherence to the existing system prompts about not talking to the kids about sex and drugs and mental health issues.

I work in the same field for one of my company, in europe, and if you're not self hosting sorry but your worries are not something I can accept because models are very much not reliable on that front, let alone when you let the host decide HOW to serve a model (ressources allocated, different version of the same model, etc ...).

I'm not being a d**, just saying, the problem you have is something that I have faced EXACTLY, and at least here it's not working until you host in house or remote but on raw hardware. Otherwise it keeps having subtle changes, and you will notice no LLM API providers has guarantees about these.

It's not that you're a d*, it's just that you lack any kind of nuance

There's a whole spectrum between self-hosting open weight models and having a cloud provider swap models from under you

Should you self host a model if want to maximize predictability to the limit? Yes. Does that mean it's wrong for someone hitting a model on API to expect that it won't switch to a completely different model under the hood from one day to another? Probably not.

Sure, but if you're working in a field where that limitation is not "because you like it" but "because you have to" it doesn't matter. If you cannot assume it to be true, then you have to assume it isn't.

If self-hosting is the only solution you can think of to maintain model stability, it's no wonder you missed the point. We already do all those things you mentioned, even with cloud providers. Model versioning exists for exactly this purpose, just like it does with any software dependency. We can specify sonnet:v1.23 and decide if and when to upgrade.

But even if we only asked for sonnet:latest, the last thing we'd expect is opus. Model names should be indicative of breaking changes, and change management doesn't just go in the bin because of non-deterministic tools.

A dice roll is non-deterministic.

Replacing a six-sided die for an eight-sided die also keeps rolls non-deterministic.

That doesn't mean it's fine to just replace the dice mid-game.

Sure, but they are replacing same generation model to another not just switching opus to gpt.

But it’s more like them swapping Opus 5 for Sonnet 5.5 or something. IMO OP’s right to argue against it.

Crossing the street and Russian roulette both have non-deterministic risks of injury. And yet I would be bothered to find out that that on my way to work, I was playing Russian roulette by surprise.

Yes and if you decide to play not using a game rulebook but a website that call it "game A" you can't be shocked if "game A" switched from one to the other, even though the doc said opposite yesterday.

That would absolutely be a dick move on the part of the website and would confuse users.

Also, in this case, the game name is not “Game A” but something like “Deep Seek v4 Pro”, which they have previously chosen to use to describe Deep Seek v4 Pro, not Deep Seek v4.1 Flash.

You should look at why you are protecting your earlier comment rather than changing your mind or understanding why so many people disagree with you.

    > those things are not deterministic

Determinism was an explicit goal of DeepSeek-V4. From their paper: https://arxiv.org/html/2606.19348v1#S3.SS3

    > we implement end-to-end, bitwise batch-invariant, and deterministic kernels with minimal performance overhead

Of course, providers may not implement deterministic inference for various reasons, but it is possible.

Most people are not even referring to CUDA batching nuances.

They think that sampling is an inherent part of Transformers.

Even on this site, it is regurgitated with confidence.

The user cares about the distribution of outputs. That distribution is structurally determined by the distribution of inputs (i.e. prompts), the weights of the model, and (these days) the dynamics of the harness guiding successive generations.

The only way to characterize whether a choice is 'right' is to characterize the output distribution (i.e. evals)! Changing the underlying weights necessarily invalidates whatever characterization may have been done. One may assert that one's harness regularizes outputs back toward the desirable distribution, or one may hope the different weights induce a sufficiently similar output distribution.

But no, one should not be completely agnostic to the choice of weights just because there's some nondeterminism.

These models have specific behavioral characteristics trained into them from reinforcement learning and prompts optimized for one aren't guaranteed to transfer to the new generation. Think if the difference between gpt 5.4 and 5.5 and then 5.5 to 5.6 for example. 5.5 was "better" than 5.4 for struggled more across compaction boundaries and needed much more precise instructions before 5.6 sol recovered some of 5.4's ergonomics. All from the same lab but each model was trained with specific behavioral patterns that were basically product decisions. I would be quite annoyed to find that a model provider was routing a promt optimized for one model to a different one, especially for a dumber/cheaper non frontier model that's not going to be as good at just figuring out what you meant

It’s like being told one day that one of your teammates will be replaced tomorrow by another teammate who’s more capable (has a higher test score), regardless of how long you’ve already worked together and gotten used to each other.

In deterministic scenarios, this kind of switch can sometimes actually be safe, because we’ve used various engineering techniques to converge from non-deterministic behavior to deterministic decisions. On the other hand, in scenarios that are inherently non-deterministic, the impact of such a change is much harder to predict, so we need comprehensive evaluations to assess the extent of its impact.

I think keeping models around for a defined period of time is fine, but fracturing your model offerings like that (keeping around multiple versions of the same model) is very hard to do economically. The economics of the AI ecosystem are dominated by queueing theory constraints that make it extremely cheap to serve predictable traffic loads, and extremely expensive to serve unpredictable loads, and any time you split your offerings like that, you make both less predictable and therefore more expensive to serve both versions.

If I were paying anthropic prices, I'd expect it, but Deepseek is a super scrappy upstart in comparison and intentionally arbitraging on price. I would never expect them to do that.

I imagine they need the compute. Can expand market share with more users for same amount of compute.

But I agree with you. I have a dumb workflow that worked well with v4-flash-0731 and I suspect is directing to a newer model that now breaks it.

4.1 releases tomorrow, right now you're supposed to be served by same old model

I'm not sure that anyone is running production workloads against an API that bills twice as much for a chunk of the day. One of the best things about Deepseek is that you can host it yourself and get a ridiculous multiple of usage for what the same dollar amount would yield from their API

It'd be very expensive to get a setup that can run non-flash well.

If they were still at original price I'd get a couple of DGX Sparks myself to run Flash models at a decent quant/context combo.

> I'm not sure that anyone is running production workloads against an API that bills twice as much for a chunk of the day.

I'm not sure that anyone will mind running production workloads against an API that bills half as much for a chunk of the day.

LLMs add enough nondeterminism to a workflow. Swapping them without the user knowing adds substantially more.

relying on cloud models for anything that gives you ROI is tying a loose noose.

This is true :P.

In this case, Deepseek organization is under a lot of pressure due to compute constraints. It would be better if they just throw a 404 instead of rerouting though so customers are not surprised by subtle changes in behavior.

Yes exactly. Automatic model downgrade seems horrible for a lot of production workloads, even if you are deterministically constraining the behavior of your agents.

I don't really agree, but we shouldn't have to debate it. An Auto option at each level would preclude this kind of decisioning. Pick a discrete model, that's what you get.

Pick Auto (Deepseek v4 Flash Auto vs Deepseek v4.x Flash), and let the vendor decide. I think OpenRouter uses this method.

I wonder if this is a sign of things to come for dirt-cheap model hosting: no servers running old versions, only new versions. Just to keep costs down.

its probably more expensive to run, and I'm thinking 4.1 flash is a smaller more efficient model. You can always host your own. This is what they need to do to stay competitive.

llm is not a deterministic program. the same model won't even return deterministic answer. what's the point keep the model freezed?

if you want deterministic returns, you should set the temperature to 0 to get the best possibility of deterministic.

Strict determinism is a different, but related, issue.

E.g. if I've written a role playing character using a specific model I may want to pin the character to that model until I've been able to test the model being "better" doesn't affect the feel of the character before switching. That doesn't mean I need the character's responses to be completely deterministic, but that doesn't imply I'm fine with the character having a different quality or feel of response just because the new model is out.

It'd be nice if there was a more explicit way to signal in the request "I want what you think is best per dollar for this class of answer" vs "I want this model to answer".

Because we care about the distribution of the outputs and how that impacts a specific use case.

They expect vibe coders to use their models only lol

You absolutely cannot consider an LLM production build number something to be pinned against as a static dependency in a product chain, so it's a non-issue.

It's very easy to tell who is not running production applications using these models based on comments like this

Let me give you the benefit of the doubt, can you expand what you run in production?

In the openai api and many other rapid you can pretty trivially pin not only the model but also the specific snapshot you want to use by using the model id for that snapshot. It's not that big a deal and has been available for years

I'm the founder of a SaaS in the marketing industry with tens of thousands of paying users and we have lots of AI powered workflows where small changes between model versions can make some big differences so whenever we update models we have to thoroughly test them which is why we pin them at specific versions which has been a best practice (and common sense) for the last 3+ years

Yes you can and you should. Providers have SLAs for when models roll off support and this has been the case for APIs long before LLMs. For example https://platform.claude.com/docs/en/about-claude/model-depre... and https://developers.openai.com/api/docs/deprecations

Reply on news.ycombinator.com