Hacker News Reader

Comment by me551ah | Hacker News Reader

This makes me wonder if AI companies even have a MOAT in the first place.

All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days.

And that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified. Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars. Or all of them are worth a lot less. One of those statements is true.

Also this makes me wonder if the next iteration of LLMs would be based on fine-tuning, where LLMs actually learn from your past behaviour so that it would grant some amount of stickiness to the product. OpenAI used to offer fine tuning runs for GPT-3.5, but they don't seem to do that anymore.

View on HN

> All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history

A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.

Which is why big labs have been working hard on making their harness not be stateless any longer:

https://earendil.com/posts/session-portability/

“Just take the session thread to another provider” might not be feasible anymore soon-ish.

While that particular API might be nice, and people and companies should probably push back against the obfuscation, in the end it doesn't really matter. When I hand off between different models I already have the first model prepare a markdown file for the second rather than just importing the entire original thread wholesale, because that's expensive anyhow, and also rather unfocused. They can't get their models to stop generating that sort of checkpoint because that's a fundamental operation necessary for all the harnesses to work anyhow.

The fundamental technology of LLMs and arguably AI in general strongly cuts against that sort of lockin. Handoff is a fundamental capability. There's no option to encrypt the docs or write it in some dialect only one model understands because humans need to understand it to, which stops that whole line dead in its tracks for at least the forseeable future. An AI can already today pick up such pieces, how much more easily will they do it tomorrow?

If they want to lock me in, they're going to need to provide a feature that I need so badly I can't switch and nobody else has. It is hard to see what that would be, other than being a generally better model.

Soooo… when can we expect an encrypted handoff.md to fully prevent session portability, then?

(Only half-/s)

And this is why you should not use the model provider's harness. It is to tight of a coupling of services that's only business purpose is lock-in.

Knower

Ding ding ding

Which is why folks should be jumping to pi / oh my pi as soon as possible

This is also why anthropic no finger lets you use the flat rate sub in non-claude-code harnesses

(They do but you are charged at api rates)

As a heavy pi user - this doesn’t help, for codex. The thinking traces are encrypted and you cannot move that session over to eg your local ds4. It will lack those precious thinking traces that you absolutely need to continue a session productively.

They can be deterministic. We did this at Groq, if you sent a request with exactly the same input token, seed and temperature value you would get precisely the same result every time.

This is harder to do on other architectures that themselves aren't fully deterministic though.

Was the temperature 0? Cause unless I don't understand it right, any non-zero temperature implies probabilistic next token prediction.

You did mention, seed, which I haven't seen available anywhere else (in my limited experience) and it can explain a reproducible inference result, but I feel like this matches the "letter of the law" meaning of deterministic rather than the "spirit of the law".

If you send the seed along, you can have a non-zero temperature: when it's time to select tokens, a PRNG will be seeded and used for the selection.

Using a PRNG and a seed, you can have paeudo-randomness and determinism. The pseudo-randomness is the key piece, it’s random to the extent that it cannot be distinguished from true randomness in polynomial time classically.

Perhaps they just used the exact same random seed everywhere?

Do you mean that you get the exact same result (as word for word) because from Groq's documentation, it says that determinism is best effort, and also it's not guaranteed across model versions : https://console.groq.com/docs/prompting?_sm_vck=tMW2S7k17245...

My understanding is that it's best effort because even if you control the seed/temperature, the model version and hardware specs you still have floating point math that will introduce rounding differences

an article explaining this in detail : https://news.ycombinator.com/item?id=45200925

> harder to do on other architectures that themselves aren't fully deterministic though

The TLDR here is that Groq LPUs are completely deterministic hardware, including for floating point.

Disclaimer: Formerly of Groq, now of NVIDIA, still working on LPUs.

>LLM responses are certainly not idempotent, as they are not even deterministic.

Isn't that more due to an optimization and not how the LLM itself runs?

Like a MoE LLM run on a single input should give the same output each time. But this is inefficient, as any given token is hitting 1 (or maybe 2 or 3) experts at a time, meaning all the other experts are doing absolutely nothing. So you upgrade it to take in multiple requests. But then any given expert can become a bottleneck, so when too many requests need a given expert, some of them are routed to a second or third best expert instead. Within the context of any single request, this looks like non-determinism, but it is still deterministic when considering the full batch.

For everyday users and everyday use cases, that is enough to treat it as non-deterministic (the harness might also send in unique data like current time which means one can never have the exact same request twice), but when talking about LLMs more theoretically, I think we need to consider they can still be ran deterministically even if that isn't as optimized.

Similar with temperature. 0 means deterministic, but anything higher with a seeded value is deterministic. If anything, temperature is us purposefully adding non-determinism to agents because they were too deterministic.

It is much more subtle than your specific example, which is strictly speaking a bug, though ofc it has been used during pretraining for efficiency purposes. Sglang and miles have been working towards full determinism in open source codebases, so the LLMs can help explain the subtleties encountered in actual projects if you point them to these repos.

In the simplest possible case, a distributed addition of floating point numbers is not deterministic if you don't specify the order of the addition operations. If floats are added in a first-come first-serve fashion (the simplest implementation of a reduction operation) you already lose determinism. These could be activations from multiple experts, but it could also be adding chunks of a matrix operation that uses multiple CUDA cores.

If you manage to achieve determinism in temperature zero, it is possible to extend it to determinism at higher temperatures, because at that point you only have to keep the pseudo-random-number generator state in sync across parallel instances and this problem has been solved.

I appreciate the replies on the determinism point and I’ve learned some new things here. In any case I probably should not have tagged that on, as my main point was to share that the sort of property that parent is talking about (whether true for all LLMs/providers/harnesses or not) is statelessness, not idempotency.

LLMs are, in theory, deterministic. Sampling is not intrinsic to LLMs.

Greedy decoding a single batch in most libraries will give you mostly deterministic outputs. Higher batch sizes can increase variance.

But all of this is down to CUDA and/or kernel implementation issues.

It's a more or less solved problem (e.g. DeepSeek has batch-invariant kernels, vLLM also has reproducible outputs etc), with an obvious caveat that you need to lock your hardware setup, model snapshot, and inference stack if you want any useful reproducibility across longer time periods. It's up to specific providers to actually implement it to any extent, of course commercial providers are incentivized to do the opposite.

According to [1] there is - unfortunately - increasingly more state in LLM sessions.

1: https://earendil.com/posts/session-portability

Google figured this out with their paper from 2023, We have no moat and neither does OpenAI. The moat now is the harness and being able to recursively self improve from RLHF, a great example is how Grok used to be pretty bad but since SpaceX bought Cursor, they used that data to train Grok 4.5 which is now very competent at coding and even exceeds frontier models in certain benchmarks.

https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...

Moat is not the harness. Harness itself is temporary until the models get better and slowly the code in harness will go down.

Note that the biggest GPU providers in the world are the hyper scalers and even they couldn’t allocate more if you pay for it. Because the rich companies and well funded ones are gobbling them up to the point where if tomorrow a 5T model that smokes every other model in the world is released you just can’t afford inference.

Agree. Harness can not be a moat. There are many open harnesses and they are at least on par with the providers ones. It looks like Anthropic/OpenAI's approach to vendor lock-in is not so much the inference or the harness it is functional integration across the individuals and teams in a company. I don't think this will be a moat either, but I think it's all they have outside of compute.

I meant that companies like Anthropic are locking in users with proprietary formats in their harness where it's hard to leave.

Surely the moat is the training data... with the data you can explore new architectures much easier and get step changes in performance.

The training data, at least up to now, is very abundant and basically every lab has the same data from scraping the Internet. RLHF data is what's now valuable.

Arguably the majority of codebases are not available on the public internet

Is it RLHF if there is no human feedback?

And yet investment is continuing. What are they counting on?

One possibility is that they're counting on another bailout package like the banks and others got in 2008.

The bets are absolutely huge and have been spread throughout the financial system via securitization. If the number does not continue to go up, a lot of people are going to find themselves backing their numbers with money that simply isn't there anymore. Therefore, the number will continue to go up and we (meaning American taxpayers) will back whatever it takes for that to happen.

Now, were the bets stupid to begin with? Of course. Who wants a machine that potentially unemploys major percentages of the population? No one. But the bets were made regardless, and then more bets were made on the hypothetical returns of the first round of bets, and so on and so on.

Bubbles aren't always rational.

> that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified.

They aren't, not even if we forget about the capable Chinese models.

I suspect Anthropic will implode soon when employees are unable to get the cash-out that they expected. Having so much compensation locked up in company stock is risky on a good day.

They have 2 moats.

The first is the compute. OpenAI and Anthropic secured huge amounts of compute, Google, Meta and xAI have their own huge datacenters. Now anyone can rent some cloud machines and start serving Kimi K3, but it's going to be impossible to get to a similar scale as the big 5 above. And inference has economies of scale: the more people you serve in parallel, the more efficient you are.

The second is the data. By now (and maybe even by one year ago), all the data on the internet has been used for training. You need new data. The big AI companies sit on top of trillions or quadrillions of tokens that they have generated over the years. They can use that to train new models. That data is gold, and the proof is that SpaceX was happy to pay $60B to acquire Cursor.

If you want to overtake the frontier labs, you have 2 options: use their models to generate synthetic data, and provide lots of (cheap, maybe below cost) inference to generate your own new data. The frontier labs know about the first, and I'm sure they try to limit how much others milk their models. As for the second, that's the "honest" way to compete, but it's not easy.

compute is not a moat, it's a rapidly depreciating physical asset. buying up all the shovels in a gold rush does not give you a moat, it gives you a slight advantage for the time being. someone else will just start making shovels. and the data is clearly available, hence the number of open-weight models.

Isn't a literal "moat" about temporary deterrence? I can imagine makeshift bridges could permanently make the moat useless.

well the boards have been laid across the two shores, and the enemies are now climbing the walls, i guess is what i'm saying. if they had a moat, it's been breached

your post helped me realize a change Meta is pursuing on Instagram that is to give more weight to captions and long text posts so they can have more data to training that would usually go to websites/Google. Even AI slop is good for this.

I think Anthropic/OpenAI do have a moat in the (western) enterprise market. Chinese hosted models are a no-go, and my experience with enterprise IT departments is that they will not self-host. So far signing up with a known product (e.g. Claude) seems to be the way they will go and this is the moat that the AI companies have. Alternatively there is Copilot, but that for now seems to mostly be backed by Anthropic/OpenAI models[1].

Will this continue? The field is moving too fast to tell. Kimi, Qwen, Deepseek also produce very capable models but that doesn't automatically translate into trillion dollar valuations.

However, trillion dollar valuations on Anthropic and OpenAI, such new companies, never publicly traded and such huge valuations decided just by investors. This is just asking for trouble.

1. https://docs.github.com/en/copilot/reference/ai-models/suppo...

> This makes me wonder if AI companies even have a MOAT in the first place.

They don't. The moat will mainly be the tooling around AI, not the AI itself. You don't hear any company claiming their moat is the Internet.

Burdensome regulatory compliance is a moat.

These companies have AI and enough money to lobby the Pope. They can afford to reanimate members of congress and push some tactical legislation through.

But all the money in the world cannot move government too quickly.

Other moats exist too. OS or browser can undermine performance and availability of alternatives.

> This makes me wonder if AI companies even have a MOAT in the first place.

Generally speaking they do, at least from my experience when switching from one model to the other - their performance decreases, and they often do large refactors outside of the requested scope as they try to bring the code closer to 'their' style.

Which makes sense imo - they'v been trained to iterate over the code they wrote, and not code that was modified by someone else in the interim.

With their current API approach they're essentially a commodity. They need to start moving parts of the harness behind the API, otherwise they'll remain a commodity.

Recursive self-improvement changes the parameters a bit, especially for the market-leaders, and it's the one thing that makes me wonder if they'll be able to extend their lead faster than the smaller labs can keep up, but it's an option available to everyone.

> They need to start moving parts of the harness behind the API

This isn't without it's challenges however.

1. This will increase costs drastically, since they would need to a run a sandbox per use to ensure data isolation.

2. Increased latency, and this directly limits how much of the harness can be moved to the cloud before the users notice sluggishness

Oh, I'm sure. If it wasn't a bit of a tricky needle to thread we'd have seen them start to do it already.

3. Users can still move to better open harnesses

The actual moat is the same as in web services - data and user base.

Why is Google a monopoly? Do they have so advanced software that no one can outperform? I doubt it. What they have is a giant user base that generate loads of real-time data, which make Google services more accurate.

So how AI company can build a moat? Exactly the same way: by making a giant user base produce loads of real time data. Just imagine a service that will generate answers not only based on data they were trained on, but on all data from all user conversations. Imagine being at a concert, looking for a certain type of beer and instantly receiving an answer from an AI assistant about that only because some other guy in a crowd looking for exactly the same thing said to his agent "ah, here they are!".

It is not happening just yet because of making it secure and private is not yet solved, but it's just a matter of time I think.

> Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars

It’s more like a bunch of people are placing different bets. Only a few bets are going to generate a return, possibly only one, but the profit on that one bet will make it all worthwhile. That’s the theory, anyway.

There's no mandate that says any of those bets are going to generate a return

This isn’t technically true. Most model providers don’t send the thinking tokens anymore, so if you switch from one provider to another, you will be missing large parts of the conversation.

I think you're right and I think it's why Google have taken their pedal off the metal for model releases to focus on integrations and tools. And why Microsoft have backed off from the OpenAI partnership to do the same. Anthropic and OpenAI are going to massively struggle to maintain their pace and reach profitability just selling commodity tokens.

Fine tunes are a possibility but I think it offers very little uplift for the vast majority of uses beyond just stuffing enough context.

China has the moat that they are cheap/free/open. The US corps have the moat that the other option is Chinese models. At least for some time.

From my experience these open source models are nowhere near the performance offered by Fable/Opus/GPT-5.6. Whenever I tried Qwen, Kimi, Deepseek, the results were much worse and it just took much more time to get something usable. When you consider that, the frontier offerings are still much cheaper.

That might be true right now, but how long until you have to move the goalposts? In my experience with DeepSeek and Kimi, they're as capable as the frontier was four months ago, which already solves a big chunk of the coding tasks that I'm interested in.

What provider did you use ? Synthetic's Kimi is a beast

They have a moat; they don't have $1 trillion valuations.

Which anyone who hasn't been sitting in the SV echo chamber could have told you years ago after applying even the smallest bit of thought.

The major labs don’t allow assistant prefill, so you have to “summarize”

fine tuning runs of models the size of gpt 5.6 are absurdly expensive. I'd guess at least $100k in cloud gpu time for a single run, and you have to do a few iterations to get things right

Mamba/SSMs could change this picture.

The moat is the US government.

Reply on news.ycombinator.com