Hacker News Reader

Comment by bicsi | Hacker News Reader

Of course it works, Jev is nothing but an API breakthrough

Jev is rumored to be a 30B model, and it's input price is MUCH cheaper than similarly sized models. The maker is also heavily focused on having a profitable product, so it's unlikely to be subsidizing the cost, especially since they say they have more demand than what they can serve.

You can get Gemma 4 26B A4B at the exact same token input price of $0.042/M. GPT-5 nano is not much more expensive at $0.05.

https://openrouter.ai/google/gemma-4-26b-a4b-it

Exactly, imo it’s not even that cheap if you look into perspective and consider the fact that providers could subsidize the cost of cached input tokens to virtually zero if they would allow for a more flexible API (e.g. tree of message blocks instead of chain). Most of the cost is the infrastructure around keeping KV caches, estimating their lifetimes, etc. When mist people just want to run one context block with multiple subsequent variants of a second block in parallel. I still stand by my statement.

That's an interesting point, if you send a batch with a shared prefix you basically only end up paying for the sequence length difference effectively.

There is still some minor memory bandwidth issue on outputting more tokens, but the truth is that if you process e.g. 16 messages at once you wont end up being much slower than Jev even though you have to perform several autoregressive passes.

"you wont end up being much slower than Jev" -- I would even go as far as saying that "you will end up being Jev".

With this cost, does it perform the same quality and speed as Jev?

I'm quite interested in this; my current understanding is though that Jev is great when scored with response quality and latency metrics.

Seems to be slightly higher quality and substantially faster, although this compares remote API vs local deployment.

The prevalent idea of Jev's superiority in price, speed and accuracy seems to come from TypeSafe's marketing and their, I'd say even bad faith, benchmarking. In independent benchmarks the relative numbers tend to be very different.

https://github.com/Mushroom-Systems/lichen

They can't overturn the economics of attention by restricting themselves to a single token output.

Sure they are no longer memory bandwidth bound thanks to that but someone could add a similar projector to a conventional model, train with a Jev style dataset and call it a day.

Whatever they are doing on inputs must either mean they intentionally chose a Mamba successor or they suffer from the same compute costs as everyone else.

Jev claims 70-500 ms latency, including for the first request. This requires some clever engineering at least, which will take a little to duplicate.

Maybe first request is unbatched, to have fast prefill, and the subsequent ones are batched.

They also don't restrict your prompt. You can have a dumb one, where you put the variable data at the front, and the details on how to process it at the back, thus you bust the user-part of the KV cache every request.

Reply on news.ycombinator.com