Hacker News Reader

Comment by nico | Hacker News Reader

> The problem: thinking models think too much

This is partly the appeal of Jev et al; having a quick model for simple tasks, that doesn’t require that much thinking

It’s amazing all the workflows that models like that can unlock. And yes, classifiers and other ML models have been around for a while for these types of tasks, but Jev has made it easy and cheap to play and experiment. This in turn, is incentivizing people to try them for a bunch of stuff, unlocking creativity and producing a lot of new cool (and eventually potentially very useful) applications

View on HN

What are the useful applications of Jev so far? Not to sound dismissive, I just haven’t seen what people are using it for yet.

Lots of use cases! I've personally used it for the following:

1. Evals (once you have your rubric defined and tuned using a reasoning model, jev can be great for running periodic evals especially those that run daily.

2. e-commerce catalog classification 3. quick search using anything as context and query mapping to a pre-defined set.

At least for 1, evils, you’d want to use a good old reasoning model to get the best eval results.

Here's a third-party (not Jev) showcase of things people built, which helped me kind of get the appeal. https://bentossell.com/jev/ (not mine).

LLMs can be too creative and often too verbose. Sometimes there is a right answer and a way to get there with the understanding of language, but despite using structured outputs, the model insists on inventing variations not in the schema or coming up with something completely different. A model like Jev that can not do those things, and can give the same output every time with given the same input, and be able to measure probabilities has many use cases.

Why not using a cheap LLM with thinking completely disabled ? I don't think it will be much more expensive than jev.

I’ve tested this with some local LLMs and their accuracy is in general better than Jev/Laya, but they are super slow in comparison as well

For example, a typical/stock LLM can’t really play Doom in real time, but a Jev-like model can. Just because of latency

Of course, if you want the best Doom player, there are way better and faster adhoc models

LLM inference has two very different regimes of work: prefill & decode. You can think of the former roughly as processing a pre-specified prompt, and the latter as sequential processing (auto-regressive token generation) eg. "chain of thought". The latter is very important for LLMs and cannot be ignored; it deeply influences infra design, even necessitates copious amounts of high-bandwidth memory. Jev-like models can ignore the latter and therefore optimize much better for the former, consequently operating at both better cost and latency.

Reply on news.ycombinator.com