Hacker News Reader

Comment by stusmall | Hacker News Reader

This is so interesting to watch. For a couple minutes I was in awe of how quick and cheap it was. Then I saw just how bad the decision are and how it would get stuck in strange loops of going in and out of the same door to no end.

This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking up but probably won't start building around it yet.

View on HN

This entire conversation around Jev seems weird to me. Like... we started from neural nets that could do basic decision making and classifications pretty well, then trained larger and larger language models to get to where we are now. Now suddenly everyone is going crazy because someone trained a smaller model that is adequate at making decisions? We already went through the "look this AI can play pokemon terribly" phase like a decade ago.

A pre-trained universal classifier that can replace specifically-trained ones would have been considered just as much science fiction in the 2010's as the capabilities of modern LLMs. I'm not sure Jev is actually there yet, but at least it sounds theoretically doable today.

That being said, one thing having been unrealistic 10 years ago and just about possible today doesn't mean that it's going to change the world the same way another technically related, previously-impossible thing did. The Jev hype gives me a bit of the "you're still early to crypto" vibes of some later altcoins. I really like the idea, I think it's going to open up possibilities for using classifiers where we wouldn't or couldn't have trained one before. I'm crossing my fingers for an open weights version to drop. But it's still just a classifier, people have built similar things before Jev, the one thing that really stands out about it is their ability to generate hype.

> A pre-trained universal classifier that can replace specifically-trained ones would have been considered just as much science fiction in the 2010's as the capabilities of modern LLMs. I'm not sure Jev is actually there yet, but at least it sounds theoretically doable today.

That's the point. It can't. And it's not even close.

Sounds eerily familiar to how we talked about ChatGPT doing [various things that it now does easily] when it was released.

> but at least it sounds theoretically doable today

why

We have a bad universal classifier now (via Jev). 0->1, one might say.

A bad universal classifier does suggest a good one later. And that is exactly what I would call "theoretically doable"

That said, I don't think that Jev is a magic breakthrough or anything. I think it is just a particularly good narrative with an easy way to try it out.

Jev is interesting in that it's much cheaper and faster than a frontier LLM.

But I've seen nothing to indicate that the upper bound on classification tasks of a Jev-like model can exceed a frontier LLM with reasoning tokens. That seems nearly impossible even in principle (since Jev-style models are still based on LLM pretraining).

So while they're definitely on the Pareto frontier, which is valuable, they're at the "cheap" end of the spectrum more than the "good" end and I don't expect that to change.

LLMs are like lossy compression of ~all of written text ever produced, with useful recall. To the extent that the corpus contains labelled examples of the given classification task, it's not unreasonable to think that we'll be able to build a decoder for that, just like we already have a useful decoder for next-token prediction. Extend to image classification the same way we already have multimodal LLMs.

The cheap, fast and smart-enough LLM space has been wildly neglected. Jev is one of the few players truly targeting that space. And for a lot of people it is the first time they are asking "what could I build if llms were interaction-speed fast?". The answers are cool, the problem is that Jev is not, I think, smart-enough yet to have that many applications, but it's smart enough that you can start to see what they will look like.

The difference is that you don't need training here; The decision graphs can be built on the fly by an LLM and contain instructions in plain language.

The magic moment for me from the Jev release was not that there was some system playing doom: Rather it was the moment, they just changed a part of the prompt to "don't shoot, just dodge" and the behavior changed immediately.

This means you can have a system with fast decision-making but still interact with it via language.

You’re kind of screwed on the latency optimization side bc your bottleneck is a language model though. Why wouldn’t you eventually just use a specialized model that isn’t constrained by running a language model under the hood?

True - in the long run. But that's quite expensive. And you'd need a bigger org to even support this.

They even named the model after "Jevons Paradoxon" - They anticipate that their model lowers the cost of adopting this kind of AI significantly, unlocking a lot of use cases.

It is impressive, but all the hype and fake demos are selling it as a model that is as smart as frontier reasoning LLMs in the decisions it makes yet much cheaper and much faster, which is not true.

Nothing fake here and fully open source if you wanna take a peek. It does make a bunch of mistakes, often. But it eventually recovers!

https://github.com/christianmat/jev-pokemon

For those curious how it works, it’s essentially a script that plays the game but uses jev as a source of rng to make it stochastic

Yeah, there's a lot hard-coded into the typescript files that make this much less impressive than the other "AI plays" versions that have come and gone that play with less help. A Claude one that only used screenshots would always get stuck in the rocket hideout...

Does dropping a prng in place of Jev produce similar results?

For the record I did not mean to say that about your work, nice job.

I think that misses the point of Jev being ridiculously efficient while maintaining adequate intelligence for automation tasks. We have to train our minds to filter out branding and marketing.

This happens all the time in tech. A few years ago everybody got excited about static websites and server-side rendering as if we hadn't been doing that with PHP long ago.

For what it does, it classifies, orchestrates, operates and delegates tasks exceedingly well for its size and weight. It's ridiculously cheap and efficient, but if you can only see progress in terms of raw cognitive power then you'll surely miss how interesting this is.

Performance and efficiency have been neglected as everyone threw every available GPU and trillions of dollars trying (and failing) to create AGI. Though the results have been impressive.

But good enough for pennies in an instant is very useful.

The difference to me is that the models from a decade ago would need to be trained to play the game. As far as I'm aware, Jev has had zero specific training or fine-tuning to play the game. It's simply told, "Hey, here are the rules. Play." That's it.

I work in manufacturing. I think this will be fantastic for stuff like SPC.

I agree it's overhyped, but the transition to a general purpose classifier (vs a narrow scope classifier) is new and noteworthy.

Ie the famous "Hotdog" clip from Silicon Valley [0]

https://www.youtube.com/watch?v=ACmydtFDTGs

Maybe noteworthy but definitely not new. The category of zero-shot classification has been around for a while.

Example (2022):

https://developers.openai.com/cookbook/examples/zero-shot_cl...

Yea but this model wasn’t trained to play Pokémon but it can. It’s general.

So is a random number generator. That doesn't mean it can play Pokemon.

Making decisions quickly, cheaply and without having to train your own model.

Math.random can make poor decisions quickly and cheaply

Benchmark it against jev and you'll have your answer.

I mean,

> get stuck in strange loops of going in and out of the same door to no end

Math.random is statistically unlikely to do this.

Maybe math.random would've taken 10B years to get to that door in the first place.

Pathfinding is handled by the harness

No you have to use Math.sqrt(Math.random()) to outperform jev

mathrandomplayspokemon.org

The exact message I sent my friend this morning:

> the most interesting thing about this jev stuff

> is that people are seemingly like

> completely disinterested in how smart it actually is

> I haven't even heard it mentioned a single time how it actually compares to other LLMs coming up with their own classifications. Just: it's fast and cheap

After watching a few minutes of this it makes me think that maybe we should be a little more interested in how smart it is.

They know it's not. The company actually has or had public statements that they didn't like public benchmarks for comparison.

My main wonder is the difference between it and having a small llm no thinking output a single number only as a choice. Isn't that nearly the same here?

There is a lot of room for a lot of different models. For many use cases, intelligence beats out all.

For me in my day job, having extremely fast low quality decision makers over noisy inputs is very valuable. I work in security and having something that can help triage alerts, classify items and group things together is extremely valuable. It doesn't need to be perfect. Just being able to take a set of inputs from deterministic tooling and to be make general priority classifications goes a long way on helping humans look at the most important items first.

Why not spend a few tokens or so on making a frontier LLM build the classifier from scratch? Then you also know exactly what the classifier is capable of, and whether it’s easily learnable/informative data.

Why would you be confident in feeding garbage to a “cheap and fast” classifier with unknown domain-specific performance?

I know we kind assume omniscience for frontier models, but at this point the evidence is kind of out there.

Because it's not that smart especially when you compare it to other LLMs. The top LLMs have completely change the baseline of being smart.

You gotta admit that turning "our model is very dumb and can only make simple decisions" into "System One Model" is a pretty clever marketing move.

Jev is for single shot classification, not multi-step RL environments with delayed reward and explore/exploit. My guess is it would go through the door with high confidence every time unless you change the input to add the history.

This runs entirely on Jev as the only AI with a typescript harness that feeds it selective context.

I think the people behind jev made their intended use clear by labeling as "system one" - so they anticipate that there's a slower "system two" mediating jev

Like others have mentioned in this post, I think a mix of models like Jev for simple stuff + a smarter reasoning model for more strategic thinking is the optimal solution. This experiment however is purely Jev. Which sometimes can be kinda dumb.

Well in a lot of ways this demo actually is a mix of jev for simple stuff + a very intelligent harness to fill in the rest. Pay attention to the "jev counter" on when api calls actually happen and the kinds of decisions its making. Its very rarely in a tight loop, and when it is (eg in an item menu) it tends to randomly walk through options.

Its very cool, but the harness is doing a _ton_ of heavy lifting.

Oh I want to be clear, I don't want my post to come off as negative. I really do mean things are headed the right direction and I think this harness is super cool. So don't take it that way and thank you for bringing something cool into this world.

feels like pre-LLM AI playing a video game but more expensive

Reply on news.ycombinator.com