Hacker News Reader

Comment by gatapia2 | Hacker News Reader

I don't understand Jev. Its a generic classifier right? Like the classifiers we were building 15 years ago with random forrests and logistic regressions, but just generic. What's so revolutionary? And how can the accuracy be any better than a custom trained classifier that can be built in a day (an hour using Claude).

I don't understand the hype.

View on HN

> custom trained classifier

The "moat" is data. And compute. But mostly data.

It is a classifier, that you don't have to train, that seems to work well, and they have made available by API.

One of the reasons for LLM success has not just been that they are "smart" but that they are smart while not requiring you to collect your own data and perform your own training.

There is also the marginal advantage of it not requiring a PhD to know how to use. But most companies can get a competent data scientist on board, so that is not a major problem, they could find someone to train their classifiers. But even a good data scientist can't make data appear out of nowhere. If you can get a pretty good result without needing data and training, for cheap, you're going to take it.

(Let's put aside the fact that almost every company using LLMs these days is irresponsibly NOT comparing the results with their own internal gold standard datasets for validation and calibration. They are just winging it. And Jev allows them to continue to do that.)

The core feature is good quality out-of-the-box domain and cross-domain calibrated probability you can use as a threshold.

SO you do not need to have YOUR data to train classifier. All you need is to benchmark it on your process/data to see that it really is good for your domain (as authors promise) and choose a threshold for acting.

Ahh, and the most important thing you need to do - design well the input context (state), questions with their context (instructions) and output type (because the same question asked as Noul (yes probability) or Choice with yes/no variants give different probability of yes.

You don't need to train it, you can throw pretty much whatever at it and it'll work and it's extremely cheap. That's it.

I don’t think its much cheaper like luna is double the cost. and you can easily use luna to generate a training set and train your own model. I think the main things its faster so a lot of people go the Wow moment I can classify my whole db at once. Or this model decides so quickly. Speed is the key differentiator, it’s a strong one, always.

"Double the cost" so jev is half the price. That's a huge difference.

I also don't get the point. So what if the output format is different? It's still just as hallucination prone as before, right?

I was watching some youtube hype demo of it to pick items (each an emoji) from a pile according to questions. One of the questions was: "what can a magnet attract?" and it picked like 8 metal items, while a literal magnet was left unpicked. Ofc the video was too busy praising it to notice. Was all I needed to see

That it can make mistakes? That’s expected, it deal with plausibility like other models no? You would need to look at the actual response and the assigned probabilities to evaluate. Generally demos are not a great way to evaluate a technology, it’s a way to get hype but the next step is to actually look at the details

But isnt the whole point of Jev that "we trained it so you dont have to"? Yet it make such a simple mistakes still.

Yes, I don’t think hallucinations will ever go away. But because it doesn’t work with language the type of hallucinations aren’t too comparable to LLMs. But it is still a risk of course, whatever system you design should take that in consideration

> What's so revolutionary?

Because it's all "AI" now <insert Ancient Aliens meme>

> Like the classifiers we were building 15 years ago with random forrests and logistic regressions

I think no one mentioned here, but the obvious difference is that Jev can spit out decisions directly from natural language input. None of these ML models could do that, and other than a full-fledged LLM (which is optimized for conversation and agentic tasks) or some classical NLP models (which underperform compared to LLMs, AFAIK), there is nothing right now that rivals Jev-like models.

Of course it's not perfect, but seems like the right step forward for quick classification/decision tasks based on natural language.

As far as I can tell from just looking at the documentation, yeah, it looks like a "better than nothing" and "definitely better than LLMs for a few tasks" tool, and quite possibly something that I'd build into a demo, but I'd be really careful deploying it in any kind of product where exactness matters.

Have you tried Jev and compared it against your alternative classifiers? Should take half an hour to do that, then you’ll have your answer (or someone who already did it can tell you here).

I did this yesterday with user data to find duplicates and determine which conflicting fields should win in a merge. Took me about an hour-ish to run the experiment. I provided a good amount of context for each pair. It was underwhelming / OK at best and didn't add any value to my existing merge workflow, so I threw it out after the experiment (I kept the findings though). It was quite easy to setup and use via openrouter and extremely cheap and fast.

For fun I also tried using it at my local LLM router (all of my prompts and responses go through it for personal analytics) to decide which model to route tasks to based complexity etc. Again, underwhelming for my purposes, and so I'm not using it.

Not sure what real world use case it's best at, but I agree that it's simple enough to implement it yourself and see if it fits the type of work you are doing.

Could you tell us? That would be simpler for everyone no?

You need to do it to know the results for your own use case. The barrier for trying things out is extremely low now, just relying on others to evaluate tools for you is not very effective as their use case is unlikely to be the same as yours and these tools vary a lot from case to case.

Reply on news.ycombinator.com