Hacker News Reader

Comment by NeumannGod | Hacker News Reader

This is a false comparison. What Jev does is fundamentally different from what LLMs are doing as a judge.

For a quick primer, Jev is able to provide confidence scores on its classification, i.e. it is able to calibrate how well it is able to predict. Being able to predict in a distribution is different from being able to calibrate confidence of the predictions which should happen from the question or domain distribution from which the decisions are predicted - being able to do that is tough and is not same as using LLMs logit probabilities which are predictions in the vocab space. Though both are loosely correlated and might converge as LLMs keep getting better, the former is a much stronger decision-making signal than the latter. Jev not beating LLM-as-a-judge might be due to various other reasons such as world knowledge etc, but Jev as a concept will always provide more reliable decisions / outputs than LLM-as-a-judge giving a scalar score.

View on HN

This feels more like mythology though.

Claude and GPT tell me all the time what they are confident and what they're not confident in. This is partially useful, but it's not a guarantee of anything, since they can be confidently wrong or unsure and correct.

Same with Jav.

Arguing that Jav in some way more useful than current LLMs because it includes a confidence score smells of someone who hasn't updated their priors since GPT-4/4o and it's particular issues.

Jav is cool because it's very fast, very cheap, and "good enough". That's it.

> Same with Jav.

There are benchmarks for these things, which Jev and others have been put through; it's not a marketing boast!

The benchmarks show it is on par or worse than even older LLM models.

* With jev-sec-bench the confidence numbers reached roughly the same as Qwen2.5-7b-instruct (confidence was off by about 6% on average in both cases).

* On the OpenRouter Banking 77 benchmark Jev did the worst out of all the models, being about 25 points off from reality.

I also think the fact that Jev hasn't published any real papers or actual technical details should make people a bit more skeptical than they have been.

That's a really weird way to test Jev. I don't see any mention of latency or cost in https://github.com/Gaurav-Gosain/jev-sec-bench

If you don't care about that why wouldn't you use the flagships??

I'm not arguing against Jev. Jev is very cool.

I'm arguing against the idea that Jev fixes a fundamental deficiency in LLMs by showing it's confidence.

It doesn't.

Jev can do a strict subset of what LLMs can do, but incredibly quickly and incredibly cheaply, which is certainly valuable.

Yes, it does. A low-latency calibrated classifier is valuable.

I think the point is if you can't get the right answer, it might not matter how fast you can make something up.

It had an accuracy of 96.5% with context! Come on, guys.

Partially so. But the fact that we're comparing Jev to SOA LLM and pre-trained classification means it's essentially satisfied its reason for existence.

The benefits of Jev, and the reason I'm excited about them:

* Pre-trained/tuned to provide only structured output

* Small, fast, essentially trivial to locally run - this alone makes them an actual candidate for real-world planning systems. Laya is a local-first 400 million parameter model, for example.

* Their confidence scores are useful, even if those scores are not correct/true probabilities.

For me and my work, they look like nearly turn-key, tiny, fast, locally-hostable models that can manage state transitions in a deeply autonomous system.

LLM-as judge requires, well, what we know as a full LLM. A fully trained classifier model requires fully training - something you cannot / won't do for a embedded autonomous planning system (by contradiction - if this worked, we'd have used it everywhere already!). Totally unfair comparison in my mind.

With llama.cpp, you can give it an ebnf grammar and constrain the output to any formal grammar you wish.

And how is the confidence score any different than the softmaxed token probabilities you get out of running every llm?

> With llama.cpp, you can give it an ebnf grammar and constrain the output to any formal grammar you wish.

With llama-server, you can use basically any GGUF model as a Jev-like classifier. See Pi.dev's codemode and the accompanying llama backend to the "classify" function. The advantages of Jev-like inference are that:

- You generate 1 token per question in parallel, instead of sequentially generating a bunch of JSON punctuation. So you get much better latency and utilization.

- You can use a custom "sampler" that sees the raw logprobs for these tokens, including all the tokens that might have been generated. So you can generate some "probability" or "confidence" numbers based on how likely the model was to have generated a different answer.

So really, no difference? It's just a nice API with some computational efficiencies.

> And how is the confidence score any different than the softmaxed token probabilities you get out of running every llm?

You're asking me how a structured output which assigns probabilities to classifications of the input text is different than the next-token weights which are calculated while generating that output?

One is internal (token prob), one is output generated by that internal mechanism e.g.,

    "{ next_state: { "evade": 0.84, "land": 0.10, "search": 0.01"...}}"

Yes. I am.

You have a ebnf grammar constrained to evade, land, search etc. Quite literally what's the difference between the log probs between those options and whatever number a very tiny model is coming up for the confidence.

Copy. The mechanism is probably similar once we abandon the "LLM that is asked to generate json" path. But the grammar alone doesn't ensure the output is a probability vector does it?

Both use the softmax operation, but "probability of emitting a given label" is not the same as "probability the decided action is correct". OTOH, taking an action to sample something to come up with "probability of emitting a label" is in fact the whole shebang if the label is the next action. If it works it works.

Who cares, this distinction is small.

For me, a planning guy, either your suggestion or Jev-like would work just fine, and you better believe I'll be testing both as I'm quite excited about being able to use less-structured, untrained systems in decision pipelines. Thanks for the heads up.

ebnf grammar is a truly underrated feature of llamacpp.

the "provide only structured output" is done by NOT having the model do it, but regular old software. So tuning/training doesn't apply for that.

I agree that there are differences between Jev and LLM-as-a-judge (e.g. Almeida's assertion that LLM probabilities have been irrevocably biased by RLHF), but I am not sure about your description of 'confidence'. Perhaps I misinterpret you or the docs, but I understand it as simply being computed from the probability distribution: https://docs.typesafe.ai/confidence#how-confidence-is-calcul...

I don't know about Jev, but in my experience, 90% of the time, the confidence score such machine outputs (e.g. simplest case a kalman filter) is bogus because the model is wrong. Or not even wrong, but just not perfect.

Mathematicians build this, and they love the beauty of it so much that they loose touch of reality.

Usably good calibration is hard. It's hard with logistic regression, it's even harder with linear support vector machines, and it makes me question my life choices with non-linear models.

It's not necessarily because the model is wrong. I've had trouble getting useful calibration out of models with an F1 of 0.9. The fundamental problem is twofold. First, it turns out that [0.0, 1.0] is a much larger set than {0, 1}. Second, the kinds of use cases where you care about calibration tend to be fussy and demanding.

As a fun anecdote, I got a chance to ask the person who invented the method scikit-learn uses for calibrated SVMs if he had any advice, and his answer was basically, "good luck."

All that said, sometimes you don't actually need calibration; you just need decent ranking. "Items with a score of 0.9 should be more likely to be positive than ones with a score of 0.5," is an easier requirement than "90% of items with a score of 0.9 should be positive." But I've had trouble getting that out of highly non-linear neural models, too, because they oftentimes produce results where the relationship between score and probability of being in the positive class is not even remotely monotonic.

But I haven't poked at Jev like this either, so I do have to allow that maybe they've found the secret sauce.

This is hocus-pocus, Jev is basically a dumber LLM constrained to certain problems, it's not magic.

Jev's "confidence" scores are just as valid as an LLMs confidence score, i.e. just as likely to be wrong or hallucinated as any LLM.

Ultimately, people are somehow thinking Jev is mysteriously more likely to be right when ultimately it's another non-deterministic magic box except this time with a stricter interface slapped on top of it.

you know absolutely nothing about the internal architecture of Jev. why are you being so confident?

Fine-tuning language models for calibrated probability predictions has been a thing since the Bert era though, that's really nothing special.

Yes, but they are not able to act on their decisions, or ask for elaboration.

Maybe someone did order those enlargement pills, and an agentic model would be able to scan email history and determine that, when classifying an email as spam.

for the uninitiated:

LLM’s are not able to give confidence scores, they make them up.

Your AI driven app is making that up. Your product manager and executive team’s demand for confidence in the UI is a totally fictional cosmetic telling them nothing. Your company sold bullshit confidence to your clients.

I’ve done this for many organizations that “formed a new team to work with the CTO on their AI strategy”, and the trappings are the same

You can have an LLM tell you how much of a schema it was able to get information about. And derive a “confidence” or level of compliance from the completeness of the schema

But this is layers upon layers of cruft that a classification model wouldn’t need

Softmax over logits doesn’t give calibrated probabilities either as a rule. I don’t want to comment specifically on Jev but as a rule it’s very hard to get good calibration because it’s somewhat in tension with minimizing training loss for neural networks, e.g. Guo et al (2017) https://arxiv.org/abs/1706.04599

While I know there are ways to improve calibration, I’d personally want to see a lot of evidence the probabilities were actually more meaningful before trusting them. I agree of course that asking an LLM to provide a confidence estimate is meaningless.

LLMs had ability to do this from the start, but at some point providers stopped exposing this api to prevent distillation. Just write the prompt, pass the data and sample 1 token returning logits, OSS inference engines can do it easily.

all llms output confidence scores on their tokens. That's literally all they generate. Picking an actual next token is a (semi) random sampling of those confidence levels.

The things that I'm actually most excited about Jev-like models are:

* Latency: when you want to make a decision within 100ms

* Robotics: I want to try make a VLA version of it and see what happens

* Costs: For something that needs a yes/no answer every 10 seconds, it seems like it would be cheaper than an LLM API.

a less intelligent decision is not "more reliable"

LLMs can provide confidence scores. Just ask it and it will provide.

Are those scores accurate? Probably not. But are Jev's scores accurate? They certainly aren't with a torture test like "I've rolled a fair die. What is the result?" (It comes back with ~80% confidence on 1 in my testing.)

It seems like people are very enthusiastic about Jev giving scores for its results, but I'm not seeing a whole lot of attention paid to how good those scores are.

That's an interesting test, but is it really what jev is supposed to be good at? I mean, this is a genuine question because I don't understand jev, but wouldn't it need a more constrained question?

Probably not. But it does seem to have some idea of probabilities if I get a little more specific. "I've rolled a fair die. I see several dots on the top." Gives me 46% for 6, and about 16% for 4 and 5. If I say "The top is mostly white." then it says 1 with 50% confidence... and 6 with 31%.

Interestingly, if I add a "unknown" option then my original question says unknown with 100% confidence. "I see several dots on top" gives 89% unknown. "The top is mostly white" gives 30% unknown and 37% 1.

So it definitely has some ability in this area, but it's iffy. This isn't exactly what it's made for but it doesn't seem too far off from the examples like taking in a customer email and deciding which team it belongs with.

This does feel like the sort of overly-confident but unfalsifiable output that llms excel at.

Reply on news.ycombinator.com