Hacker News Reader

Comment by janalsncm | Hacker News Reader

Someone else posted the jevbench site which compares jev to a bunch of other models. If you look only at the accuracy dimension:

https://benchmarkheaven.com/jev-models?w=100-0-0-0#jevc-weig...

Jev actually isn’t anywhere near the top. It even loses to open weight clones. This tells me that whatever their “calibration” dataset is, it doesn’t seem to be anything special.

View on HN

... why didn't you link to the actual benchmark which does have Jev at the top?

https://benchmarkheaven.com/jev-models

You linked to some weird subtable that labeled: " Not the default — not the JevBench Score", that can only be reached after you see what I just linked... lmao are you really this hard up about things?

Also every single question (even in the hard set) is single dimensional?: https://github.com/fstandhartinger/jevbench/blob/main/datase...

Jeeze, this is getting sad. I guess after all the mass-psychoses where people thought pointless things are going to change the world, we were due for a mass-psychosis where something interesting just has to be pointless?

Because I was specifically responding to your claim that Jev’s training recipe would give it better accuracy than others. It doesn’t have better accuracy than others. You could do as well or better by distilling qwen for example.

Jev is ranked higher than others on the overall benchmark due to speed and/or cost, not accuracy.

Well yeah, that’s the whole idea. If speed and cost don’t matter, you could use Astra.

Obviously it’s the speed and cost that make it compelling. The tradeoff is accuracy.

Enough to matter? Maybe, maybe not. It’s not like it’s way down the chart. It’s probably good enough for a lot of tasks.

Reply on news.ycombinator.com