Hacker News Reader

Comment by refulgentis | Hacker News Reader

This is incorrect.

This is the first time there's been a non-private model at the top, and I'd wager top 3.

This isn't OSS either.

Not sure where the jaundiced view comes from. I generally believe the free models tend to be overrated, but I don't think it's a good idea to pretend there's widespread gaming of this. There's a strong self-peasantization streak in LLM discussions, and truth is, no one thing may be "correct", but we can't exclude all data without making the very act of discussion meaningless.

View on HN

You have to be seriously delusional to think that a 70B model will beat Claude Opus. But go ahead and use it to prove my point.

I know the power of Zealotry and Cult in the age of internet and this is just one example of that

I'm at the front of the line, I maintain an app with integrations with all 4 major providers and llama.cpp with 4 built-in models. Yesterday, I was one of the, if not the, first to get it integrated and flag to model makers they had the wrong chat template embedded.

That's a lengthy way of saying, contra what you imply, I have practical experience & understand this stuff intimately.

I agree with your opinion re: it's unlikely LLaMA 3 70B beats all private models, but think your way of relaying it, claiming there's widespread gaming of it by advocates for open models, is obviously incorrect, and adds more confusion rather than reducing it.

Why do you take it personally? Just because you are genuinely interested doesn't mean there are army of people who is sitting their with the sole purpose of gaming leaderboards.

If you think that doesn't happen, I have a bridge to sell you

I'm not taking it personally: frankly, I'm not sure what that would mean in this context. You asked me to try it, I explained that I did, and provided context.

Thank you for clarifying: to confirm, yes, I do understand your claim is a vast cabal of OSS zealots rigs LMSys's blind A/B testing.

I don't think there's anything more of value I can contribute to a discussion on that topic.

Have a good weekend!

I'm a big fan of Elon/Tesla, but I also know Elon/Tesla (a vast majority of them) is a cult. See? it's easy to have those seemingly 'conflicting' thoughts

Then explain to us how they game it? The test is blind. You don't know which models give the answer. If you trick them into disclosing their name, it is discarded right away and doesn't count towards the score.

Genuine question, because as far as I know there is no feasible way to game this score. But if there is, I want to know.

> If you trick them into disclosing their name, it is discarded right away and doesn't count towards the score.

However that process works would be the thing to circumvent, like by getting it to disclose some letters of its name, or any value that is different between models but the same or similar for each one. I assume it doesn't provide the numerical token values in the output or it would be trivial.

It's not beating Claude Opus. It's rank #5, tied with Bard and Claude Sonnet.

It was during the time of posting, which kind of proves my point. It is gamed. A statistically random sample of users would have yielded the same results.

https://twitter.com/Teknium1/status/1781328542367883765/phot...

How does that "prove" it's gamed? If anything it proves that the system does exactly what it's supposed to do: correct itself over time using more and more samples from users.

Reply on news.ycombinator.com