Hacker News Reader

Comment by londons_explore | Hacker News Reader

The lmsys leaderboard has 700k votes.

Who did all those votes? I did like 5 of them...

I'm slightly suspicious someone might be gaming the votes, because billions of dollars of company valuations ride on who has the best AI...

View on HN

How would they game them? As I understood it, this leaderboard is one where you vote on which output is best without knowing which LLM prepared it.

LLMs are not Facebook's core business at all, and it actually sees in their valuation: if you look at Meta's valuation over the past few days, markets don't care about Llama 3's release or even meta AI.

It gives access to paid models for free...

I bet a bunch of people use it for actual life tasks.

But not when they are not sure the result would be of quality, in the arena which is used for ranking.

I throw about a dozen canned prompts at it each day if I have time, mainly puzzles, knowledge and coding tests. I rewrite those from the first principles occasionally.

It's mostly gamed by open source zealots, who have plenty of time in their hands to send "OSS, BEST" on the internet

This is incorrect.

This is the first time there's been a non-private model at the top, and I'd wager top 3.

This isn't OSS either.

Not sure where the jaundiced view comes from. I generally believe the free models tend to be overrated, but I don't think it's a good idea to pretend there's widespread gaming of this. There's a strong self-peasantization streak in LLM discussions, and truth is, no one thing may be "correct", but we can't exclude all data without making the very act of discussion meaningless.

You have to be seriously delusional to think that a 70B model will beat Claude Opus. But go ahead and use it to prove my point.

I know the power of Zealotry and Cult in the age of internet and this is just one example of that

I'm at the front of the line, I maintain an app with integrations with all 4 major providers and llama.cpp with 4 built-in models. Yesterday, I was one of the, if not the, first to get it integrated and flag to model makers they had the wrong chat template embedded.

That's a lengthy way of saying, contra what you imply, I have practical experience & understand this stuff intimately.

I agree with your opinion re: it's unlikely LLaMA 3 70B beats all private models, but think your way of relaying it, claiming there's widespread gaming of it by advocates for open models, is obviously incorrect, and adds more confusion rather than reducing it.

Why do you take it personally? Just because you are genuinely interested doesn't mean there are army of people who is sitting their with the sole purpose of gaming leaderboards.

If you think that doesn't happen, I have a bridge to sell you

Then explain to us how they game it? The test is blind. You don't know which models give the answer. If you trick them into disclosing their name, it is discarded right away and doesn't count towards the score.

Genuine question, because as far as I know there is no feasible way to game this score. But if there is, I want to know.

> If you trick them into disclosing their name, it is discarded right away and doesn't count towards the score.

However that process works would be the thing to circumvent, like by getting it to disclose some letters of its name, or any value that is different between models but the same or similar for each one. I assume it doesn't provide the numerical token values in the output or it would be trivial.

I'm not taking it personally: frankly, I'm not sure what that would mean in this context. You asked me to try it, I explained that I did, and provided context.

Thank you for clarifying: to confirm, yes, I do understand your claim is a vast cabal of OSS zealots rigs LMSys's blind A/B testing.

I don't think there's anything more of value I can contribute to a discussion on that topic.

Have a good weekend!

I'm a big fan of Elon/Tesla, but I also know Elon/Tesla (a vast majority of them) is a cult. See? it's easy to have those seemingly 'conflicting' thoughts

It's not beating Claude Opus. It's rank #5, tied with Bard and Claude Sonnet.

It was during the time of posting, which kind of proves my point. It is gamed. A statistically random sample of users would have yielded the same results.

https://twitter.com/Teknium1/status/1781328542367883765/phot...

How does that "prove" it's gamed? If anything it proves that the system does exactly what it's supposed to do: correct itself over time using more and more samples from users.

Reply on news.ycombinator.com