Google Gemini 1.5 Pro leaps ahead in AI race, challenging GPT-4o
venturebeat.com
venturebeat.com
Image was similar to this: https://i.pcmag.com/imagery/articles/03Rqi88BidPCECjR5HeA0ex...
Anyone with even a passing knowledge of cartography will tell you that is one dirty, dirty map.
More generally, the cat is so out of the bag at this point that I'm starting to equate calls for more filtering and stricter regulation as just attacks on AI in general. Anything that will stick, anything but the kitchen sink.
Maybe that technically, filtering is a separate component, but from the reviewer perspective it doesn't matter, it is a misinterpretation, an incorrect answer.
A filtering problem would if a model refused to answer about how porn is considered in some country because it is a sexual topic. That would be a correct interpretation (porn is a sexual topic indeed), but a filtering failure, because talking about that from a legal, cultural, religious,... perspective is usually not considered offensive.
Anyways, ChatGPT doesn't have any problem with these questions (in text form). "broadband household penetration by state" is properly interpreted, though, if asked to, it would suggest using the world "coverage" instead to make it less awkward, which is a perfectly reasonable answer. Asking about how pornography is considered in some country also results in a perfectly appropriate answer about legal aspects, culture, etc...
The filter can be implemented differently (e.g. RLHF or not), leading to different ways in which it surfaces. The way I see it, these differences can be stark precisely because it's conceptually orthogonal. If it was closer to the knowledge, the models would filter the same because their knowledge corpus is roughly the same.
Some other factors probably at play are the companies' different tolerance levels for Type I and Type II errors, and different aversion levels to jailbreaking.
At some point the meta-knowledge aspects that define AI behavior will become a more integral part of the preparation, like Education students taking required classes in Pedagogy. I think that's when things can really start to go wrong. I'd much rather have a chatbot that just wants to please me than one that thinks it knows what's good for me.
/s
Maybe the model is high-quality if you can get a proper answer, but I would much rather have a lower-quality result with less filtering, guardrails, and condescension.
https://info.umkc.edu/womenc/2016/05/20/showin-off-her-map-o....
Yes, and it's not even related to the filter. LMSYS Leaderboard is consistently all over the place lately. It's clearly some uncaught systemic bias or maybe outright manipulation, either way LMSYS results are not useful or sometimes even hard to believe/nonsensical for anyone who actually compared the models in the corresponding tasks for practical matters. It wasn't like that a few months ago.
Benchmarks and tests are too easy to fit the model to, and there is a proliferation of them so they can be cherry picked.
So unless we can point to something better, we can be constructive and just improve LMSYS by using it.
To be more constructive, here's a few things that make me instantly suspicious of this leaderboard in its current state. I'm talking about the API experience of course, not web frontends with unspecified amount of instructions behind the scenes.
- Anything ranking above Anthropic models in non-English languages, maybe except Gemini.
- 4o mini ranking above Sonnet. Seriously? It handles every task I give it worse than Sonnet, which is consistent with its purpose.
- Sonnet and 4o ranking way above Opus in multiturn conversations. Every model is plagued by self-reinforced repetition in multiturn, but 4o and Sonnet are especially bad at this, the latter is nearly unusable in multiturn due to this. They're clearly not optimized for multiturn chats. While Anthropic models have terrible long context degradation and lost-in-the-middle issue, Opus is pretty reluctant to repeat itself, and can actually be used in multiturn, despite often being less performant than both.
I also have plenty of questions regarding open-weights models as well.
Either way, to be truly constructive, you could point to an alternative, or to a way to improve the current alternative. For example, by improving manipulation detection, or by differentiating rankings by query category.
At least with GPT and Llama, positive intent is assumed or questioned, but with pushback I’m able to move forward.
But yeah, constant leapfrogging is the steady state. I'm thankful 2024 is not 2022, when we were just waiting on ChatGPT updates like we wait on new iPhones. Yay for competition, non-hegemony, and the fact that it incentivizes open models - even if only as competitive torpedoes of sorts.
When Anthropic Sonnet doing great at text and coding, OpenAI with their reach via web, mobile, and desktop app, and free deepseek interface with great coding model, who needs to turn to Google with their 20 years old looking interface?
To start, they need to start making modern interface and modern billing to catchup rather than tying it to GCP
I mean, don't get me wrong, I think these models are pretty good as they are and they can be useful if they could run on devices natively which seems to be something that might be happening. That's exciting.
But in terms of these models getting better... I don't know. I think we've been doing very incremental upgrades rather than big changes for a while (a while being like 1 year... but that's how fast this tech has moved).