LLM Leaderboard | Hacker News Reader

LLM Leaderboard

chat.lmsys.org

41 pointsolalonde17 comments

GPT-4, and I'm surprised, but even Turbo, win out over every open-source LLM on the leaderboard, though Anthropic's Claude beats Turbo which is true in my experience.

How likely is it that this will change anytime soon? Or are LLMs at the level of GPT-4 just out of reach for anyone other than a huge company like OpenAI?

GPT4 uses that MoE approach so its effectively a fine-tuned model for about a dozen different domain spaces.

Not really. That's not what/how MoE models work. It's not an ensemble.

Its only 6 months in for GPT4

I think open source things will match gpt 4 in a year or a little after. OpenAI will need additional models or competitive advantages at the same clip they’ve been doing. This is going to be very expensive for them.

What I really predict will happen is that late 2024 hardware refreshes (being worked on since mid 2023) will prioritize everything related to accelerating LLM training and output

More RAM to different chips, different busses, different coprocessor architecture that we currently call GPUs

and this will lower the barrier of entry enough

GPT-4-base was completed in early 2022, its existence only went public much later. Whatever OpenAI has right now is a year and a half more advanced than current models. Since open source models can't even touch GPT-3.5 yet, they are very far behind.

OpenAI has been cutting deals left and right with newspapers, publishers, and other organizations to train on vast high quality private datasets that simply are not available to open source development. Looking at how datasets are at least as important as volume of compute, we can expect the gap between OAI and open source models to continue to widen.

None of this addresses the gigantic (possibly 1:1000) advantage services with constant workload have when it comes to memory bottleneck efficacy.

I think there will be diminishing utility of “smarter” models

People won’t need them

They’ll need more efficient models: smaller parameter count, faster output, thats just as smart as some current baseline benchmark

I also think this will happen. Graphics cards with humongous RAM, etc.

But I don't see how that changes the advantages that companies such as OpenAI have. Consumers will have access to GPT4-level LLMs, OpenAI will have a next gen offering.

A more efficient and faster GPT4-level LLM coming from a smaller model running in consumer hardware is going to be good enough. It will be competitive if it is easier to train on current events and keep updated, by anyone for nearly free.

OpenAI's next gen offerings need to be something entirely different than smarter, and good for them if they do. All the multimodal capabilities have to be out, readily available, and stuff I haven't thought of yet.

I think everyone's right, or "accurate" that OpenAI will have far more resources, I don't think its important.

At the end of last year, OpenAI was talking about GPT-5 being a dead-end and something besides transformers needs to be made to do much better. We all wait for this.

> OpenAI will have far more resources, I don't think its important.

Who knows which direction they go, but I expect improvements like the following would obsolete GPT-4 and equivalents for most people:

Multi-modal? I.e. text, audio, image, video, physics in video and interactive simulation, etc. the result will be major new features and a much deeper understanding of many topics.

Audio and video input output? Engage with the model using voice, tone, gestures expressions and any artifacts you have: whiteboards, objects, whatever. Bring the model into conversations and meetings with other people to add its input.

Long term memory, summarizing previous interactions, providing long term continuity? “Your” instance keeps up with you without a context limit. For days. Or months. Or longer. Have multiple instances for work and home. Share an instance with family, as a joint assistant.

Live audio and long term memory would create the ultimate voice memo machine. Never forget anything again.

An accurate confidence in its own accuracy? Clear distinction between answers backed by real information vs. confabulation.

Built in ability to use the web and other tools, such as programming and executing calculations on the fly? Much wider access to up to date information and ability to come up with complex calculated answers in real time.

Ability to manage your resources? Make sure bills are paid, proactively suggest ways to save, services to cancel. Manage your health by proactively suggesting & tracking changes, making appointments for troubling symptoms, etc. Remind you of tasks with all information you need proactively collected. Schedule one off services to solve problems, and follow up to be sure they were solved. Not only create a trip itinerary, but manage the whole trip, including intervening to solve inevitable problems. On and on here.

Will they? It's pretty clear that the trend is toward smaller but more efficient models, which further reduces their moat.

There's another type of player you shouldn't underestimate: companies that base their business model on hosting Open Source models. The consumers will pay far less than they do for GPT-4, and perhaps in a year or two, for sometimes that approaches the same quality.

OpenAI will have to do everything they can to stay ahead of the pack. But they're an impressive company, and have a good chance of doing that.

Of these, I think I'm most surprised by Vicuna-33B.

For having MIT Bench and MMLU scores significantly worse than #4 and #5, it fairs quite well (47% and 42% choice rate 1v1). Its only vs the #3+ things get bad (although notably also does better vs GPT-4 than many)

Really, the choice rate's not bad when you really look at it. Even vs the top its 1/3, 2/3. Nowhere near as bad as NBA/NFL vs high-schoolers or similar "Professional Grade" situation.

Also the first Non-Commercial top option.

Where is the new giant Falcon model?

I was wondering the same thing. I'm wondering if it's because it hasn't been fine-tuned yet?

There are no community fine tuned models here either, many of which outperform the open source base/chat models on many tasks.

More comprehensive list:

https://lifearchitect.ai/models-table/

Reply on news.ycombinator.com