Hacker News Reader

Comment by greatpostman | Hacker News Reader

TLDR: openai still crushing the competition. Competing models beat benchmarks, but are borderline worthless on real tasks.

View on HN

I think that might have been more true several 8-12 months ago. But now it feels like the momentum has swung towards open source. Multiple models are close to or exceeding GPT 3.5 now. They can do a lot of useful things.

I have come to the conclusion that as much as possible I should try to wean myself off of OpenAI immediately. Because it's just not necessary or desirable to be tied to a single vendor anymore for many tasks. And in 2024 the open source capabilities will continue to increase. Soon everyone with a relatively new computer will be running things like LLMs locally. Within a couple of years it will be integrated into every OS or browser.

Why are people benchmarking against 3.5? To me, the real race started with GPT-4. 3.5 and 4 are completely different beasts and that open models are catching up to 3.5 doesn't mean much. I've yet to see anything come close to 4.

Because 3.5 is a very useful model. Getting there with an open model already is pretty cool.

You are only allowed to criticize Open AI if you are paying a monthly subscription to Open AI

> Multiple models are close to or exceeding GPT 3.5 now

Only in benchmarks as GP said. All the models I have tried either hallucinates like crazy or refuse to answer everything. Nowhere close to GPT 3.5.

How do you define "real tasks"? Even small models that can be run on consumer CPU can produce a coherent summary of an email, as an example. Isn't that a real task?

Going to call BS on this. I've delivered multiple projects this year using open models.

> delivered multiple projects

Very curious about the type of projects you've delivered.

You can look at https://github.com/neuml/txtai. Biggest thing of 2023 was RAG with models like Mistral.

Got it. Thank you. I thought you had delivered consumer facing applications when i read that comment.

I've delivered consulting projects using RAG and Mistral 7B. Not saying it's easy to get the prompts right but I've been able to build systems that work well. Just trying to counter the myth that paid APIs are the only way to do this.

Also curious.

I'm a sample of 1 and also relatively inexperienced. But I felt I quickly reached the limits of what was possible when I tried doing sentence classification with OSS sentence embedding models. The issue was with negation. I'd attributed too much magic to embedding models - they don't really understand language.

Not to say there isn't very capable tech out there. Just to add a datapoint that "sentiment analysis"-like approaches in blogs don't always scale to your particular use-case.

Edit: conscious I've drifted from the topic of chatbot type models, but felt relevant somehow.

Reply on news.ycombinator.com