Hacker News Reader

Hacker News

Comment by tmikaeld | Hacker News Reader

Full thread

tmikaeld

Is there a place that compares/scores all of these local LLMs?

Would be nice to know how close they are to the paid ones.

Edit: Found one here:

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

View on HN

Off topic: Jesus Christ that table is bad. How badly have they messed it up that reversing sort order takes multiple seconds on my pc? It was so slow that I thought maybe it is sending a new request to the server every time I change the sort order but no, it is just that slow.

The power of modern Javascript. So what if my pentium 133mhz could invert an entire table with hundreds of cells in lotus 123 back in the early 90s in the blink of an eye. The fact that 20 years later with computers 100x faster inverting a table takes longer is progress! Also don't forget to mock people who care about performance by screaming "premature optimization" without fully understanding what it means.

Blaming poorly optimized sorting on a whole ecosystem of a language feels lazy at best. It wouldn't have mattered if it was done in Python, PHP or whatever, if the developer who programs something doesn't know how to sort things efficiency, there is nothing to say they'd do a better job in a different language.

Bad programming is just bad programming, no need to pull what language is being used into it. Just happens to be JavaScript because that's what browsers natively support. But if Python (or whatever) was used instead, the very same programmer would have done the very same mistake.

Careful. I just got a slap on the wrist for calling out lazy commentary, apparently it’s “taking a swipe” at someone.

> Blaming poorly optimized sorting on a whole ecosystem of a language feels lazy at best. It wouldn't have mattered if it was done in Python, PHP or whatever, if the developer who programs something doesn't know how to sort things efficiency, there is nothing to say they'd do a better job in a different language.

Counterpoint: It's not what programming languages do, it's what they shepherd you to

https://nibblestew.blogspot.com/2020/03/its-not-what-program...

Yes the entire eco system is bloated and badly optimized. A lot of the web has become laggy, slow and battery drainers due to modern js frameworks.

You're right it isn't the language but since js is the only front end language it gets called out.

Just so we are clear, I love JavaScript and write it everyday but I avoid the “modern” eco system like it is the plague.

>Blaming poorly optimized sorting on a whole ecosystem of a language feels lazy at best.

The sorting is actually fine. On my Mac the sort completes in less than 1ms.

The performance issue is because of all of the hooks of the svelte framework trying to do their thing in the middle of the sort, and then another frame-sizing script trying to do its thing in the middle of the sort. This is an endemic problem in Angular apps as well.

Not javascript's fault, or even the sort algos fault.

Thanks for doing somewhat of an analysis, I was just guessing it was a shitty sorting algorithm, but thanks for clarifying the actual problem :)

...who are you arguing with?

No one, just making an observation.

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

It doesn't include a comparison to the paid ones, but it's a great place to look at different ones and see which ones you should experiment with and sink time into.

Found it a minute after i posted ;-)

Have you tried any of the high-scored ones?

They change somewhat frequently. We were previously working with some of the highest ranked that we could handle and we were getting acceptable results for the parameters of our tests.

That said, if you’re looking for these to be a similar quality to OpenAI’s ChatGPT 4, they’re not even remotely close.

> if you’re looking for these to be a similar quality to OpenAI’s ChatGPT 4, they’re not even remotely close.

I figured as much, but it's incredibly hard to find any comparison that's kept up to date.

What I was after was a local javascript LLM coder that i could train on a local codebase, but even that was fruitless except for some unmaintained "promptr" project.

I guess OpenAI embeddings is the only option.

I have tried Stable Beluga which was released recently. Even at 7B, the smallest, it does a decent job.

I've been playing a lot with Llama 2 13B recently, and it's really not bad at all. With oobabooga[1] you get a proper UI for it and even get an OpenAI-compatible API, so you just change the endpoint in your OpenAI library and it all works. I've been using that to test changes to my bots.

As another poster mentioned though, it's nowhere near the level of GPT-4. It's close enough to GPT 3.5 though, you should try it out!

[1]: https://github.com/oobabooga/text-generation-webui

Thanks, I've had oogabooga bookmarked for a while, it's time!

What happens when the average approaches 100%?

i.e., do these benchmarks taken together capture anything resembling general intelligence?

When the benchmarks approach 100% it'll be time to create harder benchmarks. We'll know we're getting close to general intelligence when it's a major challenge for humans to create harder benchmarks.

you trigger the on-site atomics to prevent a containment breach.

There are more evals than you can shake a stick at, but none of them are comprehensive or without their own issues. If you're looking for a lists of lists, I keep one here: https://llm-tracker.info/books/evals/page/list-of-evals

It would be good to know what the hosting/scaling costs of each of these are too on cloud providers. Do all of the llama based ones basically share the same performance characteristics? If cost is the primary concern, is OpenAI (3.5) the only game in town right now or should we look elsewhere?

https://tatsu-lab.github.io/alpaca_eval/

Reply on news.ycombinator.com