Local LLMs are much better than their reputation

I always read "well, if I only could run the 600b model, it would be usable, 70b and 8b are just bad.

This is not my experience at all. Especially models that "think" before they write are astonishingly accurate even down to 14b.

I always think "damn, you noticed THAT, wow".

View on HN

I am guilty of this, to a degree. I tried every local model around the time of the original llama 3 release, and for a while after that. Mostly the models around 70b. They always felt like toys compared to the SOTA hosted foundation models.

After that, I stopped trying them all out, because API access is so inexpensive, why wouldn't I use the good ones?

More recently, I tried a few local models, and they are more impressive than they used to be... but still far behind their bigger brothers. They are good enough for some small tasks, but also less reliable in their output, so I'm back to API access for pennies.

It's not the price, it's the privacy that drives to local ones.

I should have mentioned that also: getting access to HIPAA-compliant models was the tipping point for me.

Can you explain to me how HIPAA makes sure that this works?

Reply on news.ycombinator.com