Hacker News Reader

Comment by quest88 | Hacker News Reader

That's a strange question. It sounds like you're saying the author isn't serious in their use of LLMs. Can you describe what you mean by competitive and serious?

You're commenting on a post about how the author runs LLMs locally because they find them useful. Do you think they would run them and write an entire post on how they use it if they didn't find it useful? The author is seriously using them.

View on HN

It's the missing piece of the article. How is their experience running LLM's locally compared to GPT-4o / Sonnet 3.5 / etc, outside of data privacy?

In my experience, if you can make a reasonable guess as to the total contained information about your topic within the given size of model, then you can be very specific with your context and ask and it is generally as good as naive use of any of the larger models. Honestly, Llama3.2 is as accurate as anything and runs reasonably quick on a CPU. Larger models to me mostly increase the surface area for potential error and get exhausting to read even when you ask them to get on with it.

> if you can make a reasonable guess as to the total contained information about your topic within the given size of model

Can you expand a bit on how you might do this?

There's something you don't know that it may know and you want to see what it knows. This is like just a sentence or maybe a few both input and output. All the other talk about this model vs that model vs agents vs rag vs prompt engineering is all about practitioner worries. Keep in mind the thing is probably wrong as you would with any of them. Or that they are subliminally telling you something wrong you may accidentally repeat in front of someone at some time where it really matters and you're going to let everyone and yourself down. Which is current state of all of these things, so, if you're not building them, or are an NLP specialist working with multidisciplinary researchers on a specific goal of pushing research, then these things all have the same utility at the end of the day. Some of the most short sighted systems advice seems to just spill out of Claude unsolicited, so, whatever the big models are up to isn't entirely helpful in my opinion. Hopefully they'll be pressured to reveal their prompts and other safety measures.

Reply on news.ycombinator.com