Hacker News Reader

Comment by niek_pas | Hacker News Reader

Can someone tell me what the advantages are of doing this over using, e.g., the ChatGPT web interface? Is it just a privacy thing?

View on HN

Privacy is a big one, but avoiding censorship and reducing costs are the other ones I’ve seen.

Not so sure about the reducing costs argument anymore though, you'd have to use LLMs a ton to make buying brand new GPUs worth it (models are pretty reasonably priced these days).

I never understand these guardrails. The whole point of llms (imo) is for quick access to knowledge. If I want to better understand reverse shell or kernel hooking, why not tell me? But instead, “sorry, I ain’t telling you because you will do harm” lol

Key insight: the guardrails aren't there to protect you from harmful knowledge; they're there to protect the company from all those wackos on the Internet who love to feign offense at anything that can get them a retweet, and journalists who amplify their outrage into storms big enough to depress company stock - or, in worst cases, attract attention of politicians.

There are also plausibly some guardrails resulting from oversight by three letter agencies.

I don't take everything Marc Andreessen said in his recent interview with Joe Rogan at face value, but I don't dismiss any of it either.

Privacy, available offline, software that lasts as long as the hardware can run it.

Yeah, that's me. nCapture a snapshot of it, from time to time — so if it ever goes offline (or off the rails: requires a subscription, begins to serve up ads), you have the last "good" one locally.

I have a snapshot of Wikipedia as well (well, not the whole of Wikipedia, but 90GB worth).

Which Wikipedia snapshot do you grab? I keep meaning to do this, but whenever I skim the Wikipedia downloads pages, they offer hundreds of different flavors without any immediate documentation as to what differentiates the products.

You can use Kiwix: https://kiwix.org/en/

wikipedia_en_all_maxi

I guess that means English ... and maxi? As I say, was something around 90GB or so.

Was hoping you had more insight than "maxi sounds good" which is the also the best I have.

Privacy, freedom, huge selection of models, no censorship, higher flexibility, and it's free as in beer.

Ability to have a stable model version with stable weights until the end of time

For work I routinely need to do translations of confidential documents. Sending those to some web service in a state that doesn't even have basic data protection guarantees is not an option.

Putting them into a local LLM is rather efficient, however.

This is a way to run open source models locally. You need the right hardware but it is a very efficient way to experiment with the newest models, fine tuning etc. ChatGPT uses massive model which are not practical to run on your own hardware. Privacy is also an issue for many people, particularly enterprise.

That said though, there's currently no practical way to reach ChatGPT's performance locally, but I have hope that will change eventually. Of course, even today you can reach quite good performance locally, and for a lot of people it's more than good enough for their needs.

I did a blog post about my preference for offline [1]. LLM's would fall under the same criteria for me. Maybe not so much the distraction-free aspect of being offline, but as a guard against the ephemeral aspect of online.

I'm less concerned about privacy for whatever reason.

[1] https://engineersneedart.com/blog/offlineadvocate/offlineadv...

You can chug through a big text corpus at little cost.

you get to find out all the steps!

meaning you learn more

Yeah, agreed. If you think artificial intelligence is going to be an important technology in the coming years, and you want to get a better understanding of how it works, it's useful to be able to run something that you have full control over. Especially since you become very aware what the shortcomings are, and you appreciate the efforts that go into running the big online models.

Yeah but I think of you've got a GPU you should probably think about using vllm. Last I tried using llama.cpp (which granted was several months ago) the ux was atrocious -- vllm basically gives you an openai api with no fuss. That's saying something as generally speaking I loathe Python.

Reply on news.ycombinator.com