Hacker News Reader

Comment by rightbyte | Hacker News Reader

This article seems very corporate centric. Like, I am able to run a ChatGPT3-ish code LLM locally on a 2015 midrange laptop. Just like this:

    wget https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct-GGUF/resolve/main/deepseek-coder-6.7b-instruct.Q5_K_M.gguf

    git clone https://github.com/ggerganov/llama.cpp

    cd llama.cpp
    make

    ./main -ngl 32 -m ../deepseek-coder-6.7b-instruct.Q5_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -i -ins

Haven't people realized what they can run themself?

View on HN

I appreciate the sentiment. It definitely seems possible, but it doesn't ever seem as easy as copy + paste if you ever want performance or to go outside the same generic tutorial.

> Just like this:

Just is doing a lot of heavy lifting there.

Is this the first model you came across?

We're there any dependencies you had to install?

Did you have to check video card compatibility?

Where did you get the command arguments. It looks awfully complicated for a "just", as if there's a lot of options that aren't straightforward to use.

Etc

The Makefile for llama.cpp is really good, it will work the way he said.

You can also use ollama which makes it into a one line install and then a simple command to run any popular model.

check out LM Studio

no command line involved, has a huggingface browser built in to load in the model of the day, has a chat-like interface for chatGPT like use, and can create a local server to run your local model for your programs to interact with it using an API that is identical to OpenAI's

This is the easiest way, by far. Then it's Ollama, then llama.cpp. But LM studio is just "install this app, select an LLM you want, done".

Ye ... with "just like this" I actually mean "proficient with compiling C projects on an Unixy system" which is like years of dev, "power user" or admin experience (not being sarcastic here). For the audience here I would say that "just" is about right though.

I have no clue about anything LLM related. I just made it run after reading some comment on HN pointing in its direction.

My point is that these locally run LLMs seems way "underreported". I even tried to make a "Show HN" post about it but it got zero interest.

But maybe I am missing something?

With almost every AI story I read on here (granted, I don't read that many, I find them quite boring), the top comment seems to be something like "you can do the same on your machine, see ollama". Case in point, your comment was the first one listed for me. So the interest for running LLMs on here seems quite high. Don't know why your post didn't catch anyone.

The Makefile for llama.cpp is really good and doesn't require hacking to make it work.

Even on Windows?

It's probably a hell to run it on Windows. But again, for the audience here, that's not expected to be a large roadblock.

It's actually really easy to run this stuff on Windows. I've run oogabooga, koboldcpp, tabbyapi, and more with no issues.

It builds on Windows with a couple tweaks to the includes with mingw64, I think some of the Windows-related includes need to be explicitly included.

This was a few months ago and things may be better now.

No they haven't. It's actually quite weird how the whole hype-cycle is working this time; I think tech journalists are so caught up in the drameh of OAI and whatever they haven't a clue that most people can run most of these novelty UCs on their own. You don't even need a speedy rig!

Ye it is strange. And in the tail around Sam Altman there seems to be these semi-religious lunatics believing that only some government mandated big tech monopoly can save mankind from the LLMs.

Meanwhile amateurs and uni researchers post really interesting work on a site named after a hugging emoji ...

I followed your instructions, and it works without a hitch. macOS-13.6 on an Intel iMac with a Radeon Pro 5700 GPU. My first direct contact with an LLM, and I don't understand most of the command line options yet, but it's certainly interesting.

A key difference is that for basic use cases, yes, this is a comparable experience.

Where it stops being comparable is general application across literally millions of use cases. The ChatGPT system has proven itself a valuable utility across industries, people, and use cases. No open model I know of can match it yet.

They don't have to match the market leader, they just have to be "good enough".

There are oodles of use-cases where sending your data to an outside provider is a complete no-go. In these cases OpenAI/Google/whoever-products aren't relevant competition.

Part of the problem is that "good enough" can be really difficult to figure out, whereas GPT-3.5 (and of course GPT-4) are almost a guaranteed success with just basic prompting and context fed in via the prompt.

And yes, there are indeed use cases where sending data to an outside provider is a no-go. The bet OpenAI is making is that they can solve for that later while building their business on use cases where it's fine to send data to an outside parameter. It may also simply not be something they care about. In my own work I know of a massive financial enterprise that has prioritized ~30 or so features where it's fine to send that data. OpenAI is not struggling to get their money.

It remains to be seen if OpenAI will also capture this market, or if fine-tuning open models to be "good enough" wins out over time. The point isn't that, though. The point is that their models are so broadly applicable that _anyone_ can get some value quickly without much work.

I keep hearing this, but which company has this policy? The privacy policy of openAI's enterprise(or Azure's) is not a lot different than say AWS, which everyone uses.

"everyone"? What are you talking about?

Have you ever worked in any sector that has security policies?

Even if you haven't, perhaps spend 2 minutes using a search engine?

Here is a first page result for you: https://www.tomshardware.com/news/samsung-fab-workers-leak-c...

Military, banking, health care all store their data on third party servers. It's the standard thing basically everywhere to do so.

Your own link shows Samsung using ChatGPT. Not sure what point you're trying to make with it.

Yes, they used it. Did you read any further?

"These actions clearly put confidential information at risk, prompting Samsung to warn its employees about the dangers of using ChatGPT. Samsung Electronics informed its executives and employees that data entered into ChatGPT is transmitted and stored on external servers, making it impossible for the company to retrieve it and increasing risks of confidential information leakage."

> Even if you haven't, perhaps spend 2 minutes using a search engine?

Obnoxious comment considering your link shows the opposite of what you're claiming.

Obnoxious? Yes.

Opposite of what I claim? No.

"Samsung Electronics informed its executives and employees that data entered into ChatGPT is transmitted and stored on external servers, making it impossible for the company to retrieve it and increasing risks of confidential information leakage."

> Haven't people realized what they can run themself?

Lots, lots of people do not care.

Never underestimate the value of usability. I appreciate being able to run these locally - I do! - but for a lot of people a website is going to be infinitely more convenient.

I’m reminded of the classic Dropbox HN comment (https://news.ycombinator.com/item?id=8863): why use Dropbox when any Linux user can just use curlftpfs?

This is generating answers a lot slower on my machine (X1 carbon) than GPT-3.5. :(

Why would I run it myself when I can run GPT-4 for less than the electricity cost? Running these yourself is not free. OpenAI is heavily subsidized.

Ye well at some point getting a sausage from Deliveroo or Uber Eats was a no-brainer since it was cheaper than in the actual fast food joint, due to some VC selling a dollar for pennies.

Should limit the stock price prospects though.

Would you say that this could run (smoothly) on my X220 with i7-2640M?

Imagine some old sci-fi movie where the computer write out the answer to a query, as if it actually was some scene worker typing it to some terminal. On this:

    NVIDIA GeForce GTX 1050 Ti
    Intel® Core™ i5-8300H × 8
    32,0 GiB ram

So I guess your computer would be a faster typist?

A few years ago peak HN comment was complaining about wget | sh.

Nowadays we just run full on opaque models directly from huggingface without thinking twice about reading anything. Interesting how times change. I wonder what supply chain attacks will come from huggingface, must not be long now.

That is, by the by, why models are no longer distributed as Python Pickle files, which can root your box if the model being loaded is malicious.

Reply on news.ycombinator.com