An LLM playground you can run on your laptop | Hacker News Reader

An LLM playground you can run on your laptop

github.com

268 pointsarchb56 comments

As an alternative for purely local LLMs, I've been having fun with this setup: https://github.com/oobabooga/text-generation-webui

The oobabooga setup feels a lot more mature and has a larger community. Skimming OP's repo, there seems like a lot of fiddling and faffing with JSON to get things running.

Nice stuff all the same.

Do you know what is the way to find models compatible with oobabooga/text-generation-webui ? I downloaded one with the included script and that worked, but if I try different ones it seems there are so many formats so no idea say how do I search huggingface or google to find the correct format. I would like to try this new quantized LLAMA versions with the GUI, I can run them in the CLI on the CPU but llama.cpp uses ggml formats .

https://rentry.org/nur779 (scroll down to the Huggingface ones)

Neat, eventually would like to run a purely local LLM Emacs shell https://github.com/xenodium/chatgpt-shell. For now ChatGPT only, but working on making more generic/reusable.

This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?

In principle you can use subprocess.run with llama.cpp (especially now that the mmap patch has landed, so model load time is negligible on subsequent runs) and then use stdin and stdout to interact with it. Multiple local sessions should work too. I’ve not tried it out yet... though I was looking for an afternoon project to work on!

Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python/pytorch/CUDA). (And I see that since last week it now has an HTML GUI too.) https://bellard.org/ts_server/

However, the GPU version is only available commercially. I'd like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp's author wrote "I expect LibNC [used by ts_server] will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap." [1])

EDIT: But if you wish, here's a Python interface to llama.cpp: https://github.com/PotatoSpudowski/fastLLaMa

[1] https://news.ycombinator.com/context?id=35195270

Textsynth is awesome, if you don’t want to use a fine tuned version of the models…far as I can tell, there’s no documentation on how to convert a fine tuned version of a model (like alpaca) to work with textsynth.

I did try llama 30b Q4 on textsynth/CPU. I did not measure, but TextSynth seemed faster than llama.cpp

There's a llama.cpp fork (I think?) with a build in HTTP server for an API.

I'm on mobile and can't find it right now though.

I have a short example in my recently published book [1] on downloading a HuggingFace model and using it locally with LangChain.

[1] https://leanpub.com/langchain

EDIT: GitHub repo https://github.com/mark-watson/langchain-book-examples

Yes, there are python bindings for llama.cpp and the text-generation-webui already uses them for local inference. https://github.com/oobabooga/text-generation-webui/wiki/llam...

"pip install llamacpp" or https://github.com/thomasantony/llamacpp-python

I think langchain lets you do this.

An LLM playground whose UI you can run on your laptop.

It supports local models.

I wonder how people that don't read properly before doing something will fare in a world of text interfaces/ai.

Better? Because the AI will help them?

Does it matter who generated the text people don't bother to read? And how will lack of attention to detail affect prompt generation?

They will answer "Please don't comment on whether someone read an article, please review https://news.ycombinator.com/newsguidelines.html (actually, a bad excuse, but it is what it is).

Are there any good local models? Gpt-2 is pants once you've gotten used to 4.

Llama, Alpaca, Dolly, Vicuna

I just tried llama (30B) and alpaca.

I'd say they are closer to GOT2.5

GPT2 is ancient news. There are now local running models which, allegedly, can reach the same performance as GPT-4. Look up llama.cpp[1] and all the various community generated models.

[1] https://github.com/ggerganov/llama.cpp

Very much allegedly. They’re good, but not great.

Does llama.cop not run on GPU?

It doesn't, it's CPU-only.

-Emily

llama.cpp is CPU only but llama runs on GPU using the HuggingFace Transformers library. You can run the llama model which far outpaces GPT-3.5 on a couple of $200 Tesla P40 GPUs at faster speeds than GPT-3.5 Turbo, completely locally.

Incredibly difficult to measure it, but it seems llama is closest to GPT-3[0].

Which is pretty cool all things considered. Certainly better than most of us would have expected a short while ago, right?

[0] https://arxiv.org/pdf/2302.13971.pdf

I don’t know about out of the box, but I’ve been having a great time training my own domain-specific models from scratch and utilizing them locally. It does seem that the more domain specific a task is, far fewer params are needed.

Having fun right now trying to build a model in GDELT, not much luck so far, but I’ve pushed less than 5% of the data through so far.

I’ve also been experimenting on fine-tuning Llama on my personal data archives, which seems promising but it’s pretty expensive to do so. Hoping someone will release a ~13B param model of Llama that they’ve trained with transfer learning from the 65B llama model and other data. FWIW even the 7B llama model running through llama.cpp after being quantized performs (subjectively, but substantially) better than GPT2.

This has been one of my most expensive, but also most rewarding, hobbies thus far.

What is the ballpark cost of this hobby, if I may ask?

Do you train exclusively in the cloud?

Really depends. I’ve pretty much almost burned through all the cloud credits I’ve been able to find at this point. Probably spent 20 grand over a year in credits? Never more than a couple hundred of my own money. Still trying to find a better solution. I have some 2080 TIs that I use locally. I tend to prototype architectures locally then rent out some A100s for a few hours to see how it goes, continue training when I can spare the cash.

I do a lot of CPU and sharded training too. I really wish there were better options for hobbyists to play around with this stuff.

got a quick guide as to available cloud credits for GPUs?

Training domain-specific models from scratch is both very difficult and also not possible in some cases. Sometimes we want to generate some niche content where there isn't enough training data to create a domain-specific model. We would have to somehow find a model that can produce the niche content through understanding what we're asking rather than through actually being trained on it. Producing "understanding" is not something you can do at home unless you have a bunch of A100s and have studied machine learning for years.

-Emily

Totally agree there. Even people that have years of ML under their belt may have a hard time there. And the cost (and scarcity) of those GPUs really are a major barrier to entry.

No, this is running LLMs on your laptop, although it can also connect to remote ones.

Eg: "Automatically detects local models in your HuggingFace cache, and lets you install new ones."

Llamma.cpp and HuggingFace models are all local.

My bad, I was wrong to miss the huggingface support.

Its amazing can run Alpaca llama 33B parameters, totally can handle japanese and korean where the earlier ones like 7B parameters could only do english (any other languages was horrible). All able to run on my M1 macbook.

How good is it with Japanese compared to GPT?

Dang, I don't have a laptop, can I also run it on my desktop...?

;-)

By laptop they mean you don't need a beefy gaming PC with 200 RGB lights.

Can it run Crysis?

GPT? No, but it can run for president.

Text only, it probably can and almost certainly could - it was tried on other scenarios (e.g. DnD).

...It is a good AGI test - trying the consistency in the "spacial" and temporal development of the construed world.

As long as it you take it and keep it on your laps while running this, should be fine.

Every desktop is a laptop if you're strong enough.

And every laptop is a desktop if you're one of those deskful types.

You can even upgrade it to 128GB cheaply nowadays and run the much bigger models (LLaMA 65B 16bit).

Awesome !! Great … I haven’t gone through it yet but from just what I saw on the GitHub page it would be awesome to have a CLI with standard arguments built into it for everything that can be done through the flask interface. Thanks !!

Why does it use port 5432 by default? That's the default port for PostgreSQL. Does this use PostgreSQL?

Awesome! Does it support safetensors, new ggml format, triton or cuda model files?

I added the playground to the GUI list here:

to https://github.com/underlines/awesome-marketing-datascience/...

Thanks for this! The compare feature is very cool, especially being able to play around with GPT4 settings (I'm still on the waiting list for GPT4 API so having access to this now is fantastic).

Someone with more inclination than I at the moment might be able to say something interesting about this being from Nat Friedman (former CEO of GitHub).

Good to see a CEO still coding

He wants my $5 though!

He didn’t build this. This is a repl.co bounty project he paid for.

Reply on news.ycombinator.com