Localllm lets you develop gen AI apps on local CPUs | Hacker News Reader

Localllm lets you develop gen AI apps on local CPUs

cloud.google.com

101 pointssrameshc74 comments

Even Google is making a llama.cpp wrapper.

Technically a wrapper of a wrapper, though llama-cpp-python is quite excellent... TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary).

Slightly off topic, here is the best local llama.cpp wrapper I've run into:

https://github.com/Mozilla-Ocho/llamafile

You can download any .gguf model (not just the ones in their examples) and run it locally (as long as you have the ram for it). I was running 7B models with ease on an old FX8350 and now 13B models on a 5600X (32GB RAM on both machines).

This wrapper spins up a local web server that runs a simple web frontend to use immediately with no code, but also exposes an OpenAI compatible API for dev work and alt frontends (like SillyTavern).

What's the speed? I heard of llamafile before.

I wasn't really timing it, but it felt fine, especially since it prints out while generating. The 13B model on the 5600X is about 10 seconds. Tho once the conversation gets too long (about 10 replies) it will go sideways. I gotta play with the default settings. IIRC the 7B on the Fx8350 was closer to 25 seconds per response.

Oh it's a wrapper? I just skimmed and thought they reinvented llama.cpp for some reason, which would have been mildly interesting.

(edit: this is completely irrelevant remark. sorry) It's (1) written in Python, and (2) ollama appears nowhere in their Docker repo [1], and (3) it gets the models only from huggingface instead of the ollama repos. Thus maybe it is NOT a wrapper of ollama?

[1] https://github.com/googlecloudplatform/localllm

https://github.com/GoogleCloudPlatform/localllm/blob/main/ll...

I see llama.cpp in there, but I didn't go deeper than that. Ollama is not the same thing.

I wonder what need they're fulfilling here. It's not really more useful than using llamacpp-python. This is not something old Google would even release.

> I wonder what need they're fulfilling here.

The need to fill more seats on the AI hype train.

(Someone's probably got "release AI software as Google open source" as an OKR)

does Functionary work with generic models like llama.cpp? had the impression you could only use Functionary GGUFs

This question is confusing to me.

Functionary is a series of function calling llama architecture finetuned models, not a new base model, if thats what you mean.

Hence, it can be turned into a gguf like any other llama finetune.

It just has a very esoteric prompting format, but llama-cpp-python is specifically set up to leverage it.

from the parent:

> TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary).

I never heard about Functionary, this comment seems to say “I would use llama.cpp plus Functionary”. Now, since llama.cpp allows to plug in several models, I was wondering if Functionary works with several models too, ie is Functionary a library that allows to add function calling to models?

The following answer to my comment helped clarifying that Functionary is not a lib, but a specific model, or that is what I understand from the answer. So the answer is “no, you can’t use functionary with many models (eg it’s not like llama.cpp that works with multiple models) because Functionary is a model itself”

:-)

Llama.cpp is an inference engine. The author of llama.cpp designed gguf. Funcionary is a model that does function calling. You can download functionary weights in the gguf format and then run it using llama.cpp on low-end machines using CPU or GPU or a mix of both.

ok so the answer is no, thanks

The GP educated you and you still said the answer was "no"? It's clearly "yes".

I read the original question as "is the 'Functionary' thing something that works with any LLM you can run in llama.cpp, not just only with the special 'Functionary' finetuned models?", to which the answer now is clearly no.

Very interesting question, as I also read top-level post as saying it's a llama-cpp-python feature, and therefore I could shove some random dolphin-mixtral or whatever I have on my hard drive at it, and it will work.

A package called localllm (three l's in the middle, nice) which is focused on running models in GCP.

> run LLMs locally on CPU and memory, right within the Google Cloud Workstation

And in the code https://github.com/GoogleCloudPlatform/localllm/blob/main/qu... there's just a shebang halfway down?

    import sys

    
    def main(argv):
        pass


    if __name__ == '__main__':
        main(sys.argv)
    #!/usr/bin/env python3

This is awful. Did bard write this?

I'm not thrilled about https://github.com/GoogleCloudPlatform/localllm/blob/main/ll... calling their Python package "llm" and installing "llm" as a CLI command, when my similar https://llm.datasette.io/ project has that namespace reserved on PyPI already: https://pypi.org/project/llm/

I'm really pissed off at Google for doing this:

- First, they "steal" @simonw's 'llm' keyword for their own CLI tool (https://news.ycombinator.com/item?id=39296567)

- Then they create a wrapper around llama-cpp-python which is a wrapper around llama-cpp, a project they don't even credit nor support.

- Then they steal @mcapodici's article title to publish it on their blog (https://news.ycombinator.com/item?id=39297563).

- And the ironic part is that their Gemini models are closed-source, so no gguf for those.

Google trying to ride the open source hype that Meta is blessed by, except google is a mean girl who regrets opening up the transformer without taxing it, so they're really sour about it and doing these tricks now to pretend to still support open-source.

"No GPU? No Problem." Isn't exactly a complex phrase they'd have to steal, seems much more likely they came up with it independently.

May I ask why you linked to your parents comment as a source? Just seems.. odd

I actually wish that no one took the llm namespace. It's annoyingly vague.

I think it's a pretty great name for my tool, given that it lets you interact with dozens of different LLMs via plugins.

Plus I like that it's three letters, since it's designed for CLI usage.

But your tool isn't an llm. llmt for e.g. llm tool could be better

Yes - at the same time it would similarly fit other tools where some form of working with LLMs is the main feature. And with time we might get more of these.

At least it’s not called “ai”, we have that already, e.g. https://github.com/yufeikang/ai-cli

Sure, but it’s not like you own the name LLM.

I mean, let’s say you picked say, “ghost pony”; ok, sure, it’s obviously a name.

But LLM? I mean, it’s not like you can get a lot more generic than that. If your tools name is “tool” and some else makes a tool called “tool”, you can’t really complain that much…

This is kinda how I feel about HTTPie taking using 'http' in the CLI.

I like what Go does (I know, the irony). Buy a domain name, that's your namespace.

Almost every package is vague if you're just going off names of packages. Is 'pandas' some kind of screensaver showcasing the cute animals?

If the name is available, someone will take it, and I can think of no better steward of it than Simon

I thought exactly the same - it actually drives me nuts that Python packages have this confusing dual-name nature. You were there first!

This happened with one of my neovim plugins (also named llm!). In my case I was already thinking of renaming the project so I just went through with that. The other project's maintainer was nice enough to apologize and I ended up with a name I liked better.

https://github.com/gsuuon/model.nvim/discussions/37

Yeah, this feels super reactive. Like someone pitched an idea for how to convert AI enthusiasts to paid google AI suite...

I want to know: are they contributing back to llamacpp? Are they paying GG?

LLM is now an ubiquitous term, not sure anyone can claim ownership of the keyword. It also looks like the package is a local install only, not pushed to PyPI. So unless you want to use this project, you'd never know.

You should rename your project.

Missed /s

> We suggest using a machine type of e2-standard-32 (32 vCPU, 16 core and 128 GB memory), an admittedly beefy machine.

Average price per month $966.56

https://gcloud-compute.com/e2-standard-32.html

You could buy 4-6 machines like that per year at that price. Gives you a taste of cloud mark-up. Bandwidth markup is far more insane.

For $1000 you could basically build one today. 5950x (16c cpu) is $400, 128gb kit of ddr4 is $250-300, maybe enough budget left for mobo, ssd, psu, case, cooler if you bought used bits (may also need cheapest gpu on Craigslist depending on your needs). All new ~$1200. Aws/gcp prices are nuts compared to on prem, although hetzner and co would be much cheaper.

Yeah, Cloud is definitely a money maker.

If someone has a microcenter nearby, they could build an 8 core ddr4 system with 128gb of ram for maybe $800.

$3200+ could build a 64 core/128gb ddr5 rig.

https://www.newegg.com/tyan-s8050gm2ne-9534-amd-epyc-9534-2-...

Price a HP z440 with those specs on eBay or Amazon.

Yep, I built gaming PCs for years, and I have a few laying around.. Add some ram and they a great servers (without memory safety though).

Without memory safety? Do you mean ECC?

Non-ECC issues are statistically rare. You might never see them if you aren’t running a lot of systems at heavy load at scale.

Yeah, ecc. I do have a lot of data intensive services running. But my systems are mostly amd, and I'm pretty sure my amd CPUs can actually use ecc memory, so I'll probably upgrade some day.

I've run non-ECC systems at heavy load for uptimes in excess of a year and not had issues. It really is statistically rare... but of course common enough that if you run whole data centers you will see it.

You can also get exactly that same VM as a spot instance with a 10GB disk and external ip address in the us-west4 region for $142.04/month. (Spot instance prices vary dramatically from one region to another, but that one is very discounted in us-west4.)

Also, is this localllm kind of like a Google competitor to some aspects of ollama?

One thing I'm happy about is that the gguf file format is getting widely used, here endorsed by Google. I vastly prefer working with gguf and have adopted it for my project, hopefully it gets the critical mass to become a "native" format and we can stop with pytorch checkpoints.

On the contrary, it's not one-size-fits-all.

It doesn't support everything other backends need (like certain quantization schemes). If it somehow does, then its not universally compatible and that's just going to create more confusion.

It supports grammar and that's more than enough in practice.

Grammar has nothing to do with gguf, as it's logit processing (and that is being integrated into other backends as well).

How can you call something “local”LLM and have it rely on Google cloud?!

> “…within the Google Cloud Workstation”

Local from _their_ perspective, deep inside the beast.

Exactly my thought.

Hey that's my title ... almost :-) https://martincapodici.com/2023/07/15/no-local-gpu-no-proble...

hvjvhcgxgxggcgcgcgcgcywww.@aliyun.com

The idea of making it easier to download modes from HuggingFace, and serve them via llama-cpp-python could be a good contribution, but unfortunately there are two different names that the "Localllm" project introduces for itself - Localllm and the 'llm' command, both of them problematic for quite a few of reasons, perhaps they could just drop naming it anything to do with local? E.g: Simon Willison's well known `llm` package on PyPI already uses the `llm` command. A name for the command that differs from the project name creates extra work for no good reason.

Look at this positioning Google use in the blog post: "localllm combined with Cloud Workstations revolutionizes AI-driven application development by letting you use LLMs locally on CPU and memory within the Google Cloud environment." - implying a local setup that's actually reliant on cloud resources is a contradiction.

Maybe they could rename it with a single name that has no conflicts or misleading terms that makes it clear it's to help setup and run inference without confusing the matter about how local things might sometimes be. It won't be nice to read the scathing critiques in some of the other comments! Maybe they could fix up the mess in querylocal.py!

Being able to run the same code locally as in a cloud platform is a nice development pattern but optimal inference setup for many ML situations is a tricky problem, are you really making good use of the CPU / GPU / TPU?

The underlying server tech: llama-cpp-python recently added support for draft models that could make inference much faster. How easy would using that from Localllm be? I'd be concerned `Localllm > llama-cpp-python > llama.cpp` might be too many layers of systems to optimise or code with. Milliseconds often matter.

Having a fast, well priced solution is not easy, people want as many tokens per second as they can get, ideally with no warm-up. How would they solve that?

People looking at fast inference options might like to see Banana's parting words, as they wind down their serverless GPU cloud, in their swansong, they suggest a few hosted inference options people might want to consider: https://news.ycombinator.com/item?id=39288915

For the interested, I've been having fun with llama.cpp. I've got it running well, if a bit slowly, on my 2019 intel MacBook Pro. There's configs you can tweak that get it to run faster, mainly dealing with memory mapping. I've been working on documenting my findings but haven't published yet.

I say all that to encourage people to try it out with different models. It's not production quality but totally usable for testing things out even on older machines.

As a former Google employee working on their OSS stack, this feels a bit off brand.

Here's a whole list of indie, OSS AI frameworks you can use: https://github.com/janhq/awesome-local-ai

Disclaimer: I'm one of the core devs on Jan.

The amount of obfuscation & head-in the-ground was nauseating to read

Looks like the repo (localllm) only contains a smallish Python project to do the model downloading. When it comes to serving the model, it calls out a Python module llama_cpp.server, which is not part of the repo, likely something inside the Docker image. A bit disappointing given how the article sounds.

This blog post has a distinct "written by an LLM" feel to it that I can't quite put my finger on.

Bullet points starting with bolded text, longer, objective, soulless sentences devoided of "I".

> pip3 install openai

Why is that required?

Everyone uses that for openai compatible api requests.

Its actually great. Most backends and frontends "just work" with each other because they all talk with openai (albeit with some caveats).

I think you might be thinking of litellm? OpenAI is for OpenAI.com (although you can override the base URL it’s far from ideal).

What do you mean "far from ideal"? It's literally just passing in your services URL rather than using the default when instantiating the client. Something you will need to do with any client you use.

I wondered that too. I see a comment, but I still wonder why its "required" here.

Google really doesn’t seem to have much of a coherent plan on AI by the looks of it

They've really jumped the shark. Creating a wrapper for llama.cpp (a project they don't even credit here) and acting like that's even interesting, then somehow implying this is a feature or benefit of using Google Cloud Workstations? This is the most technically bankrupt product page I've ever seen in my life and for a company that used to mean something to people to do it is pathetic.

it's actually a wrapper around llama-cpp-python. they ought to be embarrassed.

Google really has fallen off. This is intern level stuff.

https://www.sfgate.com/tech/article/google-workers-company-c...

What an awfully misleading name. This doesn't run locally, it runs on the cloud. What the fuck was that guy thinking?

Reply on news.ycombinator.com