Ava PLS: open-source application for running language models locally | Hacker News Reader

Ava PLS: open-source application for running language models locally

avapls.com

88 pointscztomsik24 comments

We changed the URL from https://tomsik.cz/posts/ava-oss/ to the project page, because it doesn't look like this project has been discussed on HN before. Readers might want to look at both links!

Actually, it was, here's the initial thread: https://news.ycombinator.com/item?id=37561085

The big news here is that it's now open-source: https://github.com/cztomsik/ava

I'll give it a try. I hope it's better than gpt4all, which I found out to be very buggy.

It should work fine if you're on Apple silicon, everything else is unfortunately second-class citizen, because I don't have any other devices and it's currently just me working on it. But if you'd like to maintain the windows, linux or intel-based mac releases, ping me on discord and I'd be happy to discuss and to accept PRs.

edit: [deleted]

Intel works in Ava, it's just slow, for some reason.

ah, got it. yeah the llama.cpp speeds I've seen are like 5-10 tokens/second on intel but I've been impressed with the relatively ancient hardware llama.cpp works on (2017 iMac with 8GB RAM at 5.5 t/s with a 7B model!).

it's ok, you can put the link back I don't mind the competition :)

yes, I know it definitely works, so it has to be some stupid mistake but it's impossible to reproduce it without actual intel mac :-/

I had enough trouble with supporting monterey, which also does not work 100% AFAIK

LM Studio works pretty well for me. https://lmstudio.ai/

LM Studio is great :) Also Faraday is great. But neither of them is open-source and this is going in a little different direction too (QuickTools, Workflows, there will be RAG and Vision support)

looks nice, congrats on open sourcing it cztomsik! Good to have another reference implementation, excited to dig through and check out how you're integrating llama.cpp and SwiftUI. I went with an architecture based on hitting examples/server on localhost.

Would you be interested in adding with otherbrain [0] to help build an open human feedback data set? Happy to help with PRs if so.

If you're interested in supporting open model training, we'd love to have you!

[0]: https://www.otherbrain.world/human-feedback

It goes a bit against the original idea, this should be 100% air-gapped. No phoning home, no data-collection, everything is private.

Makes sense, thanks for considering (and fwiw I'm also building with a private AI / offline philosophy). In my implementation, the compromise is 1) being super loud with an alert warning the user when they're sharing feedback and 2) providing a setting where the user can turn off feedback buttons. I had gotten user feedback around wanting to tell the AI when it was doing a good/bad job and this seemed like a privacy preserving compromise that could actually improve models. Anyway, good luck and congrats again!

Kudos for saying no. I can't believe we're living in an age where open source developers are being asked openly to leak the private conversations of their users to the public.

I couldn't get it working. Its stuck on Waiting for the model for all the models I downloaded.

How long did you wait, depending on the model size, it can take a while. Can you check logs? It's in the Settings -> System tab.

Which models do you currently support or have plans for? Looks cool tho

everything what llama.cpp supports

Any benefit over text generation webui?

It's small, all-in-one binary. You only need to download model, and you can use builtin downloader to do that.

Does an API exist that I may have overlooked?

In the video @ https://github.com/cztomsik/ava, I see "Workflows" but not in the app. What are these, and are they coming to the app?

Are quick tools basically the same as saving a playground prompt with variables?

Yes, it's at localhost:3002/api/ (and api/*.zig files respectively)

The Workflows screen is feature-flagged for now and only visible in the DEV build.

Yes, that's the whole idea behind QuickTools - to make repetitive tasks available quickly. It's also supposed to be in the main menu (drag-drop, reorderable, etc.) but I didn't have time to work on that yet.

LICENSE - MIT

https://github.com/cztomsik/ava/blob/main/LICENSE.md

In other words, the best family of licenses in the world

Reply on news.ycombinator.com