QuiLLMan: Voice chat with Vicuna-13B | Hacker News Reader

QuiLLMan: Voice chat with Vicuna-13B

github.com

174 pointspierremenard60 comments

Open articleView on HN

For "actually serverless" voice chat, check out https://whisper.ggerganov.com/

what is the point when the Web browsers mostly all already support it

https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_...

Engineering curiosity and fun

When you want much better, state of the art quality.

Compare it yourself.

The browser seems to do very well for me, and in multiple languages. What about this, can it do languages?

Whisper supports many different languages.

Whisper can do you one better: you don't need to know the spoken language ahead of time, and you can mix languages in the same sentence.

It does all sorts of things better like upgrading words into proper nouns, and doing it based on context.

Honestly it's the first time speech recognition finally feels "production ready" after years of Siri/Alexa.

In Chrome that's using Google's servers to transcribe the audio (last I checked) and it doesn't work on Firefox.

This is so good. I tried playing around with whisper.cpp last week and got absolutely terrible performance. I played around with the "step size" as well as "n" (not sure what these do, I should probably read the docs seriously).

I tried a lot of tweaks but this one is definitely the best I've seen.

How did it know how to spell my name correctly when I just spoke into my microphone??? The two L's usually trips up the transcription models. What????

>How did it know how to spell my name correctly when I just spoke into my microphone???

if the variations are pronounced the same? luck, probably.

My last name is pronounced "Collé" but when I've tried other whisper demos it always spelled it Cole or Coll.

This is the first time I've ever seen it get the spelling actually right (missing the accent is forgivable) ;)

After further testing, it seems like it is stochastic and I basically just got really lucky. Just wanted to post this for posterity. ¯\_(ツ)_/¯

wow, you weren't kidding. I tried the tiny model and it got what I said perfectly. Super impressive.

Hmm curious what the differences are, it’s the same person who wrote whisper cpp that hosts that website.

Really cool, I was able to play with it for a bit, but now seems like HN hug-of-death kicked in.

It's hard to death-hug these wasm demos, as they are all static files that can be easily served via CDNs

Which of the models did you use?

This isn't me, this is ggerganov's work. His demo is using gpt2, and whisper models (https://ggml.ggerganov.com/).

This link is the most worthwhile hijack of an HN thread that I've ever seen.

How is tiny.en so damned accurate?

How much faster does this run natively?

Do the NEON instructions work for arm devices like an RPI or is it just tuned for Apple?

With this and Alpaca 13B you could probably replace an entire window manager.

Edit: it seems like it was less than a year ago that local speech recognition was a slow slog through a swamp of crufty, complicated research projects with most the high quality training data hidden behind walled gardens. Now I can stutter-step over a word and a demo in my browser correctly transcribes what I meant sans stutter. What happened?

SIMD support in wasm is a bit weird. You'll need to compile into multiple targets, and many compile flags are not supported: https://emscripten.org/docs/porting/simd.html

Yes NEON works on recent Pis. RPi CPU tuning: https://github.com/ggerganov/whisper.cpp/blob/70567eff232773...

Been using textgen and downloading tons of models, the models are all over the place. The problems of accuracy and short term memory are major issues that people are trying to implement work arounds.

Check out textgen, it has voice in/out, graphics in/out, memory plugin, api, plugins, etc, all running locally.

https://github.com/oobabooga/text-generation-webui

> it has voice in/out,

Do you know how to get this working? I looked through the read-me and didn't see any options for it.

https://github.com/oobabooga/text-generation-webui/blob/main...

You need to enable the extensions.

I only did voice out locally with silero_tts, it also supports voice out with eleven labs api.

Voice input is via whisper tts.

Any models and/or plugins you recommend?

That's a pretty cool showcase of modal [1]. From a marketing perspective I have to congratulate, this is a really well done way to get people to check out your platform.

1: https://modal.com/

+1 modal.com is the first thing I checked after reading the readme.

+1, props to modal.com, well done and their site is nice

Is it basically Vercel for ML?

I also clicked through but immediately abandoned wondering:

What’s the state of container based ML deployments?

Can I take a container orchestration if these services and just put them on a vps w a GPU and run this?

Is there secret or just special sauce in ML infra?

Nice to see Tortoise being used - I still think it's the best TTS system out there now. Generation time is slow, but quality is incredible. I wonder if the code can be optimised to speed up the generation, but I don't think the author is maintaining it any longer.[0]

[0]https://github.com/neonbjb/tortoise-tts

You might want to check out Mimic 3 by the Mycroft team. Its also open source and runs offline.

https://mycroft.ai/mimic-3/

It took quite a bit of digging to find the repo link https://github.com/MycroftAI/mimic3#readme and it's AGPL-3 for those interested in such things

Tortoise looks really nice! The output is very "polished" and audiobook-like. It's a contrast to Bark[0] which is far more expressive but unpredictable.

[0]: https://github.com/suno-ai/bark

Coqui also looks interesting.

https://github.com/coqui-ai/TTS

Support for it was recently added to vocode:

https://github.com/vocodedev/vocode-python/pull/56

If you're generally interested in TTS check out Bondsynth AI, it's a product I've been working on for long-form text-to-speech (think ebook to audiobook or website to audio). It's still in beta, so it's free, but I'm looking for feedback.

Demo at: https://www.youtube.com/watch?v=OmQup3kst5s

Signup at: https://signup.bondsynth.ai

Very impressive demo! FYI https://signup.bondsynth.ai doesn't work, but https://bondsynth.ai/signup.html does

There's a lot of use cases just waiting for a good system that can do at least live speed generation.

I pitched this on a recently thread, but it was 12+ hours after it was posted, so I'll try again here.

What I really want is a program to waste the time of phone calls making unsolicited sales pitches.

It would do voice to text, run a simple language model to generate responses, then synthesize the voice back. It doesn't need to be a sophisticated model, not much more sophisticated than the classic "Eliza" program. A few years back someone did this with a canned loop of vague responses and it fooled the sales people for surprisingly long:

https://www.youtube.com/watch?v=XSoOrlh5i1k

It seems like it could all run locally for low latency. Probably the most important part to get right would be a TTS system that isn't immediately pegged as a robot.

Have you seen Lenny? It's a much lower tech solution that seems to work quite well.

https://lennytroll.com/

Yes, that was the youtube video I linked to. If that can be successful with some telemarketers, I'm sure a chat-gpt aided program could be even more successful at frustrating telemarketers.

just in case someone finds that YT video entertaining, you'll really love https://www.youtube.com/@KitbogaShow

Very cool - the demo was simple, functional and clear.

It was a bit laggy, but for a free demo from an open source project, I should be the one being shamed!

Well done.

Interesting. How does local development work or remote debugging work if the entire production toolchain is abstracted away with proprietary software?

This docs guide has some answers: https://modal.com/docs/guide/developing-debugging

Does anybody know if there is an easy-to-deploy setup for tortoise-tts only?

Yup, the code in the above repo is based on https://github.com/metavoicexyz/tortoise-tts-modal-api

Off tangent but can someone at Apple please just replace the Siri word recognition with whisper. We can finally have multi language support and not dogshit recognition.

Sorry if this is off-topic, but those are some really good book recommendations in the demonstration image! If those are coming from Vicuna, that speaks well of it.

How about a self hosted version?

Why is it serverless? It clearly has an API server.

The "API server" only runs when requests come in. Otherwise Modal scales everything down to 0, and you don't pay for anything. It's similar to putting Lambda behind an API gateway.

Because the resources are on demand, and the provider takes care of all the server management.

https://en.wikipedia.org/wiki/Serverless_computing

I think serverless just means that you hire less devops people for your specific team.

It’s a cost-saving measure that costs you mortal wounds in integration expense, and eventually building out a real infrastructure plan.

Is this the first time you've heard the term serverless? It means your code is plug & play, you don't have to manage the infra (I.e. a server) yourself.

I know the term serverless - but it was fairly ambiguous in this context.

I presumed the same as the person you're replying to.

I understand the concept of serverless as in AWS Lambda, but my first impression reading the title is serverless as in no backend. That would be a lot more amazing.

The word you're looking for is "unhosted"

It should be serverless but Amazon ruined it.

As Michael Bolton said in Office Space: "Why should I change my name? He's the one who sucks."

"serverless" is a throwback to 2013 a decade ago when it was a novelty to not know where your servers were or what they were made of, some people never move on. but in this case, this is a company advertising their serverless architecture for that specific crowd.

this is distinct from "client side" where no servers are involved, aside from maybe something that serves a static web page.

Reply on news.ycombinator.com