koljab

Didn't find many coqui finetunes too so far. I have David Attenborough and Snoop Dogg finetunes on my huggingface, quality is medium.

Coqui can to 17 languages. The problem with RealtimeVoiceChat repo is turn detection, the model I use to determine if a partial sentence indicates turn change is trained on english corpus only.

Yes, you're absolutely right. I'll provide UV and conda support soon, especially for Windows. I'm using python 3.10 still, maybe that's the issue. You can always mail me your current problem or log an issue, I really care about opening up the repo for as many ppl to use as possible.

That would be absolutely awesome. But I doubt it, since they released a shitty version of that amazing thing they put online. I feel they aren't planning to give us their top model soon.

Maybe possible, I did not look into that much for Coqui XTTS. What i know is that the quantized versions for Orpheus sound noticably worse. I feel audio models are quite sensitive to quantization.

LLM and TTS latency get's determined and logged at the start. It's around 220ms for the LLM returning the first synthesizable sentence fragment (depending on the length of the fragment, which is usually something between 3 and 10 words). Then around 80ms of TTS until the first audio chunk is delivered. STT with base.en you can neglect, it's under 5 ms, VAD same. Turn detection model also adds around 20 ms. I have zero clue if and how fast this runs on a Mac.

Lasinya voice is a XTTS 2.0.2 finetune I made with a self-created, synthesized dataset. I used https://github.com/daswer123/xtts-finetune-webui for training.

With the current 24b LLM model it's 24 GB. I have no clue how far down you can go with the GPU is using smaller models, you can set the model in server.py. Quite sure 16 GB will work but at some point it will probably fail.

Yeah I know the voice polarizes, I trained it for myself, so it's not an official release. You can change the voice here:

https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...

Create a subfolder in the app container: ./models/some_folder_name Copy the files from your desired voice into that folder: config.json, model.pth, vocab.json and speakers_xtts.pth (you can copy the speakers_xtts.pth from Lasinya, it's the same for every voice)

Then change the specific_model="Lasinya" line in audio_module.py into specific_model="some_folder_name".

If you change TTS_START_ENGINE to "kokoro" in server.py it's supposed to work, what does happen then? Can you post the log message?

Did not look into that one. Looks quite good, I will try that soon.

Yes, I tested it. I'm not that sure what they created there. It adds some noticable latency compared towards using raw websockets. Imho it's not supposed to, but it did it nevertheless in my tests.

That would be quite easy to integrate. RealtimeSTT already has wakeword support for both pvporcupine and openwakewords.

It's not aware. The information that it had been interrupted would be something we can easily add to the next user chat request. Where exactly is harder, because at least for Coqui XTTSv2 we don't have TTS wordstamps (we do have them for Kokoro though). So adding the information where it had been interrupted would be easily possible when using Kokoro as TTS system. With Coqui we'd need to add another transcription on the tts output including word timestamps. That would cost more compute than a normal transcription and word timestamps aren't perfectly accurate. Yet directly after an interruption there's not that much concurrent need for compute (like in the end of turn detection phase where a lot of stuff is happening). So I guess with a bit of programming work this could be integrated.

It's in fact using Silero via RealtimeSTT. RealtimeSTT tells when silence starts. Then a binary sentence classification model is used on the realtime transcription text which infers blazingly fast (10ms) and returns a probability between 0 and 1 indicating if the current spoken sentence is considered "complete". The turn detection component takes this information to calculate the silence waiting time until "turn is over".

This character prompt has undergone so many iterations with LLMs it's not funny anymore. "Make her act more bold." - "She again talked about her character description, prevent that!"

I doubt TTS will be fast enough for realtime without a Nvidia GPU

Dia is too slow, I need a time to first audio chunk of ~100 milliseconds. Also generations fail too often (artifacts etc)

All local models: - VAD: Webrtcvad (first fast check) followed by SileroVAD (high compute verification) - Transcription: base.en whisper (CTranslate2) - Turn Detection: KoljaB/SentenceFinishedClassification (selftrained BERT-model) - LLM: hf.co/bartowski/huihui-ai_Mistral-Small-24B-Instruct-2501-abliterated-GGUF:Q4_K_M (easily switchable) - TTS: Coqui XTTSv2, switchable to Kokoro or Orpheus (this one is slower)

Thanks a lot, great insights. Exactly the kind of feedback that I need to improve things further.

Actually good question.

I'd say probably not. You can't easily "unlearn" things from the model weights (and even if this alone doesn't help). You could retrain/finetune the model heavily on a single language but again that alone does not speed up inference.

To gain speed you'd have to bring the parameter count down and train the model from scratch with a single language only. That might work but it's also quite probable that it introduces other issues in the synthesis. In a perfect world the model would only use all that "free parameters" not used now for other languages for a better synthesis of that single trained language. Might be true to a certain degree, but it's not exactly how ai parameter scaling works.

That's a great question! My first implementation was interruption on voice activity after echo cancellation. It still had way too many false positives. I changed it to incoming realtime transcription as a trigger. That adds a bit of latency but that gets compensated by way better accuracy.

Edit: just realized the irony but it's really a good question lol

Tried that one. Quality is great but sometimes generations fail and it's rather slow. Also needs ~13 GB of VRAM, it's not my first choice for voice agents tbh.

This is coqui xttsv2 because it can be tuned to deliver the first token in under 100 ms. Gives the best balance between quality and speed currently imho. If it's only about quality I'd say there are better models out there.

I only tested it on my 4090 so far

I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to evaluate these latest models, they look strong.

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs.

Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8

The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and RealtimeTTS (supporting engines like Coqui XTTSv2/Kokoro) to achieve around 500ms response latency, even when running larger local models like a 24B Mistral fine-tune via Ollama.

Key aspects: Designed for local LLMs (Ollama primarily, OpenAI connector included). Interruptible conversation. Smart turn detection to avoid cutting the user off mid-thought. Dockerized setup available for easier dependency management.

It requires a decent CUDA-enabled GPU for good performance due to the STT/TTS models.

Would love to hear your feedback on the approach, performance, potential optimizations, or any features you think are essential for a good local voice AI experience.

The code is here: https://github.com/KoljaB/RealtimeVoiceChat