ChatTTS-Best open source TTS Model
github.com
github.com
“ To limit the use of ChatTTS, we added a small amount of high-frequency noise during the training of the 40,000-hour model, and compressed the audio quality as much as possible using MP3 format, to prevent malicious actors from potentially using it for criminal purposes.”
I’m having a hard time understanding why they have degraded the training, audio, and thus the output. It’s not like this is the first or only text to speech system.
Still, here's a quick guide to getting it to work on Metal:
--requirements.txt additions--
torchvision==0.18.0
accelerate==0.30.1
--gpu_utils.py patch--
def select_device(min_memory = 2048):
logger = logging.getLogger(__name__)
if torch.backends.mps.is_available():
device = torch.device('mps')
return device
Edit: it also seems tho hallucinate/become increasingly unreliable with longer sentences.
Video with examples: https://b23.tv/uumKPam (bilibili)
In my mind that was the last hurdle to cross before being able to fool people regularly with a non-human voice.
Great work!
Hook that DSL into a prompt, [uv_break]gg. [laugh]
As pointed out by u/modeless, there is an established leaderboard and this model isn't on it (yet)
Robocalls -> I want to know I’m speaking to a robot
Audio books -> reasonable. An accurate tone is pleasant
Singing -> ever heard of vocaloids? They’ve existed for at least a decade or two
That it was technically already possible does not mean there isn't benefit from improved quality. In fact, Vocaloid itself has been improving and now uses AI.
Would also add making movies, podcasts, news broadcasts, etc. available automatically in a huge range of languages. You wouldn't want movies dubbed by Microsoft Sam (beyond initial comedic effect).
You'd be surprised how common something like this used to be in Poland, though admittedly we used an Ivona voice for this, which was a lot more pleasant.
Having a single narrator narrate the entire movie, overlaying the original audio track, is already common here, much more so than dubbing or subtitles. This is for historic reasons, in the communist era, obtaining the raw audio tracks for dubbing was often impossible, all the translators often had was a normal copy of the movie in its original language.
In the early 2000's, we had a lot of early / unofficial pirate releases, and they had to be translated into Polish somehow. Subtitles were certainly one method, but as we're all used to the single-narrator style, many people didn't mind listening to a somewhat decent synthetic voice instead.
Commercials/tutorials/corporate training videos -> Voiceover work
TV shows -> Dubbing in various languages
Fast food drive-throughs -> Taking customer orders
E.g. scamming. For anything that is just about conveying information through audio, like voice assistants, traditional TTS already works fine.
Most of us just need a machine to have the ability to speak to us in a way that sounds halfway human, but not as horrible as old open source TTS systems.
My current go to is Piper TTS: https://github.com/rhasspy/piper
It's MIT-licensed, supports ~30 languages and multiple voices[0]/quality/licenses.
Voice output samples: https://rhasspy.github.io/piper-samples/
Discovered just now that recently there's also been some efforts to train Public Domain/CC-BY licensed voices specifically for Piper TTS too: https://brycebeattie.com/files/tts/
---- footnotes ----
[0] e.g. One of the English voice sets has ~900 voices(!).
however if i am going to listen to 3 hours of audiobook, i would still pay for a better TTS versus Piper. (now i buy a lot of audiobooks read by voice actors but not every book i want to audio-read has a market to support a voice actor being paid to read it)
however Piper is definitely going in the correct direction though, huge progress.
Overall, this tech is a boon for the disability community.
A couple of years ago I started development of a tool to help with the generation of game audio such as NPC dialogue, "barks" or narration for those without access to/budget for human voice actors: https://rancidbacon.itch.io/dialogue-tool-for-larynx-text-to...
One thing I found interesting is that writing a small "scene" and then hearing dialogue being spoken by a variety of voices often prompted the writing of further lines of dialogue in response to perceived emotion contained in voices in the generated output. Plus it was just fun. :)
The version of the tool on that page is based on Larynx TTS which has continued development more recently as Piper TTS: https://github.com/rhasspy/piper
I'm yet to publish my port which uses Piper TTS though: https://gitlab.com/RancidBacon/larynx-dialogue/-/tree/featur...
Though I did upload some sample output (including some "radio announcer" samples in response to a HN comment :) ): https://rancidbacon.gitlab.io/piper-tts-demos/
Obviously there's variations in voice quality, and ability to control expression is currently limited but beats hearing my own voice. :D
>The Real-Time Factor (RTF) is around 0.65.
What is the state of real time tts models?
It seems to me that English TTS is already extremely good, even if you're looking at implementations that are far from being the best ones for English.
...and sometimes I wonder, if it's really economically efficient for that many players to compete on making English TTS yet another hair's breadth better than the next guy's, while TTS for languages other than English is this vast field of unmet market demand. At least these guys are doing Chinese, so: good for them.
Last time I looked into TTS systems for German, Google was the only game in town. What I wouldn't give for a viable alternative! It doesn't even need to be open source, I'd be quite ready to pay top dollar.
Will you still pay top dollar if it is open source though? :D
Piper TTS[0] (MIT Licensed; developed by main dev of Larynx TTS, Mimic3 TTS & Rhasspy voice assistant) has support for ~30 languages, at least some of which have multiple voices available--in a range of quality & data licenses.
And, particularly fortuitous for your needs, potentially, there's at at least one German voice that was recorded[1] specifically for Piper[2] (with emotion variants and CC0-licensed, no less :) )...
Check out `thorsten` & `thorsten_emotional` on the samples page: https://rhasspy.github.io/piper-samples/
I can't speak to the quality of the German voice specifically but for English at least I've found Piper's quality & range of voices of use[3].
---- footnotes ----
[0] https://github.com/rhasspy/piper
[1] https://www.youtube.com/playlist?list=PL19C7uchWZeojjI5FUk3q...
[2] In addition to other German voices based on other sources: https://huggingface.co/rhasspy/piper-voices/tree/main/de/de_...
[3] Somewhat of an understatement.
> Will you still pay top dollar if it is open source though? :D
There is one OS project that I do support financially. If I get commercial value out of any of the above, I imagine I would do the same.
Eleven Labs is still very far above open source models in quality. But StyleTTS2 (MIT license) is impressively good and quite fast. It'll be interesting to see where this new one ends up. The code-switching ability is quite interesting. Most open source TTS models are strictly one language per sentence, often one language per voice.
In general though, TTS as an isolated system is mostly a dead end IMO. The future is in multimodal end-to-end audio-to-audio (or anything-to-audio) models, as demonstrated by OpenAI with GPT-4o's voice mode (though I've been saying this since long before their demo: https://news.ycombinator.com/item?id=38339222). Text is very useful as training data but as a way to represent other modalities like audio or image data it is far too lossy.
The one that you can currently access in the ChatGPT app (with subscription) is the old one which is ASR->LLM->TTS.
Practically it doesn’t really matter, but I’d like to know for sure.
> Prior to GPT-4o, you could use Voice Mode to talk to ChatGPT with latencies of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4) on average. To achieve this, Voice Mode is a pipeline of three separate models: one simple model transcribes audio to text, GPT-3.5 or GPT-4 takes in text and outputs text, and a third simple model converts that text back to audio. This process means that the main source of intelligence, GPT-4, loses a lot of information—it can’t directly observe tone, multiple speakers, or background noises, and it can’t output laughter, singing, or express emotion.
> With GPT-4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.
Do you mean like as a simple text to speech application? There is a huge need for better quality audiobook output.
True human-level audiobook reading would require understanding the whole story, which often assumes general cultural knowledge, which you'll only get from a model trained on LLM-scale data. If you asked GPT-4o's new end-to-end voice mode to read an audiobook you'd probably get a better result than any TTS model. I bet it would even do different voices for the characters if you asked it to.
To the extent that authors provide supplemental notes or instructions to human actors reading their books, that information would be helpful to provide to an automated audiobook reading model. But there is no reason for it to be in a different form than a human actor would get. Additional structure or detail would be neither necessary nor helpful.
However, in my opinion it would be a huge benefit, if this kind of metadata would be put into the ebook file in some way, so that it would be something extractable and not has to be detected. I think it would be enough to ID the characters and tag a gender and a mood in the book together with citations, so that you could add different speech models for different characters. That would also allow to use different voices for different characters.
I wrote a little tool called voicebuilder (which I will open source next year). It's a "sentence splitter" which is able to extract an LJSpeech training dataset for an audio file, epub file and length matching. Works pretty accurate for now, although it needs manual polish of the extracted model. Still way faster than doing it manually.
This way you can build speech sets of your favorite narrators and although you would never be allowed to publish them, I think for private use they are great!
Voice acting is quite literally done a sentence or at most a paragraph at a time. Often the recording order is completely different from the script.
An actor may very well record his final scene on the first day of a project, after the whole character arc has transpired. But you know, acting. They get fed a line with stage direction and do a bunch of takes and somehow it works.
Heck you might be a full blown Italian who can't say a word in English but with the right kind of jacket it comes out a banger: https://www.youtube.com/watch?v=-VsmF9m_Nt8
You mention Eleven labs being ahead, check out Suno. There is no LLM-scale anything involved there. The voice in this context is a musical instrument and there are lots of viable ways to tackle this problem domain.
Sure, voice acting for games or movies is done piecemeal. But the actor still gets information about the story ahead of time to inform their acting, along with their general cultural knowledge as a human. Most crucially, when acting is done in this way it is done with a human director in the loop with deep knowledge of the story and a strong vision, coaching the actor as they record each line and selecting takes afterward. When the directing is done poorly, it is pretty easy to tell.
Sure, for a movie or game you could direct a TTS system line by line in the same way and select takes manually, but it would be labor intensive and not at all automatic. And to take human direction the model would need more than just the text as input. Either a special annotation language (requiring a bunch of engineering and special annotated training datasets), or preferably a general audio-to-audio model that can understand the same kind of direction a human voice actor gets.
It is, though, I've just done it a few days back. Once you have a clean text extracted from the book (this is actually the difficult part, removing headers, footers, footnote references, etc.), you can feed it into edge-tts (I recommend the Ava voice) and you get something that is, in my opinion, better than most human performers. Not perfect, but humans aren't either (I'm currently listening to a book performed by a human who pronounces GNU like G-N-U).
Inflection, emotion, tone, character-specific affects and all that can really change the audiobook experience for fiction. Your mentioning of footnote references and GNU suggests you're talking about nonfiction, perhaps technical books. For that, a voice that never significantly changes is fine and maybe even a good thing. For fiction it'd be a big step down from a professional human reader who understands the story and the characters' mental states.
You can find it on YouTube easily if you want an example.
> For fiction it'd be a big step down from a professional human reader who understands the story and the characters' mental states.
On the contrary, I don't want the reader to understand anything, I just want the text in audio form and I will do the interpretation of it myself.
I think how you tend to listen might also matter. I mostly use audiobooks when I'm driving or otherwise doing something else that is going to claim a portion of my attention. Following the narrative and dialog is easier when the audio provides cues like vocal tone changes for each speaker / narrator.
But it does seem like the Chinese version is better than the English one for this TTS, which would make the arena not quite as applicable to it as they're all focusing on English.
Works great in the car on speaker with the kids -- endless Harry Potter trivia, science questions, etc. I was completely blown away by the voice. Truly conversational.
This is not what was meant by "audio-to-audio" or "end-to-end". It's not a statement about the UI, it's a statement about the whole system.
> it may in fact be employing STT->LLM on the backend
It certainly is, and additionally TTS after the LLM, all connected solely by text. This is not audio-to-audio, it's audio-to-text-to-audio, and the components are trained separately, not end-to-end. ChatGPT has already done this too, for months.
See OpenAI's GPT-4 blog post: https://openai.com/index/hello-gpt-4o/
> Prior to GPT-4o, you could use Voice Mode to talk to ChatGPT with latencies of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4) on average. To achieve this, Voice Mode is a pipeline of three separate models: one simple model transcribes audio to text, GPT-3.5 or GPT-4 takes in text and outputs text, and a third simple model converts that text back to audio. This process means that the main source of intelligence, GPT-4, loses a lot of information—it can’t directly observe tone, multiple speakers, or background noises, and it can’t output laughter, singing, or express emotion.
> With GPT-4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.
Strictly speaking, this is not open source, as the commonly accepted definitions of open-source software include freedom of use and modification.
But in an industry where "OpenAI" is 100% proprietary, I guess "open-source" doesn't really mean much.
In addition, the Hugging Face repo[-1] states the license as "Creative Commons Attribution Non Commercial 4.0" (lacking the "ND").
Unfortunately, this combination of license imprecision and restrictiveness seems par for the course with a lot of academic TTS projects. (And, even for commercial "Open Source" TTS projects it's often the case that while code might be OSS licensed, none of the the voice data/models are.)
The current version[0] of the README repo states:
* "The open-source version on HuggingFace is a 40,000 hours pre trained model without SFT." (Presumably refers to model.)
* "At the same time, we have internally trained a detection model and plan to open-source it in the future." (Not directly relevant.)
The included "Roadmap" indicates related completed & uncompleted tasks:
* "[x] Open-source the 40k hour base model and spk_stats file"
* "[ ] Open-source VQ encoder and Lora training code"
* "[ ] Open-source the 40k hour version with multi-emotion control"
However, as noted, the current LICENSE[1] file states:
* "Attribution-NonCommercial-NoDerivatives 4.0 International"
And the README also contradicts the license:
* "This repo is for academic purposes only. It is intended for educational and research use, and should not be used for any commercial or legal purposes."
* "The information and data used in this repo, are for academic and research purposes only."
And this part of the "disclaimer" would make me concerned about potential licensing issues in regard to code and or data from other sources:
* "The data obtained from publicly available sources, and the authors do not claim any ownership or copyright over the data."
The code in the repo itself appears to have no license information contained within it.
My go-to actually Open Source licensed Text-To-Speech project (with a range of voice[2] model licenses[3]--including Public Domain & CC-BY[4]) is Piper TTS: https://github.com/rhasspy/piper
---- footnotes ----
[-1] https://huggingface.co/2Noise/ChatTTS
[0] https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676...
[1] https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676...
[2] Voice samples: https://rhasspy.github.io/piper-samples/
[3] Though I would also caution that (at least by my interpretation) some of the voices listed as CC0/PD or CC-BY also note that they've been "fine-tuned" on models which have more restrictive licenses and thus probably can't inherit the voice data's more permissive license.
[4] However, these Public Domain and/or CC-BY voices do appear specifically created to be data license-compliant: https://brycebeattie.com/files/tts/
I'll concede that the README is misleading but, as far as I can tell, it's not making any claims that the repo is open source. They may hint at it in an underhanded way, like having "open-source@20noise.com" as an email that you should contact them at, or make promises about open sourcing it "in the future", but they make no specific claims about their model being open source.
I also agree that corporations and many academics do a source of "open washing" by claiming "source available" is equivalent to "open source" but my point remains, this project doesn't actually claim to be open source.
And thanks for the link to piper. I do wish there were a list of "awesome (actual) open source tts/tts/misc." list somewhere, so, for those of us that care, can figure out where the actual open source models and data are.