Hacker News Reader

I feel like you. Relatively simple formant models gets "close enough" that it feels like you should be able to do well with very little.

One of the things I've long wanted to do but not found time for, is to take a few different variants of formant synths, and try to train a simple TTS model to control one instead of producing "raw" output. It's amazing what TTS models can do with "raw" output, but we know our brains aren't producing raw, unconstrained digital audio, and so I think there's a lot of potential to understanding more and simplifying if you train models constrained to produce outputs we know ought to be sufficient, and push their size as far down as we can.

Too late to edit, but to any one who needs "convincing" of the flexibility of a formant synthesizer, you should 1) play with Pink Trombone[1], a Javascript formant synthesizer with a UI that lets you graphically manipulate a vocal tract, and 2) have a look at this programmable version of it[2]

[1] https://dood.al/pinktrombone/

[2] https://github.com/zakaton/Pink-Trombone

Thanks for those links, that's superb. Sounds surprisingly like the "Oh long Johnson" cat - https://www.youtube.com/watch?v=kkwiQmGWK4c

That reminds me of Google's blob opera: https://artsandculture.google.com/experiment/AAHWrq360NcGbw

What a fun toy! But I mostly get sounds of a drunk man who tries to say something but cannot.

Check out the videos in the second link - it gives some better examples of what you can do with it.

Funny that the Pink Trombone website does not work with screen readers.

It's entirely visual in that it is a graphic of a vocal tract where you can move the tongue and shape the mouth, so in that form trying to make it do anything with screen readers would mean basically doing what the second link does, and create an API for it and hook the backend up to an entirely different UI.

The videos on the other link does show that the guy who has done that API has done stuff to the UI to allow interfacing to it in non-graphical ways, but sadly I don't see any online demos of those alternative user interfaces anywhere, which is a great shame. Sadly it doesn't look like the videos are very accessible either, as they're mostly demos with no commentary about what is going on on the screen, without which they're mostly just random sounds.

Yeah, Pink Trombone is awesome[0]. :D

Thanks for the link to the programmable version--I don't think I'd been aware of that previously...

[0] And, from personal experience, also rather difficult to "safely" search for if you don't quite remember its name exactly. :D

It is possible to synthesize an English voice with a 1.5MB model using http://cmuflite.org/ or some of the Apple VoiceOver voices, which is just crazy to me. Most of the model is diphone samples for pairs of phonemes.

I don't know of any way to go smaller than that with software. I tried, but it seems like a fundamental limit for English.

I don't know of any way to go smaller than that with software. I tried, but it seems like a fundamental limit for English.

If you include "robotic" speech, then there's https://en.wikipedia.org/wiki/Software_Automatic_Mouth in a few tens of KB, and the demoscene has done similar in around 1/10th that. All formant synths, of course, not the sample-based ones that you're referring to.

SAM is awful but at the same time tantalisingly close (one of the demos of it apparently draws on a great reverse engineered version by Sebastian Macke and refactoring efforts by a couple of others including me - I spent too many hours listening to SAM output...) - especially when comparing to the still awful Festival/Flite models -, that I keep wanting to see what a better generic formant synth used as constraint on an ML model would produce.

That is, instead of allowing a generic machine learning model to output unconstrained audio, train it on the basis of letting it produce low bitrate input/control values for a formant synth instead, and see just how small you can push the model.

Me as a blind dude that is listening to a sinth all the time I don't cair about the size of the thing I just want good SQ.

That's fair, but those two are in constant tension. We can do very good speech if we ship gigs of samples. Better quality smaller models make it easier to get better quality speech in more places.

You probably could by what I call the "eastern-european method." Record one wave period of each phoneme, perhaps two for plosives, downsample to 8 or 11 kHz 8 bit, and repeat that recording on-the-fly enough times to make the right sound. If you're thinking "mod file", you're on the right track.

For phonetically simple languages, such a system can easily fit on a microcontroller with kilobytes of RAM and a slow CPU. English might require a little bit more on the text-to-phoneme stage, but you can definitely go far below 1MB.

That's effectively what eSpeak-ng is doing.

For the CMU flite voices they represent the data as LPC (linear predictive coding) data with residual remainder (residual excited LPC). The HTS models use simple neural networks to predict the waveforms -- IIRC, these are similar to RNNs.

The MBROLA models use OLA (overlapped add) to overlap small waveform samples. They also use diphone samples taken from midpoint to midpoint in order to create better phoneme transitions.

Most of the links there are dead.

That's a descendant of Festival Singer, which was well respected in its day.

What's a current practical text-to-speech system that's open source, local, and not huge?

Depending on your definition of "huge", you might find Piper TTS fits your requirements: https://github.com/rhasspy/piper

The size of the associated voice files varies but there are options that are under 100MB: https://huggingface.co/rhasspy/piper-voices/tree/main/en

Flite.

That's another spinoff of Festival Singer. What's good from the LLM era?

The speak & spell did it with a 32kB [edit, previously incorrectly wrote 16kB] ROM and a TMS0280.

I always wanted something like Flite but for Spanish.

Now try it with a trumpet! Herb Alpert's "The Trolley Song" is an underrated masterpiece of control over the sound a trumpet makes. No synthesized trumpet sound has ever done anything like this.

https://www.youtube.com/watch?v=mqr9E9Q-P5o

Haven't heard that one, it's great, thank you.

> Part of me feels that we are losing something, moving away from these classic approaches to AI

Absolutely. Seems a large amount of software developers have moved on from trying to understand how things work to solve the problem, and they are now instead just essentially throwing shit at a magical wall until something sticks long enough.

> It used to be that, to teach a machine how to [X], the designer of the system had to understand how [X] worked.

It does feel like we're rapidly losing this relationship in general. I think it's going to be a good thing overall for productivity and the advancement of mankind, but it definitely takes a lot of the humanity out of our collective accomplishments. I feel warm and fuzzy when a person goes on a quest to deeply understand a subject and then shares the fruits of their efforts with everyone else, but I don't feel like that when someone points at a subject and says "hey computer, become good at that" with similar end results.

I think it's going to be a good thing overall for productivity and the advancement of mankind, but it definitely takes a lot of the humanity out of our collective accomplishments.

I think AI will only cause us to become stuck in another local maximum, since not understanding how something works can only lead to imitation at best, and not inspiration.

I'm not convinced, because I think there will be a drive to distill down models and constrain them, and try to train models with access to "premade blocks" of functionality we know should help.

E.g. we know human voices can be produced well with formant synthesis because we know how the human vocal tract is shaped. So you can "give" a model a formant synth, and try to train smaller models outputting to it.

I think there's going to be a whole lot of research possibilities in placing constraints and training smaller models, and even training ensembles of models constrained in how they're interacting and their relative sizes to try to "force" extraction of functionality.

E.g. we have reasonable estimates at the lowest bitrate raw audio that produces passable voice. Now consider training two models A, B, where A => B => audio, and the "channel" between A and B is constrained to a small fraction of the bitrate that'd let A do all the work, and where the size of B is set at a level you've first struggled to get passable TTS output from.

Try to squeeze the bitrate and/or the size of B down and see if you can get something to emerge where analysing what happens in B is doable.

I've said this before on HN, but neural networks are not a tabula rasa. Their structure is created by people. With better domain understanding we can make better structures for AI models. It's not an either-or situation.

isn't the ultimate point of AI though, that instead of the traditional situation where the machines are the tools of humans, we become the (optional) tools for the AI? The AGI will do the understanding for us, and we'll profit by getting what we want from the black box, like a dog receiving industrially manufactured treats it cannot comprehend from its owner.

Quality of life improvements are much much more important than understandable models of speech, so we should live with, appreciate, and work to interpret and improve the current generation of complex neural TTS models.

I depend on TTS to overcome dyslexia, but I also struggle with auditory processing disorder that causes me to misunderstand words. As a result , classical TTS does not help me read faster or more accurately than struggling through my dyslexia. It causes me to rapidly fatigue, zone out, and rewind often, in a way that is more severe than when I sight read.

On the other hand, modern neural TTS is a huge enabler. My error rate, rewind rate, and fatigue are much better thanks to the natural tone, articulation, and prosody. I'm able to read for hours this way, and my productivity is higher than sight reading alone. This unlocks long and complex readings that I would never complete by sight reading alone, like papers in history, philosophy, and law. Previously I was limited to reading math, computer science, and engineering work, where I heavily depended on diagrams and math formulas to help me gloss over dense text readings.

The old tech had no impact on my life, given my combination of reading and listening difficulty, since it was not comparatively better than sight reading. But my life changed about 6 years ago with neural TTS. The improvement has been massive, and has helped me work with many non-technical readings that I would previously give up on.

The main issues I see now is not that neural models are hard to understand. For better or worse, we're able to improve the models just by throwing capital and ML PhDs at the problem. The problem I see is that the resulting technology is proprietary and not freely available for the people whose lives it would change.

We should work towards a future where people can depend on useful and free TTS that improves their quality of life. I don't think simple synthetic models will be enough. We must work to seize control of models that can provide the same quality of life improvements that new proprietary models can provide. And we must make these models free for everyone to use!

It's not at all a given that these two thing are in conflict. The best path towards free TTS might well turn out to be to identify ways of making smaller models that are cheaper to train and improve on if/when we can split out things we know to do (be it with separate neural models or other methods) and train models to "fill the gaps" instead of the entire end-to-end process.

There are also plenty of places where the current modern "neural" models are too compute intensive / costly to run, and so picking just the current big models isn't an option for all uses.

> The technique predates computers; it's basically the same technique used by the original voder [1] just under computer control.

Something similar from the 800s is the Euphonia talking machine ( https://en.m.wikipedia.org/wiki/Euphonia_(device) ).

* 1800s

Clicked that thinking someone had made a talking machine in the Middle Ages :)

Oops :)

Do you have any good resources on this?

I took a few stabs at understanding Klatt, but I feel like I had far too little DSP, math and linguistic intuitions back then to fully comprehend it, perhaps I should take another one now.