OpenVoice: Versatile instant voice cloning
research.myshell.ai
research.myshell.ai
Following the instructions in their repo (and moving the checkpoints/ and resources/ folder into the "nested" openvoice subfolder), I managed to get the Gradio demo running. Simple enough.
It appears to be quicker than XTTS2 on my machine (RTX 3090), and utilizes approximately 1.5GB of VRAM. The Gradio demo is limited to 200 characters, perhaps for resource usage concerns, but it seems to run at around 8x realtime (8 seconds of speech for about 1 second of processing time.)
EDIT: patched the Gradio demo for longer text; it's way faster than that. One minute of speech only took ~4 seconds to render. Default voice sample, reading this very comment: https://voca.ro/18JIHDs4vI1v I had to write out acronyms -- XTTS2 to "ex tee tee ess two", for example.
The voice clarity is better than XTTS2, too, but the speech can sound a bit stilted and, well, robotic/TTS-esque compared to it. The cloning consistency is definitely a step above XTTS2 in my experience -- XTTS2 would sometimes have random pitch shifts or plosives/babble in the middle of speech.
I was able to run the demos allright, but when trying to use another reference speaker (in demo_part1), the result doesn't sound at all like the source (it's just a random male voice).
I'm also trying to produce French output, using a reference audio file in French for the base speaker, and a text in French. This triggers an error in api.py line 75 that the source language is not accepted.
Indeed, in api.py line 45 the only two source languages allowed are English and Chineese; simply adding French to language_marks in api.py line 43 avoids errors but produces a weird/unintelligible result with a super heavy English accent and pronunciation.
I guess one would need to generate source_se again, and probably mess with config.json and checkpoint.pth as well, but I could not find instructions on how to do this...?
Edit -- tried again on https://app.myshell.ai/ The result sounds French alright, but still nothing like the original reference. It would be absolutely impossible to confuse one with the other, even for someone who didn't know the person very well.
But my use case is just to have a nice-sounding local TTS engine, and current text-to-phoneme conversion quirks aside, OpenVoice seems promising. It's fast, too.
I’ve noticed the same thing and I wonder if there is maybe some undocumented information about what makes a good voice sample for cloning, perhaps in terms of what you might call “phonemic inventory”. The reference sample seems really dense.
> Indeed, in api.py line 45 the only two source languages allowed are English and Chinese
If you look at the code, outside of what the model does it relies on the surrounding infrastructure converting the input text to the international phonetic alphabet (IPA) as part of the process, and only has that implemented for English and Mandarin (though cleaners.py has broken references to routines for Japanese and Korean.
Yeah, I could rent time on a server, but that's not as cool as just having a box in my house that I could use to play with local models. Feels like I'm missing a wave of fun stuff to experiment with, but hardware is expensive!
Imo the 4070 super is the best value and consumes the least amount of Watts, 220 in all the top 10.
So anything with one and some ECC RAM aka AMD should be fine. Intel non-xeons need the expensive w680 boards and very specific RAM per board.
ECC because you wrote server. We're professionals here after all, right?
If you use a mouse with more than right-click and scroll wheel, with side buttons for example, then you’ll need extra software.
Still waiting on being able to afford the next 4090 + egpu case et al. There are a lot of things this rig struggles with running OOM, even on inference with some of the more recent SD models.
Grok-1 was running on a M2 Ultra with 196GB of ram.
Has anyone had luck buying used GPUs, or is that something I should avoid?
And old mining GPUs are A-OK, generally: Despite warnings from the peanut gallery for over over a decade that mining ruins video cards, this has never really been the case. Profitable miners have always tended to treat these things very carefully, undervolt (and often, underclock) them, and pay attention to them so they could be run as cool and inexpensively as possible. Killing cards is bad for profits, so they aimed towards keeping them alive.
GPUs that were used for gaming are also OK, usually. They'll have fewer hours of hard[er] work on them, but will have more thermal cycles as gaming tends to be much more intermittent than continuous mining is.
The usual caveats apply as when buying anything else (used, "new", or whatever) from randos on teh Interwebz. (And fans eventually die, and so do thermal interfaces (pads and thermal compound), but those are all easily replaceable by anyone with a small toolkit and half a brain worth of wit.)
FWIW, I was in the same boat as you and decided to start cheap, old game machines can handle AI shenanigans just fine wirh the right GPU. I use a 2017 workstation (Zen1) and an Nvidia P40 from around the same time, which can be had for <$200 on ebay/Amazon. The P40 has 24GB VRAM, which is more than enough for a good chunk of quantized LLMs or diffusion models, and is in the same perf ballpark as the free Colab tensor hardware.
If you're just dipping your toes without committing, I'd recommend that route. The P40 is a data center card and expects higher airflow than desktop GPUs, so you probably have to buy a "blow kit" or 3D-print a fan shroud and ensure they fit inside your case. This will be another $30-$50. The bigger the fan, the quieter it can run. If you already have a high-end gamer PC/workstation from 2016, you can dive into local AI for $250 all-in.
Edit: didn't realize how cheap P40s now are! I bought mine a while back.
I guess you could give realistic voices to people that lost their voices by using old recordings, but there's no way that this is a market that justify the investment.
I can imagine a fairly big market of both people setting it up before they die, with maybe a whole load of written content and a schedule of when to have it read in future, and people who've just lost someone, and want to recreate their voice to help remember it.
I can't, and if I could, I think this would be fairly dystopian. Didn't black mirror have an episode about something similar? I vaguely remember an Asimov/Arthur C. Clark short story about the implications of time travel (ish) tech in a similar context. Sounds like a case of "we've build the torment nexus from classic sci-fi novel 'do not build the torment nexus'"
P.S.: "but what about small, indie creators" that's not who's gonna embrace this the most, it's big studios, and they will do it to fuck over workers.
Do I want to have people making exact clones of voice actors? No. The problem is that if you say "You can't get 90% close to an existing voice actor" then the technology will be able to create almost no human voices, it'll constantly refuse like gemini, even when the request is reasonable. This technology is incredibly powerful and useful, and we shouldn't avoid using it because it'll force a few people to change careers.
Also:
> This technology is incredibly powerful and useful
At what, exactly? The only "useful" case you presented is "actually, replacing voice actors with AI isn't so bad".
Edit: If you've got a cadre of volunteer voice actors that don't suck hidden somewhere, you need to share buddy. That's the only way your comments make sense.
Your vision of a world where anyone can create voice for their projects for cheap CAN NOT exist without someone getting exploited. Nor is it sustainable, really.
You said they this world would be worth some people losing their careers, but what do we gain? More games/audiobooks of questionable quality? Is this really worth fucking a whole profession over?
I really hope we can deprecate a whole bunch of professions related to fossil fuels, including coal miners and oil drillers etc.
I sympathise with the people working in those professions, I do, but times change and professions come and go, and I don't buy the argument that we should stop inventing new stuff because it might outcompete people.
As for positive uses of this technology, it might be used to immortalise a voice actor. For example Sir David Attenborough probably won't be around forever, but thanks to this technology, his iconic voice might be!
It is not unfair that, in order to have voice acting, you must have someone perform voice acting. You don't have the natural right to professional-level voice acting for free, nor do you need it to create beautiful things.
The tech is simply something that may be possible, and it has tradeoffs, and claiming that it's an accessibility problem does not grant you permission to ignore the tradeoffs.
I also don't have the natural right to work as a professional-level voice actor.
"Natural rights" aren't really a thing, the phrase is a thought-terminating cliché we use for the rhetorical purpose of saying something is good or bad without having to justify it further.
> The tech is simply something that may be possible, and it has tradeoffs, and claiming that it's an accessibility problem does not grant you permission to ignore the tradeoffs.
A few times as a kid, I heard the meme that the American constitution allows everything then tells you what's banned, the French one bans everything then tells you what's allowed, and the Soviet one tells you nothing and arrests you anyway.
It's not a very accurate meme, but still, "permission" is the wrong lens: it's allowed until it's illegal. You want it to be illegal to replace voice actors with synthetic voices, you need to campaign to make it so as this isn't the default. (Unlike with using novel tech for novel types of fraud, where fraud is already illegal and new tech doesn't change that).
Being rich to create a beautiful experience is neither required nor does it require a synthetic voice to achieve.
It does require effort and being rich can reduce that effort for sure.
Do you think Blizzard won't when the tech gets cheap and good enough?
I think the truth will be in the middle. Nobody's choosing AI over Brad Pitt or Henry Cavill, and in the various niches I think the highest performing humans will still do very well. Overwatch is a good example of that, see https://blizzardwatch.com/2018/01/24/rolling-stone-calls-ove.... AI is going to destroy the bottom of the barrel though, stuff like Fiverr and a lot of mediocre trained voice actors are going to have to get better or change careers.
Yes, we should avoid using it because it will upend the lives of a significant amount of artists for the primary benefit of "some indie games will have more voice acting and big game companies will be able to save money on voice actors". That's not worth it, how could you think it is?
Only tens of thousands? Cute. For most of the 2010s, I was expecting self-driving cars to imminently replace truck drivers, which is a few millions in the US alone and I think around 40-45 million worldwide. I still do expect AI to replace humans for driving, I just don't know how long it will take. (I definitely wasn't expecting "creative artistry" to be an easier problem than "don't crash a car", I didn't appreciate that nobody minds if even 90% of the hands have 6 fingers while everyone minds if a car merely equals humans by failing to stop in 1 of every (3.154e7 seconds per year * 1.4e9 vehicles / 30000 human driving fatalities per year ~= 1.47e+12) seconds of existence).
Almost every nation used to be around 90% farm workers, now it's like 1-5% (similar numbers to truckers) and even those are scared of automation; the immediate change was to factory jobs, but those too have shifted into service roles because of automation of the former, and the rest are scared of automation (and outsourcing).
Those service-sector roles? "Computer" used to be a job; Graphical artists are upset about Stable Diffusion; Anyone working with text, from Hollywood script writers to programmers to lawyers, is having to justify their own wages vs. an LLM (for now, most of us are winning this argument; but for how long?)
We get this wrong, it's going to be a disaster; we get it right, we're all living better the 0.1%.
> Indie games are hardly suffering for a lack of voice acting, even if you only look at it from a market perspective and ignore that voice acting is a creative interpretation and not simply reading the words the way the director wants.
I tried indie game development for a bit. I gave up with something like £1,000 in my best year. (You can probably double that to account for inflation since then).
This is because the indie game sector is also not suffering from a lack of developer talent, meaning there's a lot of competition that drives prices below the cost of living. Result? Hackathons where people compete for the fun of it, not for the end product. Those hackathons are free to say if they do or don't come with rules about GenAI; but in any case, they definitely come with no budget.
> Yes, we should avoid using it because it will upend the lives of a significant amount of artists for the primary benefit of "some indie games will have more voice acting and big game companies will be able to save money on voice actors". That's not worth it, how could you think it is?
A few hours ago I was in the Deutsches Technikmuseum; there's a Jacquard Loom by the cafe: https://technikmuseum.berlin/ausstellungen/dauerausstellunge...
The argument you give here is much the same argument used against that machine, back in the day: https://spectrum.ieee.org/the-jacquard-loom-a-driver-of-the-...
Why do you think those textile workers lost the argument?
And to pre-empt what I think is a really obvious counter, I would also add that the transition we face must be handled with care and courtesy to the economic fears — to all those who read my comment and think "and therefore this will be easy and we should embrace it, just dismiss the nay-sayers as the Luddites they are": why do you think Karl Marx wrote the Communist Manifesto?
Would you be happy with that outcome, or do you have another objection?
But some of this thinking is a bit like protesting the use of heavy machinery in roadbuilding/construction, because it displaces thousands of people with shovels. One difference with this type of technology is that the means to use it doesn't require massive amounts of capital like the heavy machinery example, so more of those shovel-weilders will be able to compete with those that are only bringing captial to the table.
It is creating a new and fully customisable voice actor that perfectly matches a creative vision.
To the extent that a skilled voice actor can already blend existing voices together to get, say, French Harrison Ford, for it to be evil for a machine to do it would require it to be evil for a human to do it.
Small indie creators have a budget of approximately nothing, this kind of thing would allow them to voice all NPCs in some game rather than just the main quest NPCs. (And that's true even in the absence of LLMs to generate the flavour text for the NPCs so they're not just repeating "…but then I took an arrow to the knee" as generic greeting #7 like AAA games from 2011).
Big studios may also use this for NPCs to the economic detriment of current voice actors, but I suspect this will be a tech which leads to "induced demand"[0] — though note that this can also turn out very badly and isn't always a good thing either: https://en.wikipedia.org/wiki/Cotton_gin
I can see it empowering solo creators in similar ways that modern music tools enable solo or small-budget musicians today.
That falls into “replacing voice actors”, mentioned by the OP.
That is as absurd as saying LLMs are increasing the demand for writers.
> because AI voices aren’t anywhere near being able to replace real voice actors.
Even if that were true—which it is not; the current crop is more than adequate to read long texts—it assumes the technology has reached its limit, which is equally absurd.
Or my favorite books in my own voice.
Or my lecture notes in my professor's voice.
The episodes are about 95% just a narrator with some background noises.
Elevenlabs did a great job with it and I cranked through the 32 episodes (about 4 mins each) relatively easily.
There is a longer series (about 60 hours) only in Japanese that I want to do the same thing for. But don't want to spend Elevenlabs prices to do.
iPhone Personal Voice [1] is one. It helps people who are physically losing their voice and the ones around them to still have their voice in a different way. Apple takes long voice samples of various texts for this though.
It appears from the outside that you've decided this is Officially Bad technology and aren't genuinely seeking evidence otherwise.
There's almost no reply here with a use that is a) not somewhat bad and b) has enough of an upside to compensate the downsides.
Except maybe this one, but I do know enough about accessibility to know how blind people generally use computers, which is why I asked the question.
It's almost as if any time some sort of way to make computers more human-like emerges, the first thing a subset of the men in the space do is think "How can I use this to make a woman who has absolutely no function other than my emotional, practical, and physical gratification?"
It's not just worth justifying investment. You can make just about anything worth the investment as measured by a 90-day window of fiscal reporting. H!tmen were a wildly profitable venture for La Cosa Nostra.
It's about not justifying the societal risk.
Personally, I'm eager to have more control over how my voice assistant sounds.
The range of use goes from totally harmless fun to downright evil.
My dad passed away a few months ago. Going through his things, I found all of his old papers and writings; they have great meaning to me. It would be so cool to have them as audio files, my dad as the narrator. And for shits, try it with a British accent.
This may not abate the concerns, but I'm sure good things will come too.
And possibly another one, but that would be a spoiler
Maybe a decade ago, I came across a cassette tape that he had used to record himself reading from a book for school - several minutes in duration.
It was incredibly surprising to me how much he sounded like my older brother. It was a very emotional experience, but personally, I can't imagine using that recording to bootstrap a model whereby I could produce more of his "voice".
It is healthier for us to just let others do as they wish with their time without passing judgement.
I'd like it if there were more (and non-american) voice options, but I don't think I'd ever want it to be my voice I'm hearing back.
This raises a lot of complicated issues and questions, but the use case isn't inherently bad.
In a similar way as machines allowed to produce abundance of food with less labor, the voice AI combined with AI translation can make information more accesible for the world. Voice actors wouldn't be able to voice act all the useful information in the world, (especially for the more niche topics and for the smaller languages) because it wouldn't worth to pay them and humans are also slower to than machines. We are not far from almost realtime voice translation from any language to any other one. Sure, we can do it with text-only translation, but voice makes it more accessible for lot of people. ( For example between 5–10% of the world has dislexya. )
If I am a content creator I can generate content more easily by letting my AI voice narrate my slides say. Yes that is cheap and lower quality than a real narrator who can deliver more effective real talks ...but there is a long tail of mediocre content on every topic. Who cares as long as I am having fun, sharing stuff, and not doing anything illegal or wrong.
I think the bulk of where this stuff will be useful isn't really visible yet b/c we haven't had the tech to play around with enough.
There is also certainly a huge swath of bad-actor stuff that this is good for. I feel like a lot of the problems with modern tech falls under the umbrella of "We're not collectively mature enough to handle this much power" and I wish there were a better solution for all of that.
Even that is not straightforward, unfortunately. There's this thing where the tech is going to be here, one way or the other. What we may have some influence on isn't whether it shows up, but who has it.
...which brings me to what I see as the core of contention between anyone conversing in this space. Who do you think is a bigger threat? Large multinational globocorps or individual fanatics, or someone else that might get their hands on this stuff?
From my perspective, we've gone a long time handing over control of "things"; society, tax dollars, armaments, law - to the larger centralized entities (globocorps and political parties and wall street and so on and so on). Things throughout this period have become incrementally worser and worser, and occasionally (here's looking at you, september '08) rapidly worser.
Put in short, huge centralized single-points-of-failure are the greater evil. "Terrorists", "Commies", "Malcontents" (whatever you wanna call folks with an axe to grind) make up a much lesser (but still present!) danger.
So that leaves us in a really awkward position, right? We have these things that (could) amount to digital nukes (or anything on a spectra towards such) and we're having this conversation about whether to keep going on them while everyone knows full well that on some level, we can't be trusted. It's not great and I'll be the first to admit that.
But, I'm much more concerned about the people with strike drones and billions of dollars of warchest having exclusive access to this stuff than I am about joey-mad-about-baloney having them.
Joey could do one-time damage to some system, or maybe fleece your grandma for her life savings (which is bad, and I'm not trying to minimize it).
Globocorp (which in this scenario could actually be a single incredibly rich dude with a swarm of AI to carry out his will) could institute a year-round propaganda machine that suppresses dissent while algorithmically targeting whoever it deems "dangerous" with strike drones and automated turrets. And we'd never hear about it. The 'media AI' could just decide not to tell us.
So yeah, I'm kinda on the side of "do it all, but do it in the open so's we can see it". Not ideal, but better than the alternatives AFAICT.
With game development as well, being able to be my own voice actor would save me an immense amount of money that I do not have and give me even more creative freedom and direction of exactly what I want.
It's not ready yet, but I do believe that it will come.
I would also like to give my children this as a novelty of preserved family history so if I so desire, I can have fun with them by letting them hear me from different ages.
Less-niche use-case: if you use TTS for voice-overs and/or NPC dialogue, there can still be a lot of variance in speech patterns / tone / inflections / etc when using a model where you've just customized parameters for each NPC -- using a voice-clone approach, upon first tests, seems like it might provide more long-term consistency.
Bonus: in a lot of voiced-over (non-J)RPGs, the main character is text-only (intentionally not voiced) because they're often intended to be a self-insert of the player (compared to JRPGs which typically have the player "embody" a more fleshed-out player with their own voice). If you really want to lean into self-insert patterns, you could have a player provide a short sample of their voice at the beginning of the game and use that for generating voice-overs for their player character's dialogue throughout the game.
Actually, getting it to sound "like myself" in the first place is an extra challenge! For many people even actual recordings sound "wrong", probably because your self-perception involves spoken sound being transmitted through your neck and head, with a different blend of frequencies.
After that is solved, there's still the problem of bystanders remarking: "Is that supposed to sound like you? It doesn't sound like you."
That's an insanely cool idea, and one I hadn't really considered before. Out of curiosity, how well did it work? Was it believable enough to fool players?
I used to work in the space and host read ads are the best performers compared to other types of audio ads. Imagine you can automate that ad creation as a podcaster/platform. You just give it the script and it creates a new ad in that voice.
Now, is this wrong? I think this is a separate discussion
https://huggingface.co/spaces/TTS-AGI/TTS-Arena
(Click the leaderboard tab at the top to see rankings)
It wouldn't surprise me if people recognize different algorithms and purposefully promote them over others, or alter the page source with a userscript to see the algorithm before listening and click the one they're trying to promote. Looking at the leaderboard, it's obvious there's manipulation going on, because Metavoice is highly ranked but generates absolutely terrible speech with extremely unnatural pauses.
Elevenlabs was scarily natural sounding and high quality; the best of the ones I listened to so far. Pheme's speech overall sounds really natural, but has terrible sound quality, which is probably why it's ranked so well. If Pheme could be higher quality audio, it'd probably match Elevenlabs.
https://github.com/collabora/whisperspeech
I found it to be kind of creepy hearing it in my own voice. I also tried a friend of mine who had a french canadian accent and strangely the output didn't have his accent.
1: https://en.wikipedia.org/wiki/File:Morgan_freeman_bbc_radio4...
https://github.com/myshell-ai/OpenVoice/blob/main/docs/USAGE...
I find it hard to understand why so much money is going into ai and so many startups are building ai stuff and such a product does not exist.
It’s got to run locally because I’m not interested in the restrictions that cloud voice cloning services impose.
Complete, consumer level local voice cloning = payment.
I've tried some of this ".ai" websites that do voice-cloning, and they tend to use the following dark strategy:
- Demand you create a cloud account before trying.
- Sometimes, demand you put your credit card before trying.
- Always: the product is crap. Sometimes it does voice-cloning sort of as advertised, but you have to wait for the training and the execution in queue, because cloud GPUs are expensive and they need to manage a queue because it's a cloud prouduct. At least that part could be avoided if they shipped a VST plugin one could run locally, even if it's restricted to NVidia GPUs[^2].
[^1]: To those who say "but the devs must get paid": yes. But subscriptions miss-align incentives, and some updates are simply not worth the minutes they cause in productivity lost while waiting for their shoehorned installation.
[^2]: Musicians and creative types are used to spend a lot in hardware and software, and there are inference GPUs which are cheaper than some sample libraries.
Professional consumer level software like a game or productivity app or something.
- Mental peace: if product X does what I need right now and I can count that I will be able to use product X five years from now to do the same thing, then I'm happy to pay a lump sum that I see as an investment. Even better, I feel confident that I can integrate product X in my workflows. I don't get that with a subscription product on the hands of a startup seeking product-market fit.
I can definitely make it a local app.
Out of the box it's not as good as Eleven Labs based on their demos, but those are likely cherry picked. There are some tunable parameters for the Bark model and most consider the output high enough quality to pass into something else that can do denoising.
People could pay somebody for the service of setting up the model on their own hardware, then use the model for non commercial usage.
You already don't need to pay; all of this is happening publication to implementation, open and local. Hop on Discord and ask a friendly neon-haired teen to set up TorToiSe or xTTS with cloning for you.
Software developers and startups didn't create AGI, a whole lot of scientists did. A majority of the services you're seeing are just repackaging and serving foundational work using tools already available to everyone.
It would be better for all if, in Open Source fashion, this software had a FLOSS easy-to-install packaging that provided for basic use cases, and developers made money by adapting it to more specific use cases and toolchains.
(This one is not FLOSS in the classic sense, of course. The above would be valid for MIT-licensed or GPL models).
If I had a need for an AI voice for a project I would pay the $9 a month, use it, and be done. I might have the skills to set this up on my machine but it would take me hours to get up to speed and get it going. It just wouldn't be worth it.
The other issue is that paying for the software once doesn’t capture as much of the value as a pay per use model, thus if you wanted to sell the software you’d either have to say you can only use it for personal use, or make it incredibly expensive to account for the fact that a competitor would just use it.
Suppose there were such a thing - then folks may complain that it’s not open source. Then it’s open sourced, but then there’s no need to pay.
In any case, if you’re willing to pay $1000 I’m sure many of us can whip something up for you. Single executable.
The product isn't exactly spectacular, but most of the works seems to have bene done. Just needs someone to go over the UI and make it less unstable, really.
Not even close. Perhaps I'm doing something wrong...
I don't recall their voices being the most descriptive but they had a lot. They also let layout a bunch of text & have different voices speak each line just like a movie script.
For reference, I'm transcribing research-related podcasts, meaning speech doesn't overlap a lot, which would be a problem for WhisperX from what I understand. There's also a lot of accents, which are straining on Whisper (though it's also doing well), but surely help WhisperX. It did have issues with figuring out the number of speakers on it's own, but that wasn't a problem for my use case.
Here’s an example for clarity:
1. AI is trained on the voice of a podcast host. As a side effect it now (presumably) has all the information it needs to replicate the voice
2. All the past podcasts can be processed with the AI comparing the detected voice against the known voice which leads to highly-accurate labelling of that person
3. Probably a nice side bonus: if two people with different registers are speaking over each other the AI could separate them out. “That’s clearly person A and the other one is clearly person C”
You pass N number of PCM frames through their trainer and once you reach a certain percentage you can extract an embedding you can save.
Then you can identify audio against the set of identified speakers and it will return percentage matches for each.
1. Encoding speech sounds into an IPA-like representation, decoding IPA-like into target language
2. Extracting "tone color", removing it from the IPA-like representation, then adding it back in into the target layer (emotion, accent, rhythm, pauses, intonation)
So as a result, I am a native English speaker, but I could hear "my" voice speaking Chinese with similar tone color to my own! I wonder, if I recorded it, and then did learn to speak Chinese fluently, how similar it would be? I also wonder whether there is some kind of "tone color translator" that is needed to translate the tone color markers of American English into the relevant ones for other languages, how does that work? Or is that already learned as part of the model?
Words on page are easier than still photos, which are easier than animation, which are easier than live-action TV, which are easier than IMAX movies etc.
If we move all of the rendering of the media into automation, then its just who can come up with the best story content, and you can render it whatever way you like: book, audiobook, animation, live action TV, web series, movie, miniseries, whatever you like.
Granted - the AI will come for us writers to, it already is in some cases. Then the Creator Economy itself will be consumed with eventually becoming 'who can meme the fastest' on an industrial scale for daily events on the one end, and who has taken the time to paint / playact / do rendering out in the real world.
But I sure would love to be able to make a movie out of my unpublished novel, and realistically today, that's impossible in my lifetime. Do I want the entire movie-making industry to die so I and others like me can have that power? No. But if the industry is going to die / change drastically anyways due to forces beyond my control, does that mean I'm not going to take advantage of the ability? Still no.
IDK. I don't have all the answers to this.
But yes, this (amazingly accurate voice cloner after a tiny clip?! wow) product is another step towards that brave new world.
Updated setup.py (mostly just bumping versions), and demo output: https://gist.github.com/duggan/63b7de9b5f6e8e74fe4b05af64dbe...
https://research.myshell.ai/open-voice/zero-shot-cross-lingu... I can only speak on their Dutch -> Chinese voice cloning but it's better than anything else I've tried. There is basically no "english/dutch accent" in the chinese at all. Where as the ElevenLabs Chinese voice(cloning or not) is so much worse...
Those are the real issues, it's a people problem really.
The voice cloning doesn't seem as high quality as other products I've used / seen demos for. Most of the examples match pitch well, but lose the "recognizable" aspect. The Elon one just doesn't sound like Elon, for example- interestingly the Australian accent sounds more like him.
Has anyone tried self hosting this software?
Training data bias?
Can anyone here give me a short list of relatively common ethical use cases for this technology. I'm trying to refute the ridiculous accusation that only deceptive, criminal minded people would have any use for this. Thanks.
----
Edit: Thanks for the many good-faith replies. So far I see these breaking down into;
Actual disability mitigation (of course I was joking about the circus accident). These are rare but valid. Who wouldn't want their _own_ restored.
Entertainment and games
Education and translation
No further comment on the ethics here, but FWIW I'm nervously looking at this having just written a study module on social engineering attacks, stalking, harassment and confidence tricks. :/
And yes, as bare tech, it is very cool!
Beside the fact that it seems more like a "entertainment" use case rather than "functional", is it really ethical to use someone's voice without asking/having rights to use it?
Small concern, granted, but parent seems to have specifically asked for ethical use cases.
Another might be for placeholdering - you could use an array of (licensed and used appropriately) voices to pitch a TV show, film, radio show, podcast, etc to give people a decent idea of how it would sound to get financing and hire actual people to make the real version. Ofc you'll need an answer to "why don't we just use these AI voices in the actual production?" from people trying to save a few quid.
Simple one- for fun. I'm considering AI cloning my voice and tinkering around until I find something useful to do with it. Maybe in my will I'll open source my vocal likeness as long as it's only to be used commercially as the voice of a spaceship's main computer or something. I'll be a sentence or two factoid in some Wikipedia article 300 years from now, haha.
Universal translator - if an AI can replicate my voice it could have me speak all sorts of languages in real-time.. sucks to be a human translator admittedly in this use case. Once the tech is fully ironed out and reliable we could potentially even get rid of "official" languages (eg you have to speak fluent English to be an airline pilot - heck of a learning curve on top of learning to be a pilot if you're from a country that doesn't teach English by default!)
I dunno if it'd be a weird uncanny valley thing, I wonder how an audiobook would sound reading a book in my own voice - unless I'm fully immersed in fiction that's generally how I take in a book, subvocalising with my voice in my head - maybe it'd help things bed in a bit better if it's my own voice reading it to me? If so I wonder how fast I could have AI-me read and still be able to take in the content with decent recall.. Might have to test this one!
Splintering off the audiobook idea - I wonder if you could help people untrain issues with their speaking in this manner? Like would hearing a non-stuttering version of their voice help someone with a stutter? I am purely in the land of hypothesis at this stage, but might be worth trying! Even if it doesn't help in that way, the person with a stutter would at least have a fallback voice if they're having a bad day of it :)
E: ooh, having an AI voice and pitch shifting it may help in training your voice to sound different, as you'd have something to aim for - "I knew I could do it because I heard it being done" sort of theory. The first example that popped into my head was someone transitioning between genders and wanting to adjust their voice to match the change.
I imagine there's other fields where this may be useful too - like if you wanted a BBC news job and need to soften out your accent (if they still require Received Pronunciation, idk)
Admittedly I could probably come up with more abuse cases than use cases if I put my mind to it, but wanted to stick to the assignment :)
Outside of that, more versatile text to speech is generally useful for blind people.
More emotional and non-robotic narration of text can also be useful non-blind people on the go.
Cloning a voice is a way to get lots of new voices to use as TTS.
I'm personally building a website with stories designed for language learners. I'd like to have a variety of realistic voices in many languages.
In general, though, I agree, the legitimate use cases for something like this seem relatively minor compared to the illegitimate use cases. However, the technology is here, and simply depriving every one of it isn't going to stop the scammers, as has already been evidenced. In my opinion, the best thing for us to do is to rapidly get to a place where everybody knows that you cannot trust the voice on the other end anymore, as it could be cloned. Fortunately, the best way to accomplish that is also the same way that we allow average people to benefit from the technology: make it widely available
Strongly agree with this. Sadly I don't think that transition to default distrust of voice will be rapid. We are wired at quite a low level to respond to voice emotionally. which bypasses our rational scepticism and vigilance. That's why this is a rather big win for the tricksters.
I'll probably end up just using OpenAI TTS since it's good enough, but if it could be my actual voice, I'd prefer that.