It's cool tech, yes I'm impressed at the achievement. Nuclear weapons are impressive too.
OTOH this kind of thing is getting easier and easier to do, so what's a realistic way forward?
> The FTC Act’s prohibition on deceptive or unfair conduct can apply if you make, sell, or use a tool that is effectively designed to deceive – even if that’s not its intended or sole purpose.
It seems like an awfully broad rule? But they probably could go after this startup if they noticed it.
There are some kinds of businesses where making sure the regulators like what you’re doing is pretty much a prerequisite. On the other hand, plenty of companies got where they are today by pushing the limits.
[1] https://www.ftc.gov/business-guidance/blog/2023/03/chatbots-...
Wow. I would have imagined an article from the FTC to be more... Bland, for want of a better term.
Clear, direct, confident, not overloaded with qualifiers, not afraid of metaphor, self-summarizing, signposting, and most importantly it always has an energy of some kind that government communication (in seeking to appear neutral) regularly lacks - having that energy is why it doesn’t feel “bland”. I wonder if they have internal documents to guide their writers, or if it’s mostly information stored in the heads of Lesley Fair and Michael Atleson (who between them seem write most - all? - of the posts).
I truly do not understand people like these founders, obviously they understand the future they're creating. "If not us, someone else would do it" is not an excuse. Neither is "I like money".
Next, normal people will adopt the Mafia's 'cover the mouth while pretending to use a tooth-pick whilst talking to prevent lip reading from remote viewers (same thing sports people do currently.
-
My granmother was deaf for the latter half of her life. She became an expert lip reader.
It was fun going to restaurants with her as she would tell me what people at the tables far away were talking about "oh that couple isnt having a happy time..."
It seems a bit weird to me though. I mean, looking back at old records can still pass as mere nostalgic behavior. Wanting new sentences pronounced in disguise of lost relative voices is not great in term of respect for these people to share my own feelings.
Also I guess that now there is not much preventing completely new songs with whatever lyrics staring voices of Elvis, Hendrix and Pavarotti. Actually a continuous flow of on the fly generated lyrics seems perfectly plausible at this level, isn't it?
2. Requires internet
3. What image do you verify? Between auto-retouching, manual retouching, compression, filetype conversion, an image file might be invisibly transformed 10 times in between capture and Instagram upload.
4. Useless for disproving fake images until every camera manufacturer in the world has implemented this.
5. Hostile to customers, now your picture doesn't get the green verified badge or whatever if you decide to crop it or something.
This is a general problem that I have with a lot of blockchain ideas. For example, there are a few startups that claim to verify carbon offsets by registering them on the blockchain. There are many problems with this that we don't need to get into, but the relevant one here is: what is stopping me from registering the same offset on three different chains?
Pretty sure this is not a good use of blockchain, and I don't see how it would remotely avoid misuses.
This seems to be the only realistic future. This sort of technology literally makes it impossible to trust anything electronic.
People were worried about the balkanization of the internet, but now they look like optimists.
we said: "No, because there is also technology that makes it possible to trust anything electronic with very nearly 100% reliability. But no one uses it"
I think your first statement is both technically wrong and generally wrong. Electronic trust is a solved problem...it's just that right now, it's really not as big a deal as some people are worried about it being, so we haven't generally implemented the solution. We could make electronic cars for long time before other things made them commercially viable.
I disagree that electronic trust is a solved problem. It is mathematically solved, yes, but the reason that it isn't widely used is because it's still intrusive and painful to do. A solution that isn't acceptable to the masses isn't an effective solution.
If it could be done in a way that is invisible (like HTTPS, for instance), then it would be ubiquitous. That's the part of the problem space that still needs resolution.
I send you a bunch of bytes signed with my private key (which somehow you have to verify in a trusted way) and you can be sure that I am the person who signed those bytes (unless I was compromised).
USB etc hardware keys can be used protect accounts like Gmail, Coinbase, web hosting services, many more now.
Fun fact: it's possible to receive Facebook user notification emails encrypted against your public PGP key.
Example: https://blog.elevenlabs.io/enter-the-new-year-with-a-bang/
Funny how computing perfected communication and ultimately will undermine itself.
> I don't know what the way forward is for news.
I'd say every packet of voice/img will have to be signed by the recording device and checked at rendering time.
> I truly do not understand people like these founders
Me neither. Don't do it. We don't need that, and the malevolent use of this will confuse people to an extreme point.
Even the 'good' use of having a deceased relative utter new sentences is beyond strange. This is too far gone. And I'm no luddite.
Apply your same logic to any other easily misused tech:
"We must all have easy access to bio-engineered viruses. Otherwise only..."
"We all need to have access to nuclear weapons. Otherwise only..."
Not all tech should be in everyone's hands.
That is not possible with or comparable to, things such as bioweapons.
- We built and are offering for free a tool that can identify AI generated vs human-generated audio (https://play.ht/voice-classifier-detect-ai-voices/), we will continue to invest in this tool, and we hope it helps with deploying this technology safely.
- If we get any reports of a cloned voice without consent, we block the user and remove the voice instantly.
- The price of high-fidelity voice cloning is too high for scammers to use at scale; we have been live with it for four months and haven't had any cases of abuse so far.
Like any technology, it has the potential to be abused, and we are working hard to mitigate that and deploy it safely. We will continue to observe the use cases and user feedback and improve the safety of the service accordingly.
Since we launched voice cloning 4 months ago, we have seen enough genuine use cases which motivated us to keep moving forward and figure out safe ways to make the technology useful for all.
How do you know this?
This won't be the problem. My voice calling my parents asking for money to be sent to a random account will be the problem. And none of that will be sexual, offensive, racist, or threatening.
>we are working hard to mitigate that and deploy it safely.
How?
>we have seen enough genuine use cases
What?
This is exactly what makes me so angry about "AI safety" initiatives: they are largely worrying about the wrong thing. People have been so focused on the "this may make some obscene joke, or be biased against some skin colors" that they have completely missed out on the much more serious harms that AI will cause with respect to, in this case, impersonation scams.
Congrats, people can't say the N-word with your technology, but they can say "Hi Bob, just calling to verify that we did indeed change the target account where you should wire your invoice payment."
This makes for a rather explosive combination. I could, as some tobacco exec, hire some mimic to promote cigarette sales using a celebrity's distinct voice. By the time the regulators catch up, the spot has aired a few million times & made a potload of money.
A bunch of celebs[1][2] have trademarked their voice...but enforcement is spotty.
[1] https://economictimes.indiatimes.com/news/new-updates/amitab... [2] https://www.financialexpress.com/archive/when-celebrities-se...
>This is the kind of thing that should be illegal. Now, any Plebian could essentially write a letter to anyone, impersonating anyone. Forged letters could drag us into a war with Persia - for Jupiter's sake!
I suppose if we approach the point that we can create robotic clones of anyone, anywhere, that look, sound, and move like anyone on the planet, that will be just like the post office too, right?
The analogy for this technology would be a robot that can perfectly imitate someone's handwriting and vocabulary using one letter as a reference.
https://www.uspis.gov/history-spotlight/history-of-the-mail-...
Maybe we'll need a new specialized law enforcement agency like the Postal Inspectors to deal with the inevitable wave of AI-assisted crime.
Sure, this is marketed as generating your own voice to read scripts for you YouTube channel, but are they actually verifying who's voice you're generating?
Try to deceive people by learning about their contacts, writing a convincing letter and sending it. How long does it take you to prepare one letter?
Now those AIs potentially allow you to just generate millions of those with one click. The problem is the scale: everyone can do it at no cost and at scale.
Not true. Making it illegal wouldn't make it nonexistent -- that's true. But making it illegal would provide at least some method of mitigating some of the harm.
That's more than what we have right now.
> We have to accept that this tech exists and there will be both positive and negative outcomes from it.
Of course. But that doesn't mean it's futile to try to reduce the negative outcomes.
It's already illegal to impersonate someone to steal money or scam them, and those laws were on the books before computers existed.
> Of course. But that doesn't mean it's futile to try to reduce the negative outcomes.
You can run something on a consumer GPU and it's every bit as good if you know how to dial it in. By the end of the year you'll be able to download a nicely packaged "voice cloner" from a torrent that runs on a cheap laptop. IMHO any effort on regulation is far better spent informing people rather than trying to put the cat back in the bag.
I don't think so at all. There are all sorts of things you can technically do with ease that are illegal for good reason. Laws against them aren't futile.
But I admit that perhaps I'm being overly optimistic here. I'm just trying very hard to see any way that this stuff can end up not being a complete societal disaster.
There are two hurdles a criminal has to get past:
1. decide to break the law 2. figure out how to pull off their scam
It sounds like you're saying that since hurdle #1 already exists, hurdle #2 is irrelevant? No, of course it isn't. That's like saying that gun control can't possible help because it's already illegal to shoot someone.
Adding difficulty to a crime reduces (but does not eliminate) the prevalence of the crime.
Why should the ability to impersonate a persons voice suddenly become a crime in itself?
Should we arrest Jim Carrey?
Isn't it when the thing was used to do something else illegal when enforcement is required?
You could of course clone a voice to generate that "consent" -- but at that point there's no additional harm done because they'd already have the clone.
It's unrealistic that this tech won't exist somewhere, even if the big actors stay away for ethical reasons. A voice auth practice strikes me as a good compromise.
But this possibility renders the idea unviable in the first place, does it not?
Just like with image generation models, this will massively raise the bar for what amateurs can do with a limited budget and limited time. It's hard to justify spending thousands of dollars on voice acting and art for a hobby project, but now amateurs can get something that is 90% there and substitute professional work if the project takes off.
The reason is that if you speak with lots of verbal fillers, that's actually an important part of how you sound to other people. It makes sense to clean up audio for a podcast, but not for your great grandchildren.
A voice cloner doesn't care that you say "um" too much. It's parsing audio for phonemes.
You can use Descript, CleanVoice, and other tools to achieve exactly what you just said, in a few minutes, from just the original recording.
But yes, it would be weird to generate more stuff spoken by your father by using this technology. And beyond that, what's even the point? It's not your dad.
Yes in a few years you would be able to generate a complete avatar of someone, but it isn't them, and i think it will mess with you mentally.
I happen to be someone who believes that it's wonderful your dad left you with this artifact. It was a touching sentiment then, and now it can serve his obvious purpose many times over.
He didn't record himself as a side-effect of disease, or because he loved that particular story in the sound of his voice. He wanted people in the future to be able to hear what he sounded like!
Given that he could not have foreseen voice cloning (and therefore not explicitly asked for it) I cannot think of a more obvious example of someone wanting their voice to survive them.
I wish more folks would record The Night Before Christmas.
The difference between "the paper is out there" and "there's a button to do this" is quite obvious in cases like software exploits. A report of finding a vulnerability rarely leads to a massive automated exploitation campaign, but if that report also contains a proof of concept the amount of automated attacks radically increase. I believe the same is true for many other types of crime: even a mild bar to entry will prevent a significant amount of criminals from advancing their techniques.
I think the negative impact of these voice changers is much bigger than the advantage we gain as a society. Criminals will always exist, even crafty ones, but "we can't prevent crime so let's not bother trying to do anything about it" is not a great take in my opinion.
TOTC is about resource depletion. GYI. It's not applicable here.
whilst this is cheap and easy - increasing the potential for scams in big way - even to the point of automating the scam
1) The impersonation can be carried out in real-time by the criminal themselves. No need to employ anyone else. (No trail leading to them.)
2) Pro impersonators aren't common in society. They are limited as an asset and not duplicatable. So, using one cannot spread like wildfire and overwhelm our awareness that voice impersonation is something of a common risk.
Maybe the second could hold the first in check. I think disruptive tech like this & similar advances in visuals come with a societal impact that lessens potentials for realising the bigger fears. But people just love fears.
Compare:
1) Use lots of time to find person who can impersonate specific other person. Unless you give them a lot of money or threats to shut up you can't use them to real time decieve someone.
2) Clone 1 million voices from tiktok in 1 minute. Contact 10 million relatives with a synth voice that is intelligent enough to answer questions.
We will have billions of AI's, containers, programs, agents running around trying to deceive absolutely everyone and their grandmother 24/7 soon.
Of course a human could do that manually, but with those AIs it's a completely different scale, and it can be automated (so someone with no skills can click a button "generate 10k fake identities online").
Maybe even with one click, those techs could generate a fake coworker and send phishing e-mails. Suddenly every single e-mail you receive (or friend request or call) could be a very convincing fake. You don't have to be a high-value target anymore, it's all automated.
That makes it much more dangerous: from "can be forged manually with time and resources" to "everyone can do it at scale for free".
All's well that ends well, though. We simply don't have the resources to continue this "breakneck pace".
Yes, I agree, only criminals should be allowed to freely run it.
It became more apparent to me how icky this is as the voice actor of one of the most iconic characters in the game died suddenly 10 days ago...
I’m not actually sure whether that’s illegal (in the civil/IP sense), provided you’re not using the original voices to synthesise text.
We have these measures in place and are working on others to make sure the technology is used towards the betterment of humanity.
1/ Auto moderation on text to block harmful/malicious speech. 2/ As someone pointed out in the comments, we had a manual review process in place where the user is required to read out a consent and a member from Play.ht would review it before approving the voice. We're working on improving and adding this back. 3/ The user facing service is paywalled so we don't allow everyone in. 4/ Users trying to create malicious content are flagged and reviewed. 5/ A classifier to detect AI generated speech