OpenAI deems its voice cloning tool too risky for general release | Hacker News Reader

OpenAI deems its voice cloning tool too risky for general release

theguardian.com

33 pointspg_123445 comments

pretty convinced at this point these announcements are more about building buzz for openai then any real concern for humanity

"My models are too strong for you, general public. You better find an AI lab that produces weaker models"

"OpenAI, I'm going into the stock market, and I need your strongest models."

Yep they made the exact same claim about GPT-2. Back then language models talked like markov chains. I guess OpenAI's "fake it till you make it" strategy was so successful that they can't stop even after they've already made it

Are you so deeply set in this mindset/idea that OpenAI is dishonest that you can't imagine that they might just be doing what they say they are doing?

Would you be willing to say literally anything I ask you to to anyone I specify, including telling your family that you have been kidnapped and need money?

Sometimes, when someone tells you they are concerned about something, that something is concerning. We should evaluate the actual concern. I am concerned you do not realize this.

I don’t think NavinF shows any lack of concern for the issue as such. What’s being talked about here is whether OpenAI’s being forthright concerning their motives for keeping the tech closed.

I think we can all agree it is valid to be cautious with some of this tech, but the question is whether OpenAI is being cautious or simply using caution as PR. And given their track record, the latter seems, if not likely, then possible.

Their action of removing the ban on military use of their technology has made me disbelieve their claims.

Couple that with deceptive naming practices ('Open' AI) and how much they've compromised their independence with so much Microsoft ownership.. why _would_ you take their claims at face value?

https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf

Regarding the kidnapping scenario, my family isn't stupid enough to send money to anyone that sounds like me. Impersonating the gov't and sending letters would be a much better strategy since it's much harder to verify an organization's identity. Btw voice cloning is fairly mainstream and I've already used free voice cloning apps to humorous effect. Do you think people can't clone your voice today thanks to OpenAI holding back?

OpenAI said the same for GPT2 as well.. They were soo concerned about the humanity that they switched to being a for profit company!

To be fair the obvious application would be mass phone scamming while the general public is still unaware.

Maybe they just wait for somebody else to take the brunt of the reputation damage.

moderately convinced openai is a huge scam by now.

I think they've hit a plateau or something. Compare the recent Sam Altman interview by Lex Fridman with the one only a few months prior. The dude's in limbo. You can almost hear the legal team breathing down his neck not to make unsubstantiated, deceptive claims... They got high on their own supply and now have to cut their product with ground glass to make the same dose hit harder. There is some uncomfortable AI sobriety incoming.

Given how good the newly released available-weight VoiceCraft[1] project seem to be, I would say the cat is already out of the bag.

People will soon have to learn that they should not trust voiced content. In regards to scam prevention, the key learning will to validate the security of the channel and never trust a voice alone.

If someone you know calls you from an established channel (their usual account / phone etc), it's probably safe. But any unauthenticated channel communication (unknown account etc) should be treated as suspicious.

[1]: https://github.com/jasonppy/VoiceCraft

Spoofing caller ID used to be trivial, and I suspect it still is despite STIR/SHAKEN.

It is an Easter miracle, whom amongst us has not prayed for 500+ words on some software that may or may not exist but certainly has zero impact on anything in the world?

Why the hell did they build it then?

Research? Gain knowledge? Expand their understanding?

>Gain knowledge

Gain of knowledge, and function.

"I have become Death, destroyer of worlds"

It's an age old question. Do you not build the thing that will destroy the world while hoping nobody else does as well, or do you build it first so that you can "control" it?

Defense contracting is lucrative, marketing even moreso.

> At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus

Yes it's a meme. But it's the summary of OpenAIs public demeanor

someone would have done it. better US than them

lol this marketing technique is getting very old. https://github.com/coqui-ai/TTS is already amazing and open source.

Yeah I also have an AGI already, but I won't release it, it's too dangerous for you lot.

I had a job about a month or two ago on Upwork which was supposedly going to be about a character.ai clone or something similar. After discussing it for weeks, on the last day they decided it wasn't about personas in general and absolutely had to be impersonation of Trump and Biden.

I told them it was a very questionable idea. But I needed the money to be able to make rent. So I took the job hoping that they were still interested in other types of characters and would be reasonable.

I had a basic demo with Trump's voice up in about a week. I showed them the first demo and had a phone call suggesting that this might actually be illegal pretty soon.

They were sure there was no problem and said they had spoken to lawyers.

The next day, Eleven Labs sent out an email explaining that some voices were getting banned. The attempt to clone Biden's was blocked by Eleven Labs.

They had structured the contract as a single fixed fee, although what we had discussed was weekly payments. At the last minute without discussion they had thrown in a talking head requirement.

So I received the first 451 HTTP status code I have ever seen when requesting that D-ID do a lip sync animation of Trump's head.

I ended up using a different video model from replicate that was not a great lip sync but very engaging if you squint. I did not do Biden. Did about three weeks of work and collected around $630 after Upwork took their fee.

I'm proud to say I did make my rent payment though since I had another project at the same time which was a low hourly but still helped.

Potentially unpopular opinion- this may have been done in part with the objective of enforcing regulations or setting up AI laws. Given OpenAI’s position that comes across as regulatory capture.

Wiki article on the topic: https://en.wikipedia.org/wiki/Regulatory_capture

Instead of phrasing it as stating an unpopular opinion, try asking a question with an unpopular answer instead, that way someone else takes the downvotes on your behalf.

Election year for ~4.21% of the world population… Or that’s the part they are keeping in mind with this announcement anyway.

4.21% of the population that wield closer to 30% of the world's political, military, and economic power. It's a valid concern.

Happy 4/20 everyone!

I think you forgot about India. It's 22% of the world minimum.

“This year countries that are home to 4bn people–half the world’s population–will hold national elections”

https://www.economist.com/films/2024/01/12/elections-around-...

It's already happening. There have been AI ads targeting vulnerable, elderly populations for months, promising free Social Security cash in the likeness of our current POTUS:

https://www.newsweek.com/tik-tok-scam-social-security-biden-...

Ads are 100% blocked on my devices, so I wouldn't have been aware if not for a recent trip to the doctor's office where separate ones played on two old men's phones at full volume.

[dupe]

Some more discussion on official post: https://news.ycombinator.com/item?id=39866493

Can we not already basically do that? I swear I’ve seen pretty convincing AI text-to-speech that could be used for nefarious purposes.

Yes. Eleven Labs has convincing custom model and has been around for over a year.

That was my impression too. Maybe OpenAI’s model is impressively better than anything out there right now, or they’re just embarrassed cause it’s so much worse :P

I usually would've commented something along the lines of how "open" their AI is, but to be fair, this is a pretty understandable move. Sadly, it's only a matter of time before someone else will offer this feature to the public. So "open"AI is only slowing down the process.

I don't see it as sad. Has the world fallen apart now that GPT-4 and DALL-E exist? Hardly. It's not going to collapse when we have easy voice cloning either.

And despite some commenters who lack imagination saying "I can only think of bad uses for this" there are tons of good potential uses. Video editing, game voiceovers, music, phone menus, audio books, etc.

Remember when GPT-2 was deemed "too dangerous to release"? Spreading FUD appears to be OpenAI's viral growth strategy.

https://slate.com/technology/2019/02/openai-gpt2-text-genera...

This is not interesting anymore. I am too busy looking at the spicy xz backdoor

build up for a mole leaking the code or a cyberattack originating in eastern europe or russia or a combination of both, thus including corruption and creating the need for stronger oversight of people working on AI by some clandestine agency and of course extreme regulation of open source AI and people capable of writing that kind of code, which would include extreme regulation of access to the training data and so on ... the interest groups in power censored the training data, took openai under their wing, and the fun continues. sorry, couldn't resist.

My own speculation is it’s too crap / embarrassing to release.

Are they trying to get competitors regulated?

How very Open of them

then why develop it in the first place?

Reply on news.ycombinator.com