Hacker News Reader

Comment by muglug | Hacker News Reader

These tools make it very easy to scam vulnerable people, and have pretty limited use otherwise.

I'm absolutely using celebrity voices for my Home Assistant voice. Amazon has spent the last couple years removing the voices for Alexa that people had paid for.

I’d love some more info on using custom voices in HA. I have an esp32-s3-box that I am setting up holiday to do voice with HA.

If you have a how—to, I’d love to work on one for my home. I feel like this is all right around the corner…

To be fair, they’ve got pretty serious potential for letting tech companies get paid for a seasoned voice actor’s unique delivery, tone, inflection, etc rather than the voice actor themselves.

> they’ve got pretty serious potential for letting tech companies get paid for a seasoned voice actor’s unique delivery, tone, inflection, etc rather than the voice actor themselves.

I think you mean "steal the labor of an actor"?

Sure, and people that already agree with you will feel good reading it, but other people who don’t agree see it as an attack. It’s pretty much impossible to slip a new idea into someone’s mind if your approach made them slam the door before even considering it. So what’s the benefit of saying it like that?

It calls attention to the ethical implications of using a part of someone else's personal identity without their direct involvement.

Indirect involvement can still be ok within the confines of a license agreement for using the actor's voice.

But this requires a legal framework that mandates such licenses and effective emforcement / procecution of violations.

As far as I know, most countries are lagging behind when it comes to updating legislation to set binding rules around that.

> Indirect involvement can still be ok within the confines of a license agreement for using the actor's voice.

This assumes existence of a license agreement or likeness/right of publicity law that prevents unauthorized use. But this is far from the case.

Companies have shown willingness to use actors’ voices to create synthetic voices without permission, compensation, or regard for their livelihoods. [1][2][3]

[1] https://animehunch.com/popular-japanese-voice-actors-band-to...

[2] https://www.theatlantic.com/technology/archive/2024/05/eleve...

[3] https://www.yahoo.com/entertainment/morgan-freeman-calls-una...

Of course we need laws in place to require such licensing. The fact that people are having their voice stolen now does not mean that there should never be a case where a voice can legally be cloned and used by a third party.

Precisely. We must recognize this as a fundamental issue of workers’ rights and personal autonomy in the digital age, beyond viewing it as a technical challenge. Without proper protections, voice cloning technology risks concentrating power in large companies and undermining creative workers’ economic security.

So does what I said. Someone taking pay for someone else’s work is pretty unambiguously shitty. But when you call taking anything that isn’t a physical item theft, a large percentage of people— especially in the ‘data wants to be free’ crowd— will roll their eyes, think “that’s ridiculous... they aren’t stealing anything. That voice actor still has their voice” and just stop listening. The only people that feel the impact of statements like that are people that already agree. It turns it from an intellectual discussion to a reinforcement of existing tribes. Divisive language works for rallying those who already agree around a specific cause but it’s not even useless— it’s counterproductive— for changing people’s minds. When’s the last time someone you disagreed with changed your mind by being more aggressive towards your stance, and more terse in their portrayal of the dichotomy? If you can even think of one time that it has, you’re in the extreme minority.

It’s weird to me that people look at a technology and then assume from their reckoning that they know all the uses for that technology immediately. Most technological progress happens because someone notices a creative use for something that already exists which nobody else has noticed.

I like tools like these cause they make zero trust default even more obvious, and their "pretty limited use" is saving people hours of work.

They are pretty good for leaving messages for my blind friend. I generally find calling / voice texts a waste of time (I type and read far faster than I talk or listen, not to mention the ability to reread etc), but my blind friend prefers getting voice messages when on his phone and this works for us. I type and send and when he comes back with something, Whisper makes it into text for me.

Gen AI space to everyone else: “Your computer scientists were so preoccupied with whether or not they should, they didn’t stop to think if they could just do it anyway”

How many victims will it take for lawmakers to do something about this?

It's already illegal to scam somebody. While it's always positive to protect people more, what can be done here? Any alternative I can imagine is massively oppressive of the current state of the software industry.

You can regulate large companies, you can regulate published software sold for profit, but it's impossible to regulate free and open source tools.

You essentially have to regulate access to computing power if you want to prevent bad actors doing bad things using these sort of tools.

>You can regulate large companies, you can regulate published software sold for profit, but it's impossible to regulate free and open source tools.

Regulation is putting legal limitations on things, if it is impossible to regulate free and open source tools then it would be impossible to regulate murder and lots of other things, but it turns out it isn't impossible, sure - murder happens - but people get caught for it and punished.

Sorry, but this argument is much like the early internet triumphalism - back when people said it was impossible to regulate. Turns out lots of countries now regulate it.

It depends on what you do with the tool. Going with your murder analogy, if there's a stabbing epidemic what do you do? 1) Ban knives 2) invest in public safety 3) investigate the root causes and improve on them?

I'm also not sure what's so regulated about the internet besides net neutrality in certain countries. Of course the government can put limits on the network, like banning services, but it's easy since they are rather easy to target. With content traveling on the network it's much harder to say if it's legit or not.

> lots of countries

What about those countries that don't regulate it and people will keep pumping out better, leaner and faster models from there? Spreading software is trivial, all you achieve is the public won't be aware of what's possible.

The more I think about it if anything should be regulated that's a requirement to provide third party (probably government backed) ID verification system so it would be possible for my mom to know it's me calling here. Basically kill called ID spoofing.

>I'm also not sure what's so regulated about the internet besides net neutrality in certain countries.

generally things are regulated on the internet that were not going to ever be regulated because it was on the internet - example - sales taxes, perhaps you are old enough to remember when sales tax collection would not ever be enforceable on internet transactions - those idiot lawyer don't know, it's on the internet, the sale didn't happen in that country or in that state no sales taxes will never happen on the internet hah hah. It's unenforceable, it is logically undoable, there are so many edge cases - ugh, the law just does not understand technology!

oops, sales taxes now on internet purchases.

GDPR is another example of things that are regulated on the internet that basically most of HN years before it happened was completely convinced would be impossible!!

If this thing becomes too big a problem for the societies regulations will be done, with varying levels of effectiveness I'm sure.

And then in twenty years time we will be saying what, you can't regulate genital eating viral synths because a guy can make those in his garage and spread them via nasal spray, this technology is unstoppable and unregulatable, not like some open source deepfake library!!

It's always amusing listening to techies' musings on law... lots of misunderstandings, I suspect due to the helpful but inaccurate "code but for humans" analogy.

Obligatory/relevant xkcd: https://xkcd.com/538/

Lots of countries impose exactly what specific regulations with respect to open source tooling?

The closest thing I can think of is maybe the regulation of DRM ripping tools, but they're still out there in the wild and determined actors can easily get ahold of them. So I'm not at all confident that regulation will have any measurable meaningful effect.

The fable of the "determined actor".

The "determined actor" can get bombs, tanks, fissure material. There noone says "WHELP they can get it anyway so why bother regulating it LMAO" - somehow this is different in anything not physical?

>Lots of countries impose exactly what specific regulations with respect to open source tooling?

that something is not currently regulated does not mean it can never be regulated, further it does not seem likely that they would regulate open source tooling but rather some uses and if they open source tooling allowed those uses then what would happen is -

github and other big sources of code would refuse to host it as containing not legally allowed things, so for example if they regulated it in the U.S then Github stops allowing it, and everyone moves to some European git provider.

At the same time bigger companies will stop using the library because liability.

Europe then regulates and can't be in European git repos.. at some point many devs abandon particular library because not worth it (I get it this is actually for the love of doing the illegal thing so they won't abandon but despite the power of love most things in this world do not actually run on it)

Can determined actors get ahold of them and do the things with them the law forbids them to do, sure! That's called crime. Then law enforcement catches determined actors and puts them in prison, that's called the real world!

Will criminals stop - nope because there is benefit to what they're doing. Maybe some will stop because they will think screw it I can make more money working for the man. And some will be caught sooner or later. And maybe in version two of the regulations there will be AI enhancements - this crime was committed with AI allowing us to take all your belongings and add 10 years to your sentence and deprive you of the right to ever own a computing device again...etc. etc. And some people will stop and others will get more violent and aggressive about their criminal business.

I don't know necessarily what measurable meaningful effect means, for some people it will be measurable and meaningful, for some not, for some of society the regulation would in many ways be worse than what it is fighting against. I'm not saying regulation will solve problems 100%, I'm just saying this whole they can't regulate us thing because "TECH!!!" that developers seem to regularly go through with anything they set their eye on is a pipe dream.

> impossible to regulate free and open source tools

BS. Can you imagine a legislation? Yes, thus it can be done.

As an early example, the CRA (Cyber Resilience Act) already contains provisions about open source stewards and security. So far they are legal persons, aka foundations, but could easily relate to any contributor or maintainer.

As I made the comment, I can't really imagine anything that's not so absurd that has a more than zero chance of happening.

Seriously, what can anybody do about random hacker Joe publishing under the name XoX? Even if they burn GitHub and friends to the ground, if something is useful it will be really really hard to get rid of it. Remember youtube-dl? It's now https://github.com/yt-dlp/yt-dlp

If they make anything that cripples open source development they will feel it quite soon when they realize that it also cripples their world as much of the tooling and infrastructure also depends on it.

Killing open source is like killing the internet itself.

Consequences never stopped anyone.

Your example with yt-dl doesn't matter.

Open source/free software inherently relies on copyright and all state legal infrastructure. Once you operate outside, it's no longer open source/free software.

Can you host software in a way that's really hard to block? Sure. There is onion routing and plethora of other options.

But that's no longer open source/free software. You are in a realm of dark web and marketplaces.

I do maintain a semi popular open source project that I took over after about a year of inactivity and I seriously considered quiting because of CRA. It's quite easy to cripple/kill something when it basically runs on volunteering of your free time.

Serious question: what do you think lawmakers should do?

For people's image being used without their permission: strengthen U.S. right of publicity laws with private right of action, enabling people to sue for unauthorized use of their voice or likeness.

Digital signatures as part of audio/video that can't be easily modified or faked which can trace the origin of a piece of media. Some camera manufacturers are already working on it.

How do you propose to keep watermark-free models out of the hands of evildoers? I can't build my own digital camera or laser printer, but I can certainly write software.

I don't have a good solution, but maybe legislation helps. There may not be a foolproof solution but I think the more that such devices are widely used, the less likelihood there may be of e.g. a court case hinged on bad evidence.

how many victims did it take for lawmakers to do something about Photoshop/GIMP/etc?

Bulldozing grandma is just the cost of technological progress /s

This tech is going to be ubiquitous, it's just too easy to distribute it. Grandma better starts adapting now.

Because people make it so, not because the natural order of the world gets us there

For some reason because we can validates that we should. Any jackass has the power of a research team of phds. It's kinda weird.

Demanding responsible behaviour from everybody is not going to work. Some people don't care about negative externalities that much and it's enough if only a few of them decide not to play ball. So either grandma needs to adapt which will upset some people or distributing the tech should be regulated/prosecuted which will upset another group of people.

I think either way grandma needs to adapt though since Russian scammers and trolls are still going to run scams with fake voices.

how very politically correct of you to pretend it's Russians who scam your grandmas

Insert any other country you like that doesn't have extradition agreements with the United States. Any other country the law can at least ostensibly be enforced there, even if it isn't always

Indeed. Humans ascended to dominance because we can cooperate. This every-man-for-themself idea is an aberration, not the natural order as so many claim. It’s rather astounding to think otherwise considering the logistics of how we’re communicating right now.

Cooperation works if the potential damage caused by a rouge actor is sufficiently low. Otherwise, it's too easy to sabotage things. This is why we don't want random rouge states to have nukes. AI will give so much leverage to rouge actors that it will significantly shift the game theory in favour of not cooperating.

> Cooperation works if the potential damage caused by a rouge actor is sufficiently low. Otherwise, it's too easy to sabotage things. This is why we don't want random rouge states to have nukes. AI will give so much leverage to rouge actors that it will significantly shift the game theory in favour of not cooperating.

Governments successfully collectively controlling dangerous things so they don’t fall into the hands of rogue bad actors fundamentally opposes the extreme individualist every-man-for-himself perspective in every conceivable way. It’s the absolute opposite of “it’s everybody’s responsibility to protect themselves because everybody else is only going to look out for themselves.”

And when individuals have that much leverage, collective action is the only conceivable way to oppose it. Some of those things might be cultural, like mores, some might be laws, some might be more martial. I don’t see how extreme individualism even theoretically could be more powerful.

Are you suggesting government action against putting up code like this to GitHub? It’s ok if you are, but I want to put into more concrete terms what we’re talking about.

You’re the one that made the direct government control analogy. I mentioned a number of non-individualistic mechanisms in my previous comment. I’m not going to keep engaging in a fishing expedition of things to argue about — I think it’s pretty clear what aspect of your stance I disagree with— and am going to leave it at that.

So you don't have a concrete suggestion to solve the scamming problem?

You can’t adapt around brain age making it more difficult to distinguish truth from lies.

Yeah, I don't really get the hulabaloo, if granny doesn't have the mental fortitude to keep up with the times she shouldn't be managing her own money. I guess better her son/daughter than a scammer but both are better than letting money rot. Put granny on foodstamps and pay $1 for her rent controled housing be done with it.

Are we forgetting, that there are many elderly people without living descendants?

This tech is not only great for bulldozing grandma, its great at stealing content from other creators and rebranding it as your own. Based on the github, it kind of seems like thats exactly whats being advertised as the use case. Steal content from BBC, cut it up and pull the noise out/vocals/revoice the content so the algorithm cant detect the plagorism easily. The imagine detection is no where no the audio detection for copyright strikes.

There is a massive problem with this on youtube. Pretty much every category on youtube now has a host of these bots trolling content and playing the youtube strike system like a banjo. There are channels detected to showing you how to setup these content mills. This tool can make you good money.

First generative AI destroyed Google search, and now it has pretty much destroyed YouTube. Social platforms, including this one, are probably goners too. We live in interesting times.

Quit being a doomer or keep it to yourself. This reminds me of the sound boards that were popular in the early 2000s except way more versatile. Some things are just good for people to have fun and THAT'S OKAY.

People are allowed to recognize the realistic negative outcomes of technology, especially on a forum that frequently discusses the tradeoffs of modern, cutting edge technologies.

So many AI posts are overrun with this kind of complaining from folks with limited imaginations.

On a forum that frequently discusses technology with enthusiasm you'd think there'd be more enthusiasm and more constructive criticism instead of blanket write-offs.

I would argue that being able to see the drawbacks and potential negative externalities of a new technology is not a sign of a "limited imagination", but quite the contrary. An actual display of a limited imagination is the inability to imagine how a new technology can (and will) be abused in society by bad actors.

Developing some insight on its negative potential could demonstrate imagination, but the claim that it could be used to scam people is pretty much just rote repetition by now - an obligatory point made in every article and under every post about this tech (and not something that I think actually works out in practice the way most imagine it, since cold-call scam operations that dial numbers at a huge scale expecting most not to pick up can't really find a voice clip prior to each automated call).

As for positive applications, some I see:

* Allowing those with speech impairments to communicate using their natural voice again

* Allowing those uncomfortable with their natural voice, such as transgender people, to communicate closer to how they wish to be perceived

* Translation of a user's voice, maintaining emotion and intonation, for natural cross-language communication on calls

* Professional-quality audio from cheap microphone setups (for video tutorials, indie games, etc.)

* Doing character voices for a D&D session, audiobook, etc.

* Customization of voice assistants, such as to use a native accent/dialect

* Movies, podcasts, audiobooks, news broadcasts, etc. made available in a huge range of languages

* If integrated with something like airpods, babelfish-like automatic isolation and translation of any speech around you

* Privacy from being able to communicate online or record videos without revealing your real voice, which I think is why many (myself included) currently resort to text-only

* New forms of interactive media - customised movies, audio dramas where the listener plays a role, videogame NPCs that react with more than just prerecorded lines, etc.

* And of course: memes, satire, and parody

I appreciate HN's general view on technologies like encrypted messaging - not falling into "we need to ban this now because pedophiles could use it" hysteria. But for anything involving machine learning, I'm concerned how often the hacker mentality seems to go out the window and we instead get people advocating for it to be made illegal to host the code, for instance.

Of the 11 positive applications that you listed, only the 1st, 3rd, 11th and arguably the 4th would benefit from voice cloning, which is what's being promoted here. The rest are solved merely by (improved) TTS and do not require the cloning of any actual human voice.

Also, notice how the legitimate use-cases 1, 3 and 4 imply the user consenting to clone their own voice, which is fine. However, the only use-case which would require cloning a specific human voice belonging to a third party, use-case 11, is "memes, satire, and parody"... and not much imagination is needed to see how steep and buttery that Teflon slippery slope is.

> Of the 11 positive applications that you listed, only the 1st, 3rd, 11th and arguably the 4th would benefit from voice cloning, which is what's being promoted here. The rest are solved merely by (improved) TTS and do not require the cloning of any actual human voice.

2, 5, 6, 9: It's true that in theory all you need is some way to capture the characteristics of a desired voice, but voice-cloning methods are the way to do this currently. If you want a voice assistant with a native accent, you fine-tune on the voice of a native speaker - opposed to turning a bunch of dials manually.

7, 8, 10: Here I think there is benefit specifically from sounding like a particular person. The dynamically generated lines of movie characters/videogame NPCs should be consistent with the actor's pre-recorded lines, for instance, and hearing someone in their own voice is more natural for communication and makes conversation easier to follow.

Pedantically, what's promoted here is a tool which features voice cloning prominently but not exclusively - other workflows demonstrated (like generating subtitles) seem mostly unobjectionable.

> Also, notice how the legitimate use-cases 1, 3 and 4 imply the user consenting to clone their own voice, which is fine

I think all, outside of potentially 8 and 11, could be done with full consent of the voice being cloned - an agreement with the movie actor to use their voice for dubbing to other languages, for example. That's already a significant number of use-cases for this tool.

> use-case 11, is "memes, satire, and parody"... and not much imagination is needed to see how steep and buttery that Teflon slippery slope is.

IMO prohibition around satire/parody would be the slippery slope, particularly with the potential for selective enforcement.

This is a GitHub repo, not an article on the effects of TTS. Policy discussions at the level of the parent comment feel off topic.

Reply on news.ycombinator.com