Hacker News Reader

Comment by peddling-brink | Hacker News Reader

So Open AI has the capability to deep fake any of its users that use the voice chat capability? Yikes.

Anyone can clone a voice with access to a couple of seconds of it.

We'll have to gradually get used to the notion that a person's voice, like so many other things we once thought of as intimately personal, is just a coordinate in a high-dimensional vector space.

> is just a coordinate in a high-dimensional vector space

Everything is, including types of personality, physiognomy and demographics. Anything that can be related to anything else is embeddable.

Is this a well studied topic I can research or just a guess?

Those are not contradictions.

One of the common bad takes is that since personal things can be distilled to “just math” they are no longer personal or valuable in the sort of way we value personal things. Increasing our understanding of how the world works shouldn’t devalue the world.

To put it another way, people having souls is not the only reason to treat people like people. Or as Dr Seuss says: A person’s a person no matter how small.

> they are no longer personal or valuable in the sort of way we value personal things

There were people with your face or voice before, you just didn't care. AI continues a trend: the internet has always been post-scarcity, free copying and offering an amazing array of choices. We are acting as if it's a new thing, but it's been here for 25 years.

We shouldn't value people simply because they are scarce. People are valuable intrinsically. Identical Twins don’t have less value because there are two of them. A mother is not less sad when one of them dies as compared to a mother who doesn’t have twins. Trying to make such a comparison of value is gross.

I do appreciate that not everyone agrees with that view, and that capitalism tries to value people at their output but intuitively plenty of people find that sort of thing wrong and ignoring that intuition is something I’d argue will not end well for us as a society.

But I do think there are some psychological hurdles we’ll have to overcome, as it becomes possible to mechanically copy individual personal aspects. A person’s a person no matter how small, but we aren’t all used to thinking of ourselves as very small.

But it is unintuitive to those of us who are from the past.

It always has been. The same way passwords were never secure, we just found that they are too slow to bother uncovering. The same can be said for voices/AI. Except now it's not too slow to "crack an easy password" in the voices/AI world.

The idea that everything is open, you just need the right tool, is scary to most.. but it's not once you realise how expensive the tools are.

Even simpler, mathematically anything computable is a natural number.

That doesn’t remove privacy and other rights.

https://en.m.wikipedia.org/wiki/G%C3%B6del_numbering

Fun fact: many of the TTS API providers (Google, Azure) now have voice cloning capability.

There's a reason it has gone under the radar.

A couple seconds? Which project does this?

every single voice cloning project

I'm sure you could have used google

https://elevenlabs.io/app/voice-lab

https://app.resemble.ai/users/sign_in

https://github.com/neonbjb/tortoise-tts

https://coqui.ai/blog/tts/open_xtts

it was even possible 5 years ago

https://github.com/CorentinJ/Real-Time-Voice-Cloning

Mission Impossible 3 was only the proof of concept

how many do you think "a couple" is?

Nope. If you actually tried those, you would quickly find out they don't work. It's actually really hard to clone a voice from a few seconds sample.

A few seconds, yeah. I've seen fairly convincing reproductions from 30 seconds of reading text though.

Cloning voice signature or timbre may need a bit more for a good quality. Then there are idiosyncracies in one’s voice. In addition to that, there are tiny verbal tics, expressions, cadence, feel, and some more to be able to say you have properly cloned someone’s voice. The two second sample is like a shallow clone of sorts and is indeed vector space.

The last sentence is hilarious. What do you think "properly" cloned voices are? Not every model is few shot and not every model relies on their training set for paralanguage anymore. Easiest way to try it out is properly the pro voice cloning from elevenlabs.

"Expressions" in particular is about choice of words. A few seconds definitely isn't enough to duplicate that.

The choice of words is usually yours for a tts model.

There are many models, XTTS [1] is a good one.

[1]: https://coqui.ai/blog/tts/open_xtts

I’ve had success with elevenlabs.

Did anybody with a passing familiarity with the technology not know this? There's a reason people don't trust their IP to public chat assistants, ya know. It's because they're remorselessly siphoning our data for training purposes. And you've agreed to let them use your inputs for anything they please.

That’s not what is happening here. Instead, the user’s voice output during the session is used to generate later sound by the model accidentally.

No, I get that. But that's the thing about LLMs, isn't it? They're good at filling in the next token to make a plausible-looking conversation. And if the machine doesn't expect you to be done talking, why would it not just carry on in your voice?

I was under the impression that multiple separate models were in use. I assumed it was something like whisper that converted the user’s input to text then an LLM then text to speech.

I hadn’t realized this was all in one model. Wild stuff.

> And you've agreed to let them use your inputs for anything they please.

This has always been the Google and FaceBook deal - your data for services, and the AI assistant deal remains the same. I believe people will flock to the best AI with their data, handing it on a silver platter. Will they spend half a second thinking about privacy or prefer immediate results? When you know your competition uses AI, you aim to have just as much AI assistance or more.

Reply on news.ycombinator.com