Voice clone of Anthony Bourdain prompts synthetic media ethics questions
techpolicy.press
techpolicy.press
He hated fake phoney people, or food.
He hated when his production crew would stovepipe his bits.
He hated a scene in Greese (I believe) where crew went out and bought squid, and threw them into the water to made the shoot better.
I really liked Bourdain. He was one of the few celebrities that didn't seem to change with fame, or money.
I watched him for years, and knew he was unhappy, but never thought suicide unhappy.
His death really affected me in a deep way I didn’t expect. I miss him so much.
I have no problem with a fake voice reading Bourdain’s own words.
But he also went to Waffle House and enjoyed it. And hiked through the jungle to eat and drink (mostly drink) at a local festival and sleep in a hallway with the rest of the locals.
That was what really impressed a lot of us - he seemed to get along with everyone and to value and accept their differences. The food often seemed to be largely an excuse to explore and show that message.
I think the squeamishness about AI as it becomes more and more capable will be interesting to define why we are feeling it. The machine is going to be capable of (and consequently used to do) these things whether you like it or not.
[1] https://twitter.com/OttaviaBourdain/status/14158894550057164...
This is the "fact". Half of it is a caveat to take it with a grain of salt.
I should hope people's ex-wives don't get authority over their lives later (unless it's related to kids/shared-assets/etc).
The family giving consent is way more relevant.
That's incorrect. They were still married when he died and most of Bourdain's estate was left to their daughter with Ottavia overseeing it. "The family," in this case, is Ottavia and Ariadne Bourdain.
https://www.usatoday.com/story/life/people/2018/07/06/anthon...
> estranged wife Ottavia Busia-Bourdain
He left her in charge of the trust/etc because she's still the mother of his child, and therefore a major part of his life, but they weren't together.
Creating a digital voice model of abraham lincoln and then using it for ??? is not.
All of this is being done for profit means (the orville redenbacher, fast furious, star wars).
Just because we can doesn't mean we shouldn't stop and think if we should.
If I’m honest humans have never done the moral thinking, we largely just middle through and hope to get away with it.
John Lennon's voice doing an ad for Vox amps seems in poor taste.
the Bourdain documentary we're talking about was created in order to generate profit; it has box-office showings and advertisement campaigns.
A documentary is nice, but I fail to see how it's less of a product than Vox amplifiers, aside from some shallow altruism like "We're spreading the ideas of Bourdain", which could be done entirely without a profit scheme.
No, the 'for profit' nature of this venture is just the amoral cherry-on-top.
My real problems with this type of thing are many-fold :
1) Textual messages are often times created in such a way as to be convenient rather than to be read. Inflection and tone mostly go out the window when written out, so people tend to use lots of hyperbole and metaphor to better convey nuance like sarcasm -- this can quickly make someone who is eloquent in-person look moronic in-text, but people close to the individual know better and ignore the email parlance.
2) There is a level of artistry and creative interpretation when deciding how an AI voice generates text-to-speech. What samples are used, how is the phrase tempo'd, environmental interpretation, etc. The end product is very much not-Bourdain , but it's sold to people watching the documentary as Bourdain, so it's assumed that he said these things.. which leads to :
3) Personally I believe that the post-death 'image' of anyone should only be manipulated or altered by third-party testimonial and record-level evidence.
An unaware person watching this documentary would come out of the theater with the notion that Bourdain said things that he never said.
A third party is using Bourdain's own image to sell persona/image changes to the public on his own behalf, post-death -- I have major ethical and moral qualms with this idea, even with all the express permission of those around him and his family.
Regardless of who these people are that gave permission -- they're not him. The voice/image/actions of your own persona should probably never be tampered with by someone else -- even the closest people to him can't begin to know what he'd say internally, even if they may have felt that they had a good grasp.
Now, sure, we can always argue that a celebrities image is likely sold to the companies managing them -- and I would have to agree. Legally, sure, his image likely isn't his own property, but I don't have to agree with the methodology on an ethical level, and I have a real hard time imaging that I ever will.
>Do you think the same documentary would be better if it didn’t have advertising or box-office showings?
Yes. I am less against the manipulated and possibly perjured conjuration of fake dead-people's voices when someone isn't making a profit.
(I detected a bit of snark in your question, so I replied with a bit of snark -- but it's an honest answer. Apologies if you didn't mean for it to be interpreted that way -- another incongruity in text-based-talk :) )
Just one example of the prickliness of this with the for-profit aspect in scope : an AI generated likeness of a famous voice says "Eat Snickers!" , after the family of the original voice gives all the permissions in the world to whoever may want to use it.
Tons of royalties are accrued.
Who is given the royalties?
so, tl;dr, here's the point I feel most strongly about : An unaware person watching this documentary would come out of the theater with the notion that Bourdain said things that he never said, in a way that he perhaps never would have said them. This technology, along with others like it, creates a situation in which a third party can manipulate the persona of a long-dead person in a first-person way, thus circumventing a lot of the trust checks that exist with regards to the perceived image of the now-dead.
I don't think this particular example is damaging, even if I think it's unethical -- however I do think that it's an indicator of things to come, and I believe that it is going to be heavily abused until some regulatory body makes lots of rules and regulations regarding the depiction of image of those that are no longer able to fight their own libel cases.
Most direct and honest answer I can give you :
It's (considered by many) unethical to profit from human death and suffering.
It's (considered by many) extra unethical to assume you can speak for the dead.
It's (considered by many) extra-extra unethical to try to hood-wink an audience into thinking that your Mechanical Turk is the real-deal.
This particular 'valuable art' we're discussing has flavors from all three of those types of behavior.
Impersonating someone like Lincoln in a car commercial is one thing, but it’s another is someone presented an impersonation as their actual voice in a documentary.
It’s also complicated by the fact that the AI voice is trained on his likeness and dubious as if the reproduction is a new creation or something else. Similar to the debate about GitHub Copilot.
> “If you watch the film, other than that line you mentioned, you probably don’t know what the other lines are that were spoken by the A.I., and you’re not going to know,” Neville told the reviewer, Helen Rosner. “We can have a documentary-ethics panel about it later.”
...or we can have it sooner, on Twitter, and you'll get excoriated, and rightly so. I don't care if they train an AI to imitate Bourdain rapping the third verse from Modern Major General, it's a free country, but you have to be honest about it. You can't call yourself a documentarian and get cutesy about the authenticity of the material.
Perhaps we'll see a mea culpa later on, where he will say he misjudged the amount of outrage, which is not a real apology, so lots of commenters will jump on that thereby providing even more publicity.
Creating an outrage for some publicity of a documentary just doesn't sit right with me. Yes, it is an effective step or publicity stunt, but how does people don't have that inner voice? (I don't know what to call it) that tells them that this isn't what they should be doing without proper consent of the family members and without any disclosure over what is real and what isn't.
A conscience.
1,000 Elvis' likenesses disagree.
I would just like to point out the irony in this comment. You're speaking on behalf of a dead person in objection to someone else speaking on behalf of that person.
We can debate the ethics of this whole thing without trying to lay claim to what Bourdain would have thought.
The objection isn't to speaking on the behalf of the dead, it is to using phoney tactics to do so, playing them up for emotional value
Refusing to disclose which dialogue the AI generated serves no one but the documentary maker himself. He’s not protecting a source or doing anything commendable. It doesn’t make the documentary better on any way.
It’s a media stunt.
It’s despicable.
You are essentially asking why bother investigating and forming opinions and value judgements on the the use of new technology. Humans use opinions and values to decide how they want to interact with the outside world. Even if the technology exists and no living are harmed, people can find it distasteful. people can decide not to support what they find distasteful. They can decide to voice support actors unions if the issue ever comes up. They can tell their friends and family how they would like their likeness used as these technologies become more prevalent.
I have no qualms with the tech existing, assuming I do is you creating an argument I don't have. In fact I work in ML generative modeling. But just because a tool exists doesn't mean you shouldn't be thoughtful about how you use it. Technology isn't really good or bad on its own.
> there's no individuals actually being harmed
This is debatable. I think a lot matters on how you think of dead people. I believe most people would take the stance that it is unethical to use someone's image to promote things they did not stand for. You're right that it is very difficult to impossible to harm a dead person, but there's still respect. There's also a family that lives with how you portray said person. They _can_ be harmed.
> the end-game is just going to be updated contracts which demand rights to use the performer's likeness for these sorts of purposes
If there's (informed) consent then what's the issue? (If we assume that said person knows what they have consented to)
Forgive me if I'm wrong, but your response seems in bad faith and to operate on many assumptions about me that just do not hold true. Frankly this is not the kind of conversations I want to have on HN or expect to have. I'm happy to argue about the ethics and have differing opinions, but I do not appreciate my image being depicted with a broad brush. As long as the conversation is in good faith we can discuss a lot of controversial things and disagree all day and everything will be fine. But turning
> Bourdain likely would not have liked this
into
> a fashionable hint of 21st century doom-cult luddism
Is just ridiculous. That's far too large of a leap and such comments are not welcome here. You're welcome to try again but you'll need to update your priors.
It does contain lyrics, but the singing is not AI generated.
I've had an idea to use propellerhead recycle to chop the output cloned voice into syllables, and then "play" the chopped parts in rhythm, through autotune.
The issue is you get Eifel 65 sounding autotune if your base vocals are monotonic or way off key. The only way I can think of fixing this is to use something like audacity's pitch changer that doesn't affect the speed of the sample - rough the lyrical tones in with audacity/recycle, then autotune it where it needs to go.
I'd like to say I'm too busy to get this workflow going, but mostly I'm lazy and someone else will do it first - and better - I can't improve the AI cloning software.
Check out Ableton Live’s “Simpler”
I've had a new version of this in the works for a few months. It'll launch maybe this weekend? It supports user-trained data sets, which should be pretty neat.
Your approach sounds kind of like unit selection or vocaloid. Do it!! :)
It's coming soon.
I'm working on mocap, photogrammetry, and compositing on Twitch, and I'm leveraging vocodes for subscriber interaction in the stream.
Are either of you working in this domain? Want to chat?
I use(d) "Real time voice cloning" available on github. I have used your site before, i think that's actually the reason i wanted to find software to do it myself originally!
I ended up using mine to do the announcements on all of the ham repeaters in Central Louisiana. I'm not sure how many are still active, but i had a couple of podcasters and two presidents doing the announcements for a while.[2]
Nice work, wrapping it all in a website/service. I'll bookmark it!
[0] https://soundcloud.com/djoutcold/gottfried1-1
I love your use case for ham radio. That's brilliant!
And thanks. I'm glad there's other people messing around with voice cloning, still.
It isn't wrong to replace a human with a machine, I'm not going after calculators, spreadsheets, or farm machinery.
What is to come though is a moral imperative that making machines mimic people is deeply wrong.
Our worldviews, values, and judgements are based on interacting with other humans and to have that set of long-tuned intelligence modeling of minds around us subverted by a computer model will bring out strong objections and eventually violence.
The science fiction writers had it wrong, the first conflict with machines isn't going to happen with an advanced artificial intelligence, it is going to be against naive misbehavior and trickery much like this, especially when the artifice comes into the real world (first iteration: self-driving cars). Semi-religious anti-machine fanatics will be the first wave with terrorist style attacks on machines ... that is if the sane moderates don't do enough to rein in the coming of the artificials first. Wait for a smoking gun event like a car driving itself into a gathering of schoolchildren to get things started.
Until then, it's not the machines that are the problem. It's the people behind the machines, people making business decisions about operations and deployment of the machines.
The actual gun industry has somehow managed to avoid being sunk by this problem by keeping the blame on humans despite a school shooting every few months in the US.
Cars will be explicitly advertised that you can turn the automation on, go to sleep, and arrive at the destination, but if the automation hits anything the sleeping human will remain liable. The misleading advert will not.
https://twitter.com/OttaviaBourdain/status/14158894550057164...
What would happen between two people that sound similar, or even identical, to our imperfect human senses?
I'm not sure if it should go that way. I don't like the idea that someone like Disney could hire a voice actor once and then steal their voice forever. And it feels like we are headed down that path.
On the flip-side if you could trademark your voice, someone like Disney could buy up a bunch of trademarks and use them to sue somewhat similar voices. Kinda feels like a lose lose scenario.
That's only if you're using trademarked voices to begin with though, right?
The reason I think trademark needs to exists is because with voice, like an image or wording, comes a value or expectation of value that can be invested into it. Like Bourdain's voice.
There are two noteworthy limitations here:
1. The laws require a plaintiff to sue. If Bourdain left control of his public likeness to someone (like a family member or agent), they would have standing. I have no idea if anyone receives them by default if he died without explicitly granting them.
2. You don't need permission for using a person's name or likeness in relevant news or commentary. So this documentary might be protected.
IANAL, so if anyone knows relevant case law, please share.
And, of course, these are just the laws we have now. Whether we need new laws is iffy to me. I can just dislike this documentary using a generated voice because it's deceptive. Bourdain never recorded those words, but people watching the documentary could easily believe otherwise. It's not like TFA's example of a Civil War soldier reading a letter. I know he died before quality voice recording. And many documentaries label readings of modern documents with "Dramatic reading by actor" or similar.
[0] https://www.dmlp.org/legal-guide/using-name-or-likeness-anot...
She is not his widow. She is an ex-wife.
Getting away from the specifics of this case, cheap impersonation of arbitrary individuals getting easier raises lots of questions... all those TV spy shows "voice print cloning" stuff becoming more real will make a lot of things involving non-face-to-face trust more interesting.
Consider getting a voicemail from your dead grandmother tomorrow. Public figure or not, we need to have a line somewhere where it’s no longer okay to scrape a profit from every little thing we can think of. Can we let dead guys just be dead?
We hang photos to remember. A voicemail or more would help our kids hear grandma.
The snide remarks from the documentary maker; “We can have a documentary-ethics panel about it later.” make it seem like he is trying to get a fire burning around this.
To discuss it now would only benefit the commercial interests who purposely baked this outrage manna in their advertising oven.
In this case, for a documentary, I see how that’s a defensible used case. They could hire an actor to read the email, imitating Bourdain’s voice as closely as possible, to the same effect. Other uses certainly could be problematic. Discussing them and working out legal and cultural rules is very relevant right now. We’ve already had posthumous celebrity event appearances over the past couple of years. Famous actors could be in movies with needing to be involved very much - mainly an IP license. I have no doubt record companies would love to create new music with all the artists of the 60s-80s, dead or alive.
But on a more exciting front, I think synthetic voice can be made like fonts. Celebrities and voice actors will be able to sell their synthetic voice like how fonts are sold today. You'll be able to change the voice of your Alexa, Siri to a voice you downloaded from a marketplace. Netflix may even let you select the narrator's voice for your favorite documentary. Hundreds of years later, people may be watching a documentary in Morgan Freeman's voice while having no idea that he was actually a famous actor in the past.
The electronic signature is completely different: there is no visible artefact anymore, but a process that seals a document and (under appropriate legislation), certifies that the signer is who he is. A visual object is sometimes added for aesthetic purposes.
Because he didn't need to do it. Using a voice actor would have been completely standard and they would get a credit at the end, which I'm sure they'd appreciate.
And Bourdain definitely would not have liked it.
Bourdain didn't die that long ago, and made many recordings. It certainly is confusing.
I thought that given is ex-wife had not said okay and given Bourdain's past actions he may not have agreed with something like this using this technique was a poor choice, not necessarily that viewers are shown generated instead of recorded content
I didn't really follow Mr.Bourdain at all, but given the clarity from other people who did, and believe he would have hated this; then it's not the act of making a voice clone that's unethical it's doing it outside of the wishes, or what would be the perceived wishes of that person.
In other words this is unethical because had they gone to ask permission from the estate/whoever could make that call now that Mr.Bourdain is no longer with us they would have likely gotten a no.
It makes it particularly egregious that if these people were so interested in making this clone of his voice that they likely would have followed him closely enough to know he wouldn't have agreed to this if he were alive.
I guess you can boil my position down to without permission, explicit or implied, it's unethical and not only that, legally I'd put it under impersonation.
Some day we may be reduced to nothing more than a pile of ashes and a USB stick with our personality reincarnated into it. That's far better than simply being a pile of ashes, and may be the closest we get to some kind of immortality, a way to keep our ego wandering the digital world long after we've died and gone to wherever you choose to believe.
Combined with advanced AR technology, all it may take is putting on some glasses to see the ghosts of your ancestors wandering the real world and interacting with it. Well, maybe not our ancestors. We will be the ancestors, and our descendants will be the ones watching us, tending to our digital souls.
His narration of the documentary March of the Penguins really hit the mark for me.
People who complain about that sound a bit like those mythical tribes that are afraid of photographs because they may rob their soul.
EDIT: i'm not sure that this is wrong, only confused
We already DO have this same controversy about edited photo and video so there is no difference: most of the people who think this is wrong would also agree that creating a fake photo/video would be wrong.
It’s scary.
(I agree with you, I just have been corrected in the past with actual on-ground knowledge that hadn’t been reported yet - don’t know if that’s the case here :))
I don't know what the referenced video is, or if it is a deepfake though.
I've consumed lots of random media from many WhatsApp and Telegram groups. I've lost track of lots of it in the stream of information. That stuff doesn't get indexed on Google. The media narrative is tightly controlled, ministers claimed things were under control on TV whilst us on the ground knew this was false. Zello has also been extremely useful for community security. But trust has been an issue because of the open nature of these platforms.
I've been trying for this whole week to come up with tech solutions to this, but there are also social issues technology alone cannot overcome. Eg: Community members organised in adhoc security patrol groups engaged in friendly fire because they did not know they were friendlies. A telegram bot to query registration plates makes sense...but that leads us back to apartheidesque access control (no!) and the issue of trusting the entity that controls the db.
This is the best reference I could find of it in 'mainstream' (News24) media: https://youtu.be/Jq14RiZyJRA?t=3767
ps: n8n is a great tool!
Arguably Sassy Justice is worse, Donald Trump is alive, and I don't think Sassy Justice was an attack, but it wan't flattering either. Sassy Justice was not audio, somewhat clear satire but no warnings.
It reminds me when Boko Haram kidnapped the 200 Chibok schoolgirls, they killed and kidnapped 1000's of school boys first. You don't have to like Trump or care about boys but it can come back to bite you.
I just hope synthetic media in docos is pointing out to people doco's aren't the truth tellers people think they are. They are entertainment not purveyors of truth.
I don't want to go down the rabbit hole here but everyone hears about 'fake news', they know what it is, they just label things they don't like 'fake news'. I'm not sure that's helping.
Either way we are going to find out.
IMO we need to mandate disclosures around deepfakes. It's impossible on a peer-to-peer level, but commercially it should be clearly disclosed.
Once people get more experience seeing and creating deepfakes on their own they'll be less trusting of random videos they see in the future
GPs whole point about body image is that, as a society, we don't have that immunity. Basically no one has a healthy ideal of what a good body image is because we never even get to see them.
I live alone, and if I go into town there's a good chance I'll see more doctored photos of people than I see actual people (depending on how busy things are, which is usually not very).
There has been research on detecting photo manipulation and there's plenty of things people can watch out for if they're trying to "prove" an image was altered, but a lot of edits I see are so bad/obvious that the people making them aren't really trying to "fool" anyone with them. They just think it makes the photo look better. They'll do things like jack up color saturation to impossible/unnatural levels, or remove every pore from their skin, etc.
That's not true for many people (I'd go as far to say most), and we're so inundated with it that unedited media is the minority. Ad campaigns get PR for being "unedited" and even then they're heavily art directed (casting, lighting, styling, etc) to compensate.
The effects are so widespread that they're subliminal, even if you're conscious of the scope that they occur. Billboards, tv, movies, newspaper, magazines, products on shelves, menus at restaurants, wedding photos, family christmas postcards... it's inescapable.
Even if you're some paragon of mindfulness and truth in image editing and can somehow isolate yourself from its influence, you're still subject to it because of how it impacts the way everyone else behaves and sees the world.
We should learn from these mistakes.
Have you ever worked at a restaurant? It's not as unusual as you think. A lot of us on HN are in bubbles of savvy people because of our tech-related professions, and most people are NOT savvy. Many people never consciously think about the images they're subjected to.
It's not the norm, but it's not completely unusual. I've had many more people complain about the disparity in less severe terms. It's a very weird world out there, and if I've learned anything it's that I'm incredibly lucky to have any amount of self-awareness because a lot of people are running around out there on pure id... unaware of just about anything. If you're at all skeptical about anything you're ahead of the curve.
"The advertiser, and the person designing the advertisement, shall further ensure that the advertisement where a body's shape, size or skin has been changed by retouching or other manipulation, shall be marked."
https://translate.google.com/translate?sl=no&tl=en&u=https:/...
Even if they used a voice actor to mimic Bourdain, they should have informed the audience that an actor was reading the script.
Aren't the lines being read by robo-Bourdain actual things he write and/or said?
I feel like there's a difference there even if it is a bit weird to not credit robo-Bourdain in those instances.