Deepfake Offensive Toolkit (real-time deepfakes for virtual cameras)
github.com
github.com
Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse.
I'm not saying that they shouldn't have created this, or that they don't have the right to release it. But to create it, release it, and then pretend that any misuse was entirely separate to you is at best naive.
My only point was: you can disclaim anything you want, but that won't stop people from suing you if they feel like it (this can happen anyway, even if you disclaim any and all responsibility) and creating something that can very easily be abused and then thrown out into the public even though it is standard practice in the security community may well end up being a boomerang.
There is some precedent for the creators of such software being sued:
https://knowtechie.com/a-company-that-makes-iphone-hacking-s...
So I would not be too quick to dismiss that this could happen to smaller entities or even open source creators of such tools.
https://www.nytimes.com/2022/02/15/nyregion/sandy-hook-famil...
That’s not remotely the same thing as holding a creator liable for the misuse of his work by unrelated third-parties.
Frankly, the illiberal push towards “ban everything that could be misused” is disheartening, especially when we see it applied to a speech-based tool like this one.
Also: note that the degree to which everything is labeled 'speech' is fairly uniquely American.
(Rightfully) labeling code as speech is what allowed us to break down that barrier.
Would you also be questioning the ethics of djb in 1995? https://en.wikipedia.org/wiki/Daniel_J._Bernstein#Bernstein_...
Because once created something will have a life of its own. The question is only meaningful prior to creating it. This goes for the atomic bomb, the invention of gunpowder, strong cryptography and any other tool.
But if you don't care about ethics then it is obvious why you would not ask that question to begin with and I think that technology without ethics is no better than your average arms manufacturer. Only there at least we have - in some cases - some control over the uses, whereas software multiplies at a speed that politics can not catch up with. This may be a good thing, or it may be a terrible thing, and if it is the latter you will only find out when it is too late if you don't spend some cycles thinking about it beforehand.
I liked the internet a lot better before I had to think about the security implications of every move I made, and the difference between 'bad actors' that are incapable of wrecking everything and 'bad actors' that are empowered by the brain children of the good actors far more than the good actors are (due to the asymmetry between destruction and creation) may be more than we can deal with.
Individual empowerment enables incredible creativity, engenders individual and societal resilience, and leaves space for those who do not fit the norm to contribute their talents in ways we might not otherwise anticipate.
The second — prior restraint to enforce “trust and safety” - requires exerting authoritarian or paternalistic control over others to achieve its aims. In the process, it gives those who prioritize power over others the tools they need to undeservedly secure it, and invariably leads to individual, social, and institutional ossification.
Having things go wrong is a good and necessary thing; a perfect world is a fantastically dull and oppressive one.
These views did not arise from a lack of ethical consideration; they were a very considered ethical foundation on which the internet was originally built. In many respects, it was originally built to stand in opposition (or at least offer an alternative) to the staid and oppressive institutions that prioritize trust and safety.
The main purpose as far as I have it down was to create a method for (government) researchers to communicate and to have a reliable communications network in wartime.
https://www.usg.edu/galileo/skills/unit07/internet07_02.phtm...
Feel free to rewrite history any way you want it though. Perhaps you were confusing the world-wide-web and the internet?
Gopher, Usenet, and IRC. Eventually, the web.
Yes, the internet grew out of government projects, but it was rooted in the openness of academic networks.
The “trust and safety” ethos is merely the staid hall monitors of the status quo doing their best to co-opt something that was only possible to create outside their influence.
Its healthy for them to lose, sometimes. Maybe even most of the time.
Opposing the “trust and safety” hall monitors is an ethical position, just one that’s (seemingly) entirely contrary to your own.
This logically does not compute.
I guess you mean in situations where guns are currently used, other strategies could be employed?
Since the goal of a firearm is to hurl a chunk of metal through the air at high speed, I can't really think of alternatives. Besides air rifles, we just have traditional powder guns filling that niche.
The usefulness of firearms extends beyond killing people, and I don't think "guns are literally designed for killing people" is very accurate. I think the majority of guns are not designed for killing people. I think the Ruger 10/22 is the best-selling gun of all time, and it's .22 caliber, which is definitely not what you'd choose if the goal of your design was to kill people.
Guns are universally classed as 'weapons' not as 'metal object throwers'.
Not really, not for the vast majority of them. For hunting or sport shooting a bolt action makes sense, but a 9mm pistol is a tool specifically designed to kill people, just like an automatic rifle is.
We're software developers, not kindergarten teachers.
How do you feel about Sqlite's code of conduct? Because if you don't like that, then you shouldn't be making moral statements in your own repositories.
This is also true of almost every other area of human effort, whether implicit or explicit. As a catch-all baseline, we have the various legal systems.
The fact that a significant proportion of the tech industry refuses point-blank to even discuss the idea of ethics is not a sign of superiority but immaturity.
I don't agree, but people will never reach consensus on this moral topic. So please don't call the other side "at best naive".
Importantly though, that aim is served by proofs-of-concept and information sharing more than it is by releasing point-and-shoot attacks. This comment puts it rather well [1].
As above, I'm not saying this shouldn't have been created, or even that it shouldn't have been released. I am saying that there are important ethical considerations and questions raised that should be addressed directly by the creators, rather than being handwaved away as in the quoted disclaimer.
https://openai.com/blog/better-language-models/
So individuals working on a technology like this have no clear societal consensus to use as a guide when making the decision. Strong arguments can be made in both directions.
There will always be people who find the very idea of thinking about things from an ethical perspective offensive; that doesn't mean they're right. A lot of valuable discussion in this space is drowned out by people shouting down attempts to explore nuance and shine a light on the grey areas [1].
I'm not asking for censorship or a government crackdown; I do think that it's irresponsible to dodge the issues raised, act as though they are settled, or dismiss any concerns as anti-progress. As you say, there is no clear societal consensus, and there are strong arguments in both directions; what I want to see is those discussions taking place, in good faith, and those issues being acknowledged, rather than ignored.
[1] https://twitter.com/emilymbender/status/1532699418281009154
What we're talking about (I think) is releasing something with a net upside, but with potential for misuse and the resulting liability?
Am I misunderstanding your point?
I do think disclaimers like this are a little juvenile (it reeks of a US-ian litigation mentality), but you can easily imagine why they put it there. Perhaps instead of the author being less naive, you need to be more empathetic.
Personally, I like when people release proof of concepts, and hate it when they release weaponized tools. Especially when I inevitably end up reading reports where APT groups are using those tools.
I wouldn't term it a failure of empathy to ask that people consider the impact of their actions on others. I totally get why they have the disclaimer, and I wouldn't ask them to remove it, but I don't think it's enough. Given the clear potential for harm here, the potential uses and misuses of this tool should have been addressed directly.
People should be happy, that there are still white-hats who're reporting exploits, even for no profit. I personally switched to gray-market, can't take the shit we get from companies, anymore.
This tool released publicly is only to bring an awareness to the topic. Everybody else who needed to exploit this, have these tools developed in private.
The Apache Foundation - great example, thank you - despite not creating weapons, has clearly put thought into how their work interacts with wider society and how to ensure positive outcomes as far as possible [1]. People absolutely don't have to agree on moral issues, but it is irresponsible not to have considered them.
IMO the creators should feel good about their actions, even if they feel bad or apprehensive about the direction of the world because this technology exists at all.
That's an argument I have a lot of time for, but it needs to be made, rather than - as at present - having the whole issue sidestepped. With any tool like this, I think there's a need for ethical due diligence as much as there is for engineering, and there's no evidence that this aspect has been considered at all.
Who you release things to, in what form, with what framing - all of these impact how something will be used. You can never eliminate bad actors, but you can encourage responsible use.
Probably same on macOS, but it's (mostly) not open source, so you can't really argue about it being an open source thing that lets you do evil stuff. You could argue about its availability though.
And it boils down to the point you (and others) were making: these are tools, the ethics are in how you use them.
[0] I know there are some forensic distros that come with anti-forensic tools. Still, Linux in general is the environment that allows that kind of thing to work in the first place, going back to the original point of whether it should be open source or not.
“Tool is expensive” is a real, effective deterrent to frivolous use of a tool. There’s a reason we pool resources into governments, which is explicitly so we (via a government) have capabilities that we (as individuals) don’t have.
The OP specifically said tools "like this"
While it’s convenient to dismiss any discussion about A and something slightly similar to A as “slippery slope,” an inconvenient fact is that the slope often is slippery. It’s increasingly evident to anyone paying attention that information warfare can yield disastrous effects in a variety of ways: disrupting peaceful transition of power; disrupting critical infrastructure; disrupting command and control of ”traditional” WMD systems, even.
So that’s sort of the spirit of my question: how far does this argument go? What is the definition of “like this?”
It's honestly an interesting moral question.
Do we aim for security through obscurity or work out how to deal with that increasingly terrifying reality? I think at some point we will be forced to do latter.
Maybe an interesting norm to try to generate in this community would be, “if you plan on publishing attack X, you should publish immediately alongside it your best guess as to the effective counter to X.”
In the case of nuclear weapons, one of those consequences is that poor countries are denied the ability to defend themselves.
This led to the bullying and invasion of countries without nuclear weapons by countries with nuclear weapons (most recently Russia invading Ukraine, but China and the US are equally guilty).
I'm reminded of a line from the TV show The West Wing, something like: "India must and will be a nuclear power. That way, we'll never be dictated to again."
We as a species have only survived ~75 years since we became aware of nuclear weapons, which is only a small portion of our overall history.
This has taught me to be mindful of the state of being unaware, because once you are aware of something it is impossible to reverse unless you're an amnesiac. Collectively forgetting something as a society then becomes infeasible without the technology and the willingness to revert our knowledge to a safer state, and a single outlier can ruin the entire system. When Jobs held up the first smartphone and we all started cheering and fantasizing, did any of us think it would lead to this?
(I also think that a pill to forget the experience of [addictive thing] would be revolutionary, but sadly our ingrained curiosity might undo the effect shortly afterward. A lot of tech takes advantage of our curiosity.)
This makes me wonder: what compels us to keep researching AI despite us also being able to hypothesize all these game-over scenarios? Why does it feel like we have no choice but to be crushed by unbounded progress?
Perhaps 50 years from we may find that nothing short of halting all research in the name of curiosity was the only option that could have preserved our culture/existence/values for another few decades.
Jobs didn’t hold up the first smartphone and smartphone proliferation has little to do with the AI research that led to deepfakes.
I'm sure there are some very worthy technical efforts to improve mankind that are being hampered by the lack of easily accessible U-235 so why are we restricting this material from Chemists that want to Chem and Physicists that want to Fizz?
That may not be a fair comparison. Nuclear weapons are difficult to make, and using them will not go undetected - the damage would be enormous. Deep fakes are a real threat today, and anyone willing to pay for the expertise (which isn't that hard to find) can have them made. By putting this out there, deepfakes are now something almost anyone can make which means the threat is imminent and needs to be addressed. I understand the argument, but still not sure about it. Just wanted to say it's different than nuclear weapons.
Now that the public knows broadly that digital manipulation is easy and they have seen it with their own eyes a ton things like that are believed less. Notice I do say less though, we still definitely have problems with misinformation being propagated with shitty-looking video/audio/photos that people should not believe in because they look/sound so inauthentic.
All that to say, these tools clearly exist. I think the public is better served by them being pervasive and well-known rather than having them be locked up in Hollywood studios, research labs, and government agencies.
In a perfect world, what should have happened was a separate team inventing a reasonably accurate deepfake detector to the world before the deepfake producer gained prominence (differential technological development), but interest, productivity and excitement on the researchers' part to release their work first seems to have foiled that.
It may be worthwhile to have someone create a bunch of deepfakes of politicians saying "This video has been faked" to drive awareness of this as a threat model.
E: Forgot the key bit, it would need to be spammed all over Facebook/Instagram/Mainstream social media.
An authorised firearms officer (AFO) is a British police officer who is authorised, and has been trained, to carry and use firearms. The designation is significant because in the United Kingdom most police officers do not routinely carry firearms, although they can be equipped with tasers. In 2019/20 fiscal year, there were 19,372 police operations throughout England and Wales in which the deployment of firearms was authorised and 6,518 firearms officers, 4.9% of the 132,467 active FTE officers. Following the November 2015 Paris attacks it was decided to significantly increase the numbers of armed officers, particularly in London.
It’s surprising how often this argument is made compared to, say, equal-opportunity access to fancy cars or good housing.
That isn't a defensive use!?
The cat is already out of the bag. Going "nooo, unpublish this" doesn't protect people from scammers, who will continue to use it.
Society must unlearn that seeing is believing. That is what is morally irresponsible, not this software.
Knee-jerk refusal to even consider thinking/talking about the use of technology is far more irresponsible than any other stance.
Even beyond that, if you're an authoritarian nation with a rather good lockdown on your news and internet, what's the downside of throwing tools like this at democracies?
In my view the age of tools like this has already been here for some time and they are going to their next step of evolution, and there is very little we can do about preventing the distribution of them without affecting other pieces of software and their distribution.
> Society must unlearn that seeing is believing.
A total lack of trust of our shared perception? Let me know how that works out. Might as well have everyone on acid 24/7.
Videos aren't the only way to understand the world, and there was never any such thing as "direct observation" in the first place.
Now consider a few months after this hypothetical fake-and-refutation, what if Biden actually does announce an emergency threat to America, and people end up waiting a few days (or weeks, or months) for the refutation to come.
The “no such thing as trust” is not coming from this innovation in particular but rather the confluence of two general trends: 1) heavy reliance on digital media and 2) ever-increasing ability to fake digital media.
Note that you don’t even need large, successful falsified media campaigns to create distrust. All you need is for people to know that the capability exists, and trust is immediately damaged. Even trivially-detectable falsifications can end up being difficult to sort through if there are a billion fake pieces of information crowding out a hundred real pieces of information. This is true even if all billion fakes are trivially detectable by a human (which they won’t be).
I too dread a little all the mischief that's going to happen while the world learns about this. But people don't have to uncritically believe "the evidence of their own eyes", even though that has been very useful corrective to many mistakes, and will I guess be less so now. Evidence doesn't work that way: people always have to interpret everything in terms of their understanding of the world, and they can get better at that in response to changes like this.
The advent of widespread photo/video evidence was a big help for people trying to interpret was real and what wasn't. Then photography was knocked out of service by e.g. Photoshop (generally "easy photo manipulation"). Now video is getting knocked out of service as well. So, yes, it removes a useful tool in interpreting the world's evidence, and it's not clear what will replace that tool (if anything).
To be clear, the "no such thing as trust" was specifically a response to GP's assertion that people will "have to unlearn that seeing is believing." Okay, fine, but then what can you believe if you not even what you see? Again this specific tech doesn't get us all the way there, but it gets us closer.
There are ways to prove authenticity, e.g. with cryptographic signing.
People have been aware that lies can be typed for as long as we've had typing, but it still works. The fact that it still works a lot of the time hasn't ended the world. People are more aware that they can't trust a photo, yet people still fall for edited photos, and the world still turns. I don't think deep fakes are any different. Videos were not very trust worthy before deep fakes. Special effects have been a thing for many many decades. The world is still learning exactly how untrustworthy video is and there will always be people who fall for tricks and manipulations, but eventually the majority will adjust and society won't crumble.
We'll put our trust in what we see with our own eyes and in people and organizations who have proven themselves to be trustworthy. We'll worry about how real something is in proportion to what it would mean to us if it were real. When it really matters we'll trust experts to authenticate a photo or video as legitimate just like we do currently. If the world becomes less dependent on information presented in formats that can we know can be convincingly manipulated we'll all be better off. History tells us technology like this in the hands of the public won't cause people to abandon faith in everything and anything they see, or destroy civilization. We all just get a little more savvy, more industries will take advantage of the cool tech, lots of people will have fun and games with it, not much really changes for most people.
No one said it'll destroy civilization, don't worry.
It's no different from religion, which was the backbone of the western civilisation for quite some time.
For example, religion claims you have free well. For a materialist, any interesting definition of free will is impossible, because we are made of quantum particles with defined statistical behavior.
Of course, we cannot prove or disprove the existence of free will in other people either…
Seems like a good thing.
Should that argument be made to the original research? I don't think it applies to one but not the other.
I admit that this limit is hard to define. It’s also extremely hard to enforce for pure data, if that turns out to be what we want to do.
We produced the software to run the drones. We didn't personally deploy them to kill those children.
We built software to guide the rockets. We didn't personally fire them at those hospitals.
We chose to startup our company providing profiles of suspects using scraped, publically available data to any government agency. What they do with the information isn't our problem.
We wrote the software, but it's not our fault or problem what happens because of who uses and because of how they use it. Our intent was good. The market and demand was there. What's wrong with providing a supply? How many software creators feel culpable? Vanishingly few. Who cares when comp is high?
We can do that. We do it all the time, assigning varying levels of blame to different parties for different things.
Determining between proximal and ultimate causes, or assigning more weight to one cause or another, is not some impossible burden.
That's a dangerous precedent. Would you apply the same logic to a kitchen knife? Or if for some reason only freemium products count (not sure why), then a pentesting tool?
I understand the underlying point you are trying to make but what are you proposing as an alternative exactly? Who gets to decide which products fall into a gray zone, whilst others are only for bad use. We already see this kind of shoddy thinking leading to keeping DALE-2 out of the public's hands (or at least that is their claim).
Regardless, I'm not proposing to ban either knives or this tool - I've been very clear about that. I do think that - with anything that has potential for harm - it's important for people to consider the possible consequences and to actually engage in the discussion about usage, rather than either washing their hands of the issue or declaring that any consideration will soon lead to arresting everyone for everything.
This is a path we've trodden countless times. With some things, we've collectively decided that no controls are necessary. With other things - poisons, nuclear weapons, a host of things depending on location - society has enacted controls of various efficacy and validity.
The responsible choice, regardless of whether you end up being for or any any given restriction for [thing], is to spend time thinking about it, discussing it, and - particularly when you've chosen to release something - acknowledging the potential issues.
I think I agree with everything you wrote except what I'm unfairly reading into this phrase: we don't need to and should not irrevocably "decide" once and for all. We might have been quite wrong to ban some poisons even in the situations that applied at the time when we banned them. If we were right to do so, times may have changed and we don't any longer. Even nuclear weapons perhaps, in certain situations.
Similar to a previous comment of mine in this thread, of course I don't know from your passing use of that word what you think about this! But I do think that that worldview, that these things are or should be "decided" in a permanent sort of way, is commonplace.
It's wordier, but more accurate, to say that I think society is (and should be) in a constant conversation with itself, deciding what to prohibit and to what extent. We regularly get this wrong, and we should never assume that our current collective position is unassailable; there should always be room for doubt and further discussion.
Make the tools available to everyone. As Elon musk says, sun is the best disinfectant.
Power however, is amoral and neutral
I am closer to your line of thinking than not, but, at the same time, from where I sit, it seems that data monitoring went too far already and regular user has to have a way to circumvent it.
For that reason alone, this tool, even if it will result in some abuse, it just evens out the playing field.
We should do the exact opposite. Science and reason would cease to exist otherwise. Individuals wouldn't need to be held accountable, innovators/engineers/scientists/entrepreneurs would be. End of a free society.
Have you thought of the chilling effect this might bring?
I've been extremely clear throughout this comment section - I don't want censorship, I don't think this should be banned, and nowhere have I called for legal consequences for releasing this. I want people to think about their actions, and to discuss the ethical issues arising from them.
And yet you've jumped to calling me morally corrupt because I want to murder craftsmen. That's not a reasonable reading of my comments, or in any way a proportionate response.
If you want to talk about chilling effects and the importance of science and reason, how would you describe your comments? You're shouting down discussion with wild accusations.
Absolutely zero to do with you personally. You're building a strawman.
To you personally: If you're going around touting "other side's morality", you should be open to seeing what happens when this kind of thinking is adopted by the society. Don't play the victim game.
This isn't so much a strawman as a slippery slope made from them. It is not reasonable to leap from 'people should acknowledge and discuss the ethics of their actions' to 'blood-drenched tyranny'.
I am arguing that your case is fundamentally flawed, irrational, corrupt, inconsistent and dangerous.
Edit: I can't respond to your slippery slope fallacy but you started with an extremely broad stance "Anything that can be created, will be created. However, that doesn't free you from all moral culpability." I am responding in kind with an extremely broad implications. I guess we're busy debating which fallacy falls in whose arm with not addressing what I countered your argument with. First came the victimization, then a strawman card, then a slippery slope fallacy card. But you never considered responding to what happens when we take your ideas and consistently apply them. I still don't understand why you feel like you're being accused. I am arguing about the points you raised. No one is accussing you of anything.
If you want to argue against my case, then please do that, starting by engaging with the position I'm actually advancing [2]. It's difficult to discuss morality at all when you're deliberately engaging in violent rhetoric rather than with me.
[1] https://fallacyinlogic.com/slippery-slope-fallacy-definition... [2] https://news.ycombinator.com/newsguidelines.html
The inventor of knives? The knife‘s manufacturer? The store who sold me the knife? I would say the responsibility lays 100% with myself.
I do not think it makes sense to pursue inventors for what happens to their creations, unless they actively encourage misuse.
Should McDonald's be banned? I do not think so. Anything can be misused.
To answer your question about what "good" uses there are for deep fakes: Cheaper and better special effects, available to a MUCH broader audience than before. I cannot even imagine all of the great use cases creative people will find for this.
Yes. I believe it should.
Regarding the deep fakes being good, I meant societal good. Not just fun tacky things that some people have a good time with. What social value does it bring to the world? We can do everything we need to without deep fakes. It has no value to the world.
Do guns kill people or do people kill people? Could be a relevant analogy.
https://www.npr.org/2022/06/03/1102755195/uvalde-special-ed-...
Sorry can't resist
For instance, I once cheated by using gcc/godbolt to generate assembly output for a class from C code. By this logic, Richard Stallman should be blamed for my misconduct.
There are any number of reasons for supplanting another face onto your own, many of which are simply good fun. If you choose to use this for scamming or perverted reasons, so be it.
Moral posturing aside, perhaps there could be an invisible watermark or something included by default to easily identify less technically inclined actors as users of this tool.
Unjustified empirical claim.
And even if it were true, it would still be insufficient for justifying public disclosure, classic is-ought.
Also, only nerds stand up for security nerds.
No. You don't.
No, it is naive to pretend facial recognition is worth anything when creating tools to defeat it is a mere academic exercise in reading papers and implementing algorithms described in those.
Don't shoot the messenger.
irrelevant
That way, the software could be used for pen testing, but it could cause a silent trigger to go off.
That could even be a way to monetize the software....
Axis has developed a way to cryptographically sign video, using TPMs (Trusted Platform Module) built into the cameras and embedding the signature into the h.264 stream.[1] The video can be verified on playback using an open-source video player.[2]
I hope this sort of video signing will be mainstream in all cameras in the future (i.e. cellphones etc), as it will pretty much solve the trust issues deep fakes are causing.
[0] https://www.axis.com/ [1] https://www.axis.com/newsroom/article/trust-signed-video [2] https://www.axis.com/en-gb/newsroom/press-release/axis-commu...
I'm not so sure. How will you verify a signature if you see a video on TV or on social media? Do you believe these devices are 100% secure and the keys will never be extracted?
> It shouldn't be too hard to film a deepfake movie from a screen or projection that don't make it obvious it was filmed. That way, the cryptographic signature will even lend extra authenticity to the deepfake!
Would you even have to go that far? Couldn't you just figure out how to embed a cryptographically valid signature in the right format, and call it good?
Say you wanted to take down a politician, so you deepfake a video of him slapping his wife on a streetcorner, and embed a signature indicating it's from some rando cell phone camera with serial XYZ. Claim you verified it, but the source wants to remain anonymous.
I don't think this idea address the problems caused by deepfakes, unless anonymous video somehow ceases to be a thing.
Similarly, it could have serious negative consequences, such as people being reluctant to share real video because they don't want to be identified and subject to reprisals (e.g. are in an authoritarian country and have video of human rights abuses).
The signature of the politician on a street corner would only be valid if it actually verifies the content of the video; and the only entity that can produce that signature is the holder of the private key.
That's what I meant: sign the video content with some new key in a way that looks like it was done by a device.
> The signature of the politician on a street corner would only be valid if it actually verifies the content of the video; and the only entity that can produce that signature is the holder of the private key.
The hypothetical politician's encryption keys are irrelevant. The point is that even authentic video is going to be signed by some rando device with no connection to the people depicted, so a deepfake signed by some rando keys looks the same.
IMHO, cameras that embed a signature in the video stream solves some authenticity problems in some narrow contexts, but it's nowhere near a panacea that "will pretty much solve the trust issues deep fakes are causing."
Of course, if one has physical access to the camera/sensor it's possible to make a fake video with a valid signature. But it's a little more difficult than simply running a deepfake script on some machine in the cloud.
If the goal is to allow a manufacturer to confirm video came from one of their cameras, I think it's somewhat more helpful than I was originally thinking, but it doesn't change my opinion that this technology would only "solve some authenticity problems in some narrow contexts." Namely, stuff like a burglar claiming in court that security camera video was forged to frame them. I don't think it addresses cases of embarrassing/incriminating video filmed by bystanders with rando cameras and other stuff like that.
> Of course, if one has physical access to the camera/sensor it's possible to make a fake video with a valid signature. But it's a little more difficult than simply running a deepfake script on some machine in the cloud.
If you're faking a video like I described, you certainly would have "physical access to the camera/sensor" that you claim made it. You're making a fake, which means you can concoct a story for the fake's creation involving things that are possible for you to acquire.
So, the best you get is that the stream shows it wasn't produced with access to a particular camera. Then impersonating a YouTuber, say, requires access to their physical camera.
Yes. But since you can't really prove who had and who did not have physical access to the camera (what are you going to use, video evidence ;) ) that carries little weight.
Let's say we're talking about a death penalty or a multi billion $ lawsuit, the stakes would be so high that to rely on something as trivial as a signature on a video stream would be way too risky because of the consequences.
Which expert would sign on the dotted line to the statement that 'the equipment has not been accessed by someone who wasn't authorized to do so'? And that's before we get into the question of whether this camera really is the only way that this particular video could be generated (can duplicates of the TPM be made by the manufacturer?), you'd have to do a lot of work to make guarantees that would stand the test of time.
All it takes is one 'DVD Jon' and you can kiss your carefully constructed scheme goodbye.
Finally, the whole chain of that video would have to preserve the signatures, which in a world that tries to balance privacy with other concerns may not be feasible and in fact could well have downsides all its own.
Personally I would much rather see the level of skepticism against all kinds of digital evidence go up than to see the trust go up due to supposedly magic signatures and other tricks to pretend that we have a perfect lock on this stuff. Every time we believe we do someone comes along that proves us wrong, usually after the damage is done. See also: fingerprints, face recognition, DNA evidence, lie detectors, bomb detectors and so on.
You have a lot more trust in the ethics of expert witnesses than I do.
Expert witnesses and liability are a very interesting area of the legal system. There are all kinds of variations on this theme ranging from blanket immunity to various forms of liability. Knowingly giving false testimony would likely lead to problems, especially if say a panel of experts would in turn testify that said expert should have known and could have known that what they testified was wrong.
I'm not aware of a case that went that far, but I would love to read more about such conflicts between expert witnesses on two sides of a lawsuit, usually the experts agree in broad terms on their testimony, and may differ in the details.
Maybe some who have served in such a capacity can chime in here?
I don't think you can show the reverse (it was you), nor can you make a 'proof' without an established chain (you're only proving that you verified something with the same equipment you previously used).
Reputation seems like it could become much more a part of social media, like, this video was signed by all these witnesses to endorse it as a true account.
Now you create a situation where hardware bypassed cameras have immense value to bad actors.
Yes that's kind of the point. Plus I'm sure they could put the whole camera in a tamper resistant case. They could make it very difficult to access the sensor.
Including focus data should make "record a screen" a bit harder too. I guess recording a projection would be pretty hard to detect, but how likely is it that people would go to those lengths, vs using a simple deep fake tool?
Please don't use quotes when you are not quoting.
This is exactly why all these identifiers and all this tracking are harmful: you can never be anonymous, there’s always someone watching who can couple everything you do to your identity. And it happens with you knowing as little about it as possible.
It's a nice piece of tech, I can see it being used in court for example, to strengthen the claim that a video is not a deepfake.
However, that's not "the" problem with deepfakes. Propaganda of all sorts have demonstrated that "Falsehood flies, and the Truth comes limping after it". As in, with the proper deepfakes, you can do massive damage via social media for example. People re-share without validation all the time, and the existence of deepfakes add fuel to this fire. And I think that we can't do anything about either.
But most videos where we're worried about deep fakes are videos that come from other people's cameras, where we don't know which signature should be valid, nor whether it should have a signature.
Signatures don’t change the provenance problem.
It seems like official channels like CSPAN, white house press briefings, and other world governments could get immediate benefit by applying these methods to all of their official communiques.
I hope deep-fake detection software can compete with deep-fake generation software, I've been tracking this need-gap on my problem validation forum for a while now[1].
That said, There are ethical usages of deep-fake videos as well; In fact I might checkout this very tool to see if I can use it for 'smiling more in the videos', remembering to smile during videos is exhaustive for me. There are other ethical usages like limiting the physical effort needed to produce video content for those with disability(like myself)[2].
[1] https://needgap.com/problems/21-deep-fake-video-detection-fa...
[2] https://needgap.com/problems/20-deep-fake-video-generating-s...
I'm sick snd tired of seeing big companies and orgs (Google is the most recent) publish an amazing application of ML but refuse to release the trained model because the model is biased and may be used in a bad way.
It's not like they release their state of the art ML stuff when there aren't any ethical issues anyway, e.g. for voice recognition.
That being said, bringing it to light also has benefits like you said. If the tool is out in the open and state of the art techniques are used, technology to detect its use will also benefit.
Democratizing access to things -- including bad things -- has a preventative effect:
1) I can guard against things I know about
2) People take me seriously if something has been democratized
The worst-case scenario is if my stalker got her hands on something like deep fake technology before the police / prosecutor / jury didn't knew it existed. I'd probably be in jail by now if something like that ever happened. She's tried to frame me twice before. Fortunately, they were transparent. She'll try again.
Best case scenario is that no one has access to this stuff.
Worst case scenario is only a select group have access, and most people don't know about it.
Universal access is somewhere in between.
And when you ask for help, people think you are the insane one because they simply can't believe your story about the insanity of someone else.
I hope you find relief sometime from your stalker. I found it (not a stalker exactly) from letting the person burn themselves with their behavior so many times without me doing or saying anything in return to them (my strategy of non-direct conflict, and it worked for me), that eventually they ran out of people to manipulate and fool.
Obviously, it's hard to imagine many situations like that, but you can imagine a process that required a 8-figure quantum supercomputer.
https://www.smithsonianmag.com/history/spies-who-spilled-ato...
For better or worse, I did not have the requisite $2B-or-so to do so at the time. My allowance was $2/week. I don't think most terrorist groups have $2B either. Even if I did have the requisite $2B, I don't think I could have done so discreetly enough to not get caught; someone would have noticed a high school student buying up things like uranium or building appropriate facilities.
I'm more concerned about biological warfare. While I don't have the knowledge to engineer a super-virus, we're at the level where a few years intense study is all it takes to have that knowledge, and a few high school students do. The cost structure there is increasingly moving into the budget of high school students too. Things like DNA sequencing and synthesis have a Moore's Law-style curve, with falling prices.
(If this sounds unrealistic, I was in high school back in the days when nerds intensely learned physics; physics is much less interesting to nerds in 2022. Hot topics for nerds change over time, and we're out of the cold war and the space age. Today's high school nerds have moved on to today's hot topics like machine learning and microbiology)
not to be too pedantic, but this is not "democratizing access", as that would involve print-outs/usb sticks/discs of the code distributed to people that can't access the internet, accessibility issues, bias considerations etc etc. as such, this is just "access".
Likewise, slavery < serfdom < caste / Jim Crow / etc. class system < equality under the law with discrimination < equality
Access, inclusion, and democracy aren't binary concepts, and progress is gradual.
I suspect this perfect deep fake technology might be a real boon to society.
Continued development towards a tech dystopia might be the only way. But I’m worried it will require the dystopia part before people will wake up and accept that moving back to the physical world might be required.
The next few decades will certainly be interesting, at the very least.
Making such tools accessible is reprehensible. It will lead to more bad actors, to less trust in media and in any objective reality, and more erosion of our institutions and society.
There is absolutely no reason whatsoever for this. It's unethical and frankly downright evil.
You’re being absurd.
The technology exists, and it’ll get better. Pretending that it doesn’t exist or banning it won’t make it go away — it’ll just be used by the least scrupulous and most powerful.
Disruptive efforts like this are most upsetting to anxiety-ridden people who think that if they could just control things firmly enough, everyone and everything will be safe.
That kind of thinking doesn’t actually work, though, and it produces a stiflingly rigid, oppressive society that deserves to be upset occasionally.
But: are those really the only options? Paranoid stop-the-world wishful thinking or all-new-trends-are-inevitable?
I realise I'm almost certainly wildly misrepresenting your point of view with the latter label! I don't really mean to ascribe it to you. But I think the conversation often ends up tacitly as a debate between those two positions, both of which are mistakes. But one other option -- as an example -- is to find more complicated ways to stop doing <bad use of tech>.
A striking historical example to me because it's so old it can hardly be said to have been an invention that needed to be responded to -- rather just an apparent invitable fact of life: For the longest time, it seems everyday brutal violence on a level we find hard to imagine was common. At the time, that would have been obviously inevitable. Nothing else had ever happened. Then it basically stopped -- turns out it never was inevitable. Why? We didn't for example ban knives as a technology, but we did a lot of things over the centuries that, to a good numerical approximation, stopped us using them for violence (banning them in certain situations is only one, I suspect not the most important and maybe not even necessary).
In ages past, impersonating another person was not nearly as difficult as it is today. Advances in identity verification and such have eliminated a certain class of problems. The emergence of deepfakes doesn’t necessarily introduce “new” problems, it just resets the progress on some very old ones.
Assuming this is solved not by banning the tech but by making progress elsewhere, some pretty worrisome implications come with that. One antidote to deepfakes would be even more progress on identity verification and tracking the source of digital media for purposes of authentication / validity.
It’s not hard to imagine the negatives in such improvements at a time where we’re already tracked far too much.
I think a more ideal development would be a return to pre-tech habits. Meeting people in person. Less reliance on virtual communication, etc. But one need only look at the backlash against returning to the office to see that such a move would require something truly existential.
I suspect you are right in the long run, but I’m curious what other factors or forces might nullify the threat introduced by this tech in the short term, or if things really have to run off the rails before people decide that fundamental changes in how we interface with tech are required.
I don't disagree that it can have that effect.
Just to note: this use of "turn back the clock" is quite distinct from "turn back the clock on technology".
> Assuming this is solved not by banning the tech but by making progress elsewhere, some pretty worrisome implications come with that.
My point was not that we can make progress elsewhere to solve resulting problems (we can) but that we can also not use the technology in the "inevitable" bad way. It happens!
> One antidote to deepfakes would be even more progress on identity verification and tracking the source of digital media for purposes of authentication / validity.
One other antidote (of many I'm sure) would be to do less identity verification in the first place.
Institutions that are corrupt and serve themselves (not the public, despite their lipservice) need to go.
Trust in media is already very low, and in fact should go lower. Deepfake tech exists, and the fact that it does, and is broadly available to bad actors, should be widely known.
Obviously the best case by far is that such weapons (in this case disinformation weapons) do not exist, but the worst case is them existing but hidden — THAT is the recipe for fooling people in the greatest numbers.
These tools existing, with widespread knowledge of their existence is sort of the least-worst case for the real world in which we live.
Yes this does have the potential to kill pretty much anything related to video and photography (everything from art to news to documentation), but that was the same when spam was a literal threat to existence of email. Unless we manage it, video and photography will be trusted for nothing but boring amusement; but better than than mass deception.
It only buys time, but that can provide the time needed to create countermeasures and ideally make those very accessible—somewhat similar to responsible vulnerability disclosure.
This piece goes into more detail: https://aviv.medium.com/the-path-to-deepfake-harm-da4effb541... (excerpt from a working paper, part of which was presented at NeurIPs).
Also, the notion that "raising awareness" is going to prevent deep fakes from being used in practice shows complete and utter disconnect from reality. Most people who are skeptical are already aware how imminently fakeable all the media really is. Most people who still are unaware will remain so, no matter how many GitHub repositories some dipshits will publish.
"I am Cofounder and CEO at Sensity, formerly called Deeptrace, an AI security startup detecting and monitoring online visual threats such as “deepfakes”." (one of the contributors of this repo)
1) It will give security researchers more freely available technology to work with in order to try and fight the malicious use of deepfakes. (I saw some interesting comments in this thread about TPM. It'd be interesting to see what other solutions are out there.)
2) It would raise the overall awareness of the general population about the existence and advancements that deepfake technology has made. I would argue that a small subset of the overall population know what the term "deepfake" means, and even fewer are aware at how far it has progressed in only a few short years. (I'm not super well versed in the topic myself, I just know that I've heard a lot of progress has been made.)
I think that since this tech is already actively being used by bad actors, the best course of action that we can take until at least a somewhat good counter to it has been adopted (and then quickly defeated) is to make as many people aware that this is something that could affect them, or their families. That this is something that could be used to get someone fired, or hurt, or killed. I think that the more that people are aware of its existence, the less impactful the overall effect of deepfakes becomes. People learn to look twice before making a call on something, because of how easy it has become to fake audio and video.
Basically any pentesting software can be abused.
It would be much more interesting to discuss the technical merits of this project than any particular moral concerns you might have, especially on this forum.
That's mostly your impression. But that may be because you believe that I have a position with regards to this particular software, which I do not, I just see this as an opportunity to gain insight in the position of the author which I find interesting and which may help guide me in similar decisions in the future, because it's something that I've been wrestling with for a long, long time. Since 1995 in fact.
> It would be much more interesting to discuss the technical merits of this project than any particular moral concerns you might have, especially on this forum.
What you find more interesting and what I find more interesting do not necessarily have to be the same things, and you are totally welcome to ask your technical merits questions.
Finally, I don't like to be told what I can and can not discuss, especially not in a thread started by someone who wrote 'AMA' where the third A stands for 'Anything'.
There's no right answer to a lot of the dilemmas raised, and people will reach their own conclusions and choose particular principles to follow. I don't want to tell people how to think, but I do think that everyone has a responsibility to actually think about these issues, wherever they end up landing on them.
This is a cop out. Yes, it is a matter of time, but still it takes people to build this stuff and they are rarely the people that use/abuse it.
> But reality is that unverified video is no longer a trustworthy medium.
No, video is no longer a trustworthy medium and in many ways never was.
Yet this question seems unfairly loaded. At the least - premature.
Though we can easily imagine it, afaik there is no evidence of any actual abuse to date.
I've often wondered whether the net effect of the whole security research community is a net positive or a net negative and I honestly do not know the answer. So yes, it is a loaded question. But asking yourself if what you can do is actually the right thing to do is always good, especially if you - as these authors imply - know up front that there is a large chance of abuse.
I just hope the person offering AMA will not be daunted by a good but blunt question from a voice that carries weight around here.
Hence my question. Because I do feel that weight and it has caused me to carefully consider the abuse potential of the stuff that I've released since then and I've only released those things that I feel have none that I can (easily) discern.
Which is one of the reasons I'm so completely against software patents and a large number of patents in general. Quite a few of them are simply things that the time is right for.
Here is one of my idea dump lists, you can check for yourself which ones are not yet done (which is probably a really small number by now) and which ones have turned out to be homeruns (and in some cases billion dollar+ companies).
https://jacquesmattheij.com/my-list-of-ideas-for-when-you-ar...
One that wasn't on there eventually led to https://pianojacq.com/, which I'm happy to report to date has not led to any kind of abuse. And no, it did not put any piano teachers out of business either.
Software engineers and other technology creators don't take a "Do No Harm" oath like doctors. Many of them have never even taken a single Ethics In Technology course at university (it was an optional class when I was in undergrad decades ago). And, even in the alternate universe where ethics was baked into engineering training, all it takes is a single rogue willing to ignore them, and now the world has to deal with it.
In the end, I decided that one oppressed person using it to improve their situation is morally "worth" many criminals using it. It's kind of like "better ten guilty men go free than put one innocent man behind bars".
Because the one innocent is usually a defender, whereas the guilty men are usually the attackers and to give the attackers an effective advantage in an arms race is a risky thing with unknown and potentially devastating outcomes.
In my own life (the live video example listed elsewhere) it would have meant that we probably would have had everything that we have today anyway, only maybe a little bit (not even that much, I'm aware of one other individual who was working on a similar concept who contacted me after our release) later.
Video conferencing, live streams in the browser without plug ins and so on all would have happened, for sure. But at least the massive mountain of abuse cases would not rest partially on my shoulders. And because I've been confronted with the direct evidence of the results of my creation for me that link is easy to make. But if you work on secure communications software you are probably not aware of the consequences.
Please read this https://www.biometricupdate.com/202205/sensity-alleges-biome...
We have warned many vendors about the vulnerability of their commercial biometrics software. The threat is currently downplayed by the whole industry. We hope this release to be a wake up call and that our team will be joined by other experts in raising the alarm.
Deepfakes are already used for spoofing KYC around the world. This is already happening, and not by using `dot`.
I distinctly remember a very similar discussion around firesheep 12 years ago: https://www.computerworld.com/article/2469667/firesheep-fire...
https://news.ycombinator.com/item?id=1828955
> There are probably going to be a lot of people negatively affected by this for quite some time to come. One thing to point out is that there are grades of things. There is "public", and then there is "top hit on Google". Similarly, there is "insecure" and then there is "simple doubleclick tool to facilitate identity theft".
> How many millions of dollars and man hours is it going to take to lock down every access point? How many new servers are going to be needed now that https is used for everything and requests can't be cached?
I'm not OP but I would answer like this: abuses of this technology are inevitable and can only be mitigated by counter-software (which leads to an arms race).
The release of this source code could kickstart the development of deepfake detection software.
Or maybe in general people need to put less weight on video evidence.
We have warned many vendors about the vulnerability of their commercial biometrics software. The threat is currently downplayed by the whole industry. We hope this release to be a wake up call and that our team will be joined by other experts in raising the alarm.
Deepfakes are already used for spoofing KYC around the world. This is already happening, and not by using `dot`.
1. How would you feel if this toolkit were used to create a embarassing and convincing deepfake videos of you and/or your family members (perhaps your parents)?
2. Why do you think you have to enable people to fake videos very easily?
And yeah please don't be discouraged to continue publishing your work and models because some people think it's can be abused. Bad guys always have access to the tech they want anyway.
It would be a nice gesture to say "I am as much at risk of the consequences of my actions as the people I have chosen to put at risk"
Are you not worried about the conflict of interest inherent in providing offensive and defensive tools simultaneously?
If someone charged money for minesweeping and simultaneously gave out mines I think that would be a fairly clear problem. I think it's a good metaphor because it captures both the conflict of interest and large potential for collateral damage.
Exactly this. At some point it's not feasible to calculate the impact of a technology's trickle down. Unless the impact is large enough.
I'm not surprised people have such a viscerally moral reaction to deepfakes.
> There are lots of tools that have been built that have arguably done more harm that no one talks about, like port scanning tools
I don't think this is a fair argument against current tools.
If we had some way of accurately measuring the secondary effects of port-scanners, should we start caring? Should we retroactively remove any tools that we later regret? (I don't have good answers to these either)
I suspect this also applies to many criticisms of crypto and NFTs as bad for the environment. The impact is simply easy to calculate...not outsized to other industries. We certainly don't evaluate, say, the video game industry based on its environmental impact in the same way.
Banks face and handle fraud on a much higher level, and have a much better training in identificacion.
My point is that a bad actor would have to show up in person and lie to a federal entity. It's not that they won't get away with it- it's that they cannot hide on the other side of the globe.
http://kaldir.vc.in.tum.de/faceforensics_benchmark/
https://www.kaggle.com/competitions/deepfake-detection-chall...
https://ai.facebook.com/blog/deepfake-detection-challenge-re...
edit*: looks like yes forensics top submission 25 Nov, 2021
Real-time, controllable deepfakes ready for virtual camera injection. Created for performing penetration testing against e.g. identity verification and video conferencing systems, for the use by security analysts, Red Team members, and biometrics researchers.
Reminds one of the vulnerable world hypothesis. It's only a matter of time before a technology comes along that is both destructive and so simple any fool can use it.
There are two very rough "proofs of a kind" I use to explain this to students, both well known in military and counter-terrorism analysis.
One is called Blotto analysis. As the number of fronts grows large (>12 is an important point) the game favours the attackers. Defenders have to always win on all fronts, but terrorists only have to win once on any of n fronts. Another scenario comes from finite board games like Go. Defenders lose when their back is against a wall. Because the space of practical human security solutions is smaller than the inventiveness of attackers, eventually the attackers win. Biometrics is the poster child for this kind of folly. There are only two edge cases. One is to create proven trustless systems. Such systems cannot involve humans in any way shape or form, and therefore fail by definition as human security solutions. The other is to eliminate all attackers.
This does indeed create a fountain of fascinating ethical conundrums with regard to security research and the possibility for strategic limitation of knowledge hazards.
It depends on how you design systems and processes.
Let Mn denote the nth defense mechanism of a system. We can build architectures in which the probability of breaches is calculated as:
(M0 is pwned) AND (M1 pwned) AND (M2 is pwned) AND (M3 is pwned).....
You can see the total probability can become small quickly, as it's computed as the product of all probabilites. The collorary is: it's possible to design secure systems from insecure building blocks (as long as those are reasonably uncorrelated!)
Insecure systems are generally designed the way you suggest in your post: if a single defense mechanism is breached, the whole system will be breached. So you basically are as secure as your weakest link.
For example: what about using this in an interview for a remote job (possibly intercontinental)? It could tip the balance in some situations (due to biases). For example, beating ageism by projecting a 25 years old version of ourselves. Or removing a tatoo or birth defect (and possibly other changes too, you can imagine).
https://findbiometrics.com/nine-of-top-10-liveness-detection...
Other virtual cameras would work too, by the way. OBS is the README as it's the easiest to set up.