Hi there, one of the maintainers here. Thanks for sharing & AMA!
Yet this question seems unfairly loaded. At the least - premature.
Though we can easily imagine it, afaik there is no evidence of any actual abuse to date.
I've often wondered whether the net effect of the whole security research community is a net positive or a net negative and I honestly do not know the answer. So yes, it is a loaded question. But asking yourself if what you can do is actually the right thing to do is always good, especially if you - as these authors imply - know up front that there is a large chance of abuse.
I just hope the person offering AMA will not be daunted by a good but blunt question from a voice that carries weight around here.
Hence my question. Because I do feel that weight and it has caused me to carefully consider the abuse potential of the stuff that I've released since then and I've only released those things that I feel have none that I can (easily) discern.
Which is one of the reasons I'm so completely against software patents and a large number of patents in general. Quite a few of them are simply things that the time is right for.
Here is one of my idea dump lists, you can check for yourself which ones are not yet done (which is probably a really small number by now) and which ones have turned out to be homeruns (and in some cases billion dollar+ companies).
https://jacquesmattheij.com/my-list-of-ideas-for-when-you-ar...
One that wasn't on there eventually led to https://pianojacq.com/, which I'm happy to report to date has not led to any kind of abuse. And no, it did not put any piano teachers out of business either.
Software engineers and other technology creators don't take a "Do No Harm" oath like doctors. Many of them have never even taken a single Ethics In Technology course at university (it was an optional class when I was in undergrad decades ago). And, even in the alternate universe where ethics was baked into engineering training, all it takes is a single rogue willing to ignore them, and now the world has to deal with it.
In the end, I decided that one oppressed person using it to improve their situation is morally "worth" many criminals using it. It's kind of like "better ten guilty men go free than put one innocent man behind bars".
Because the one innocent is usually a defender, whereas the guilty men are usually the attackers and to give the attackers an effective advantage in an arms race is a risky thing with unknown and potentially devastating outcomes.
In my own life (the live video example listed elsewhere) it would have meant that we probably would have had everything that we have today anyway, only maybe a little bit (not even that much, I'm aware of one other individual who was working on a similar concept who contacted me after our release) later.
Video conferencing, live streams in the browser without plug ins and so on all would have happened, for sure. But at least the massive mountain of abuse cases would not rest partially on my shoulders. And because I've been confronted with the direct evidence of the results of my creation for me that link is easy to make. But if you work on secure communications software you are probably not aware of the consequences.
Please read this https://www.biometricupdate.com/202205/sensity-alleges-biome...
We have warned many vendors about the vulnerability of their commercial biometrics software. The threat is currently downplayed by the whole industry. We hope this release to be a wake up call and that our team will be joined by other experts in raising the alarm.
Deepfakes are already used for spoofing KYC around the world. This is already happening, and not by using `dot`.
I distinctly remember a very similar discussion around firesheep 12 years ago: https://www.computerworld.com/article/2469667/firesheep-fire...
https://news.ycombinator.com/item?id=1828955
> There are probably going to be a lot of people negatively affected by this for quite some time to come. One thing to point out is that there are grades of things. There is "public", and then there is "top hit on Google". Similarly, there is "insecure" and then there is "simple doubleclick tool to facilitate identity theft".
> How many millions of dollars and man hours is it going to take to lock down every access point? How many new servers are going to be needed now that https is used for everything and requests can't be cached?
I'm not OP but I would answer like this: abuses of this technology are inevitable and can only be mitigated by counter-software (which leads to an arms race).
The release of this source code could kickstart the development of deepfake detection software.
Or maybe in general people need to put less weight on video evidence.
We have warned many vendors about the vulnerability of their commercial biometrics software. The threat is currently downplayed by the whole industry. We hope this release to be a wake up call and that our team will be joined by other experts in raising the alarm.
Deepfakes are already used for spoofing KYC around the world. This is already happening, and not by using `dot`.
And yeah please don't be discouraged to continue publishing your work and models because some people think it's can be abused. Bad guys always have access to the tech they want anyway.
1. How would you feel if this toolkit were used to create a embarassing and convincing deepfake videos of you and/or your family members (perhaps your parents)?
2. Why do you think you have to enable people to fake videos very easily?
It would be a nice gesture to say "I am as much at risk of the consequences of my actions as the people I have chosen to put at risk"
Are you not worried about the conflict of interest inherent in providing offensive and defensive tools simultaneously?
If someone charged money for minesweeping and simultaneously gave out mines I think that would be a fairly clear problem. I think it's a good metaphor because it captures both the conflict of interest and large potential for collateral damage.