When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.
Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on.
The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.
While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level.
This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.
I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.
It is however in the interest of the people to keep the systems running in an untainted way.
As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb.
C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors". Banality of evil. Again.
___
Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest.
Those are different things. Argh and I ran with your term aah
Not to mention, democracy is under just as much or more threat from "banal" fake news created by citizens. People gonna people. They don't need the DPRK lying to them to fool themselves.
And I agree on state actors. If a major one is invested in something like this they might have well compromised the signing project itself, the verification process, or even the court system or media. That seems like a rare and extremely high bar to guard against.
Yes, absolutely. Probably moreso. The whole point of these proposals is to try to solve for the "motivated malicious user" who is engaging in actual high-stakes fraud. There is no point in applying techniques that suppress inconsequential pranks while making serious crimes easier to get away with.
This really seems like a rehash of the perennial DRM argument: DRM restrictions provably do not reduce large-scale motivated copyright infringement, they just annoy legitimate paying users. This is the same class of solution, in that it is effective only where the stakes are low and the impact is minimal.
Also I'd like to know if a "joke" is likely fake.
Bur I don't even really think that's really relevant anyway, because whatever the density of "malicious" motivations is, the point here is that the fact that it only is an effort/motivation threshold that allows this technique to "block" malicious uses, and the motivation to overcome that threshold correlates directly with the stakes involved in the malicious use.
In other words, the more malicious the abuse is, the less effective this solution will be: the boundary of its usefulness will be wherever the line between pranksters and actual criminals happens to lie.
Your argument applies to any imperfect security technology -- aka practically all of them.
No, my idea of a criminal is someone who is motivated to commit crime, and I feel that we've already established in this thread that the approaches we're discussing are motivation gates far more than competence gates.
> Smart, dedicated, technical people can typically make more money legally.
Then who's been running all the botnets, writing cryptolocker malware, and running phishing scams for the past couple of decades?
We've always had script kiddies, and now we have people using AI itself to do malicious things. Technical skill has never been an obstacle for sufficiently motivated scammers.
> Your argument applies to any imperfect security technology -- aka practically all of them.
Ultimately, everything has weaknesses, and with enough effort, most measures can be circumvented. But how much effort is enough varies wildly between solutions.
There's a huge gulf between a "no trespassing" sign, on the one hand, and a concrete wall topped with barbed wire, on the other. The "no trespassing" sign only keeps out people willing to obey it; the concrete wall keeps out anyone who isn't willing and able to accept the time, effort, and risk necessary to climb over it or knock it down.
And the point is that using digital signatures to distinguish AI-generated media from hand-made media is much closer to the "no trespassing" side of things than it is to the wall. Maybe it's analogous to a gate with a latch you can open from the other side if you reach over in just the right spot.
And in this case we can clearly see that a solution that (a) does not substantially increase their costs -- and in fact, as I've pointed out above, only really filters by motivation, not by time, money or effort, and (b) doesn't target their potential benefits at all, is one that isn't likely to be effective.
* Pangram: Yes we should really be discouraging people from putting trust in tools like this because they can't be made totally reliable.
* Antivirus: We should be building application environments with robust security models so that malicious software has a limited blast radius (like we do on mobile, like the Linux ecosystem is trying to do with Flatpak, etc).
* HTTPS: HTTPS is a strict upgrade from HTTP so we should be using it everywhere possible. The UI symbols to indicate to users the security expectations they're getting are good practice.
* ssh: This is just an inappropriate comparison.
The HTTPS comparison would make more sense if actually 0.1% of the time when their browser said they were using HTTPS it was just lying.
Antivirus: "Actually, we should build this hypothetical better thing" is a cop-out.
HTTPS: C2PA is a strict upgrade from unsigned photographs, so it should be used wherever possible.
SSH: Totally appropriate, the entire point of the discussion is whether it's permissible to the user that they might be more secure.