Hacker News Reader

Comment by mistercow | Hacker News Reader

Aside from the fact that this was obviously never viable and the entire problem is clearly unsolvable if you sit down and really probe it for fifteen minutes, what I find most frustrating about this is that the false promise of preserving photos as reliable evidence is actively harmful.

You will not build a perfect system, or even something near perfect. The best you're going to do is make it so that it's hard to casually present AI photos as real, leaving only the cases where it really matters. In the "best" case, you've just made the public more trusting of photos in general, so that when there's actual money or power on the line that makes jumping through the hoops to fake authenticity worth it, the public is more susceptible.

The best outcome at this point is for everyone to get on the same page that photos have roughly the same probative value now as drawings. Poorly thought out snake oil efforts to prove authenticity are only going to delay that.

View on HN

In my opinion, the benefits for end users are rather minimal since I doubt an average person would ever be checking C2PA provenance data, but there is a commercial incentive for Google and others to promote C2PA, since it makes preparing training material for Machine Learning significantly easier, and perhaps as a smaller benefit justifies hardware attestation that locks users down into proprietary OSes.

> it makes preparing training material for Machine Learning significantly easier

How?

A huge problem with ML training is the ‘ouroboros issue’ - training ML with input from other MLs breaks things in very deep (but difficult to stop/detect when it’s happening) ways due to the way the internal math works (model collapse).

Right now, the Internet training set is becoming more and more contaminated with better and better generative AI images and video.

It makes the models more screwed up, and makes it very difficult for humans to figure out what is original and not too.

If there was some signal that could at least make it easier to identify ‘original’/real images…

"Model collapse" is often overstated, as this paper demonstrates: https://arxiv.org/pdf/2404.01413

The original model collapse paper assumes you train networks on 100% synthetic data produced by the previous generation. But if you maintain some portion of real data then the problem is mitigated.

Not when you also including poisoning attacks.

I remember the original paper showing issues with even a couple percent of certain kinds of synthetic data too, not 100%.

What do you mean by poisoning attacks - stuff like Nightshade or Glaze? I was under the impression that those have largely failed to achieve their goals.

Those aren't poisoners. The are attempted protectors.

I think the term "AI incest" would sum this up :)

OK, but any reliance on a fakable signal such as this seems guarateed to invite poisoning.

People got on fine until 200 years ago with the only means of rendering a picture being to draw it, as in Hogarth's calumnous image of the in fact really quite civilised Gin Lane (https://www.theguardian.com/artanddesign/picture/2012/sep/12...)

People didn't have the ability to distribute forged images to millions to form a mob in seconds though.

I don’t think there is a technical solution to this problem but I think there is a legal one.

Make it a legal requirement to mark AI generated photos and enforce penalties for posting unmarked AI generations. Social media should also mark the country of origin for each post, with the knowledge that posts from your own country are covered by these laws.

> Enforce penalties

Really, everyone should know by now it is not as simple as that. We already have lots of laws that are not enforced (fully or at all) because there is a cost to enforcing the law - time and money. Some random person uploads an AI image to Facebook and you really expect the police to expend a few thousand $ prosecuting them for what? To Server as a warning?

Some people will be untraceable. Some will claim innocence or mitigation or some disability that excuses them. Even those that might be prosecuted will be fined and won't pay which costs even more to follow up.

We should know that the legal system does not create an obedient society.

That hasn't done anything to stop scamming, so why would it apply to AI? People located outside of areas with these laws won't have to follow them, and this reduces building a immune system to such actions, making people more likely to fall for it when done by those not bound by the laws.

In a real like political misinformation, this will have the effect of making people trust non-watermarked images more, which will then be used by foreign actors to pass off propaganda as legitimate.

Also, if you don't hold people responsible for spreading an image they know is fake, bad actors can take advantage of this even within the US (they purposefully spread a image they have reason to think is fake but lacking a watermark), but holding people responsible for a strict liability crime for spreading AI without knowing it is AI seems an even worse route.

I'm not sure a law even makes the issue better in a 'don't let perfect be the enemy of good' sort of way.

I think public key cryptography offers a technical solution that is nearly as ideal here as for its existing uses for securing communication between physically remote actors -- please see my comment here for details: https://news.ycombinator.com/item?id=49444227

I say "nearly", because as soon as you need to keep a private key secure from someone with direct physical access to the device, you're entering dangerous territory. TTBOMK there's no way to make a "perfect black box", so it becomes an arms race between defensive "obfuscation" and tamper detection mechanisms in the one hand and stealth scanning techniques on the other. But this is already the case for TPMs -- that is, the situation is no worse than for an already widely accepted technology.

The EU did this, Claude and Microsoft complied and HN was practically rioting about it literally yesterday: https://news.ycombinator.com/item?id=49421158

No, this would be literally you (the user) checking "this is AI generated" before you post something publicly. Platforms can auto mark it if they detect a watermark like SynthID but they can also leave the choice up to you.

The point is making it easier to go after bot accounts that spam generated videos to influence politics. They need to disclose that its AI and lose their power or lie and get criminally prosecuted.

It's not the same thing as model providers adding a mandatory watermark to everything, and even worse one you can't verify yourself and have to trust that Anthropic is telling everyone the truth. People should have the option to use undetectable AI tools in private, even if it's illegal to post the outputs on social media.

I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing.

People are concerned that the technology will lend additional credence to the last 0.1%. But anyone who thinks about the technology for 2 minutes will see you can just point the camera at the screen. In cases where it really matters (a court of law, internet arguments between nerds) people will know it's not 100% reliable. Locks can be picked, and signatures can be forged, but that doesn't make them useless.

"C2PA Cameras Do Not Survive Contact With Reality" does not survive contact with reality where very, very few users would even think of rooting their phone so they can create signed fake images.

When I search for "C2PA" on the google play store, there are more AI-watermark-removal apps than there are signing apps. Certain types will jump through ridiculous hoops if they think it will affect their algorithmic reach on social media.

Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.

You're arguing that lots of people can spoof this, the other guy is arguing that nobody will know it can be spoofed so it will do more damage. But these are contradictory -- if fakes become common, then they will also become common knowledge. The impact of any given fake is reduced if there are more of them. The technology doesn't need to provide 100% assurance. If it adds even a little friction to the slop mills then that's increasing the signal to noise ratio.

It's actually worse if it is plausibly trustworthy for "99.9%", since that's enough that naive users will get accustomed to believing the verification badge is authentic.

When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.

Read the rest of my comment please. Is the single motivated malicious user able to do as much damage as all of the blocked attempts put together? Probably not, since if there's really all that much riding on it, people will point out it can be bypassed.

Should we also abolish Pangram, because it's not 100% accurate? Someone might be convinced a text is not AI-generated when it actually is! We should get rid of it rather than fool people into thinking it can be determined accurately. What about antivirus? We should abolish it as well rather than fool people into thinking that their software is ever 100% safe. What about HTTPS? We shouldn't call it "secure" shell because the computer you're connecting to could be compromised! I could go on and on and on.

The median instance of AI image generation isn't evidence in a court case. It's cyberbullying, or deepfakes, or fake news. It's called "slop" because there's a lot of it being churned out at low effort.

> Is the single motivated malicious user able to do as much damage as all of the blocked attempts put together?

Yes, absolutely. Probably moreso. The whole point of these proposals is to try to solve for the "motivated malicious user" who is engaging in actual high-stakes fraud. There is no point in applying techniques that suppress inconsequential pranks while making serious crimes easier to get away with.

This really seems like a rehash of the perennial DRM argument: DRM restrictions provably do not reduce large-scale motivated copyright infringement, they just annoy legitimate paying users. This is the same class of solution, in that it is effective only where the stakes are low and the impact is minimal.

Why is this being framed as 2 types of users, lovable pranksters and fraudsters? There's a whole spectrum between these 2.

Also I'd like to know if a "joke" is likely fake.

I don't suspect there is a uniform spectrum between those two. I think this is something that's going to be clinal, which we see in a lot of other comparable social contexts. The number of people actually willing to cross a moral threshold into outright crime is relatively small, but those are precisely the people who cause the most damage when they get away with their behavior.

Bur I don't even really think that's really relevant anyway, because whatever the density of "malicious" motivations is, the point here is that the fact that it only is an effort/motivation threshold that allows this technique to "block" malicious uses, and the motivation to overcome that threshold correlates directly with the stakes involved in the malicious use.

In other words, the more malicious the abuse is, the less effective this solution will be: the boundary of its usefulness will be wherever the line between pranksters and actual criminals happens to lie.

Your idea of a criminal seems to be hypercompetent and think of everything. These do exist, but most criminals are not very smart. Smart, dedicated, technical people can typically make more money legally.

Your argument applies to any imperfect security technology -- aka practically all of them.

> Your idea of a criminal seems to be hypercompetent and think of everything.

No, my idea of a criminal is someone who is motivated to commit crime, and I feel that we've already established in this thread that the approaches we're discussing are motivation gates far more than competence gates.

> Smart, dedicated, technical people can typically make more money legally.

Then who's been running all the botnets, writing cryptolocker malware, and running phishing scams for the past couple of decades?

We've always had script kiddies, and now we have people using AI itself to do malicious things. Technical skill has never been an obstacle for sufficiently motivated scammers.

> Your argument applies to any imperfect security technology -- aka practically all of them.

Ultimately, everything has weaknesses, and with enough effort, most measures can be circumvented. But how much effort is enough varies wildly between solutions.

There's a huge gulf between a "no trespassing" sign, on the one hand, and a concrete wall topped with barbed wire, on the other. The "no trespassing" sign only keeps out people willing to obey it; the concrete wall keeps out anyone who isn't willing and able to accept the time, effort, and risk necessary to climb over it or knock it down.

And the point is that using digital signatures to distinguish AI-generated media from hand-made media is much closer to the "no trespassing" side of things than it is to the wall. Maybe it's analogous to a gate with a latch you can open from the other side if you reach over in just the right spot.

Your no trespassing and concrete wall analogy again indicates the black and white thinking here.

I'm afraid it doesn't. To the contrary, treating effective security in terms of how it influences the attacker's cost-benefit tradeoffs, and evaluating proposed measures on whether the effort threshold they create is enough, is quite literally the opposite of black-and-white thinking.

And in this case we can clearly see that a solution that (a) does not substantially increase their costs -- and in fact, as I've pointed out above, only really filters by motivation, not by time, money or effort, and (b) doesn't target their potential benefits at all, is one that isn't likely to be effective.

You're conflating the effectiveness of the mechanism with its systemic impact.

* Pangram: Yes we should really be discouraging people from putting trust in tools like this because they can't be made totally reliable.

* Antivirus: We should be building application environments with robust security models so that malicious software has a limited blast radius (like we do on mobile, like the Linux ecosystem is trying to do with Flatpak, etc).

* HTTPS: HTTPS is a strict upgrade from HTTP so we should be using it everywhere possible. The UI symbols to indicate to users the security expectations they're getting are good practice.

* ssh: This is just an inappropriate comparison.

The HTTPS comparison would make more sense if actually 0.1% of the time when their browser said they were using HTTPS it was just lying.

Pangram: I'm not arguing against encouraging skepticism; I'm arguing that the technology is not useless. It's good for people to know the limitations, but it's still evidence.

Antivirus: "Actually, we should build this hypothetical better thing" is a cop-out.

HTTPS: C2PA is a strict upgrade from unsigned photographs, so it should be used wherever possible.

SSH: Totally appropriate, the entire point of the discussion is whether it's permissible to the user that they might be more secure.

You're missing all of the points that there could be by focussing on random people.

While it is always an individual tragedy when people treat each other badly (e.g. through deepfakes and all), the real threat does not exist on that level.

This is about misinformation and disinformation, so we're talking state actors. And with that, the 99.9% hypothesis does not hold true.

It's funny how people always say something is "a tragedy at the individual level" when they mean "it's not my problem." It's even crazier to dismiss the value of a security feature, just because it might make people feel more secure. That's true of every security feature! Very little of the technology that the web is built on is proof against state actors.

I like being contrarian as much as the next guy, but "Actually, having security is worse for security" is taking it a little too far.

I am repeating myself, but this is about systems, and not about people.

It is however in the interest of the people to keep the systems running in an untainted way.

As said, on the individual level it's a tragedy, but one that can be absorbed somewhat. Democracy itself failing otoh is kinda hard to absorb.

C2PA is not "having security". It is "having an illusion of security for compliance and CYA reasons, that can be fairly trivially exploited by nation state actors". Banality of evil. Again.

___

Actually, come to think of it, "security" is the wrong term there. Signatures don't secure anything. They attest.

Those are different things. Argh and I ran with your term aah

This is pascal's mugging. Democracy itself might fail! You're just inflating the stakes of your hypothetical bad outcome until it can overwhelm any positive upside.

Not to mention, democracy is under just as much or more threat from "banal" fake news created by citizens. People gonna people. They don't need the DPRK lying to them to fool themselves.

Yeah I think any additional security is good. At worst this could help in a lot of court cases. Someone presents photo evidence - it could be manipulated - it could be not. This happens already. Then someone produces an original higher quality version (like a raw photo - which I take even on my phone at all times now) and experts can verify that as the original.

And I agree on state actors. If a major one is invested in something like this they might have well compromised the signing project itself, the verification process, or even the court system or media. That seems like a rare and extremely high bar to guard against.

Exactly. It's just more information. It doesn't have to be 100% accurate in every case, to be useful.

> I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing.

Exactly: most people don't bother editing visible watermarks out of AI-generated media, because they have little or no incentive to bother doing so. This will "defeat" the 99.9% of users who are not actually trying to do anything malicious, but will be a minor annoyance to the 0.1% of users who are actively engaging in fraud, fabrication of evidence, etc.

The upshot is that not only us this not useful for its intended purpose, it will lure people into a false sense of security by creating expectations that AI-generated media will always be easily identifiable as such, and reduce the level of scrutiny that gets applied to the stuff that actually is malicious.

What percentage of users are malicious, do you think? To me the issue is precisely what the product is trying to solve: propaganda using faked footage passed off as real, which generally has no real difficulty with resources available to boost their message. Giving that any kind of stamp of authenticity is a bad idea, IMO, and the fact that it'll work on 99% of the cases that don't matter makes it even worse.

Almost everyone seems to be overindexing on this second-order effect. But second-order effects don't typically negate or surpass the original effect, because they depend on the original effect.

> I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing.

Everyone realizing that photos don't prove anything would defeat 100% of malicious users. I don't understand what people incorrectly trusting photos is supposed to achieve at this point, in your view.

Good luck with that. People are basically going to believe photos if they're aligned with what they already believe. They're not going to make strong decisions based on the C2PA tag. The idea that people will revise their entire worldview or submit to fraud based on a C2PA tag is typical HN thinking. People don't trust technology that much.

I agree with "I'm fairly certain this will defeat 99.9% of malicious users".

Also, note that C2PA should have something like Level 3 as well: "The image is mathematically proven to have come from the physical camera sensor."

It's somehow difficult to achieve this, but it's possible (although the attacks will always be possible of course).

Difficult and also solves nothing, since you can just point the camera at a screen.

There is active research addressing this problem, for example https://www.usenix.org/system/files/usenixsecurity25-park.pd...

But yeah, difficult too :)

Have you ever tried pointing a camera at a screen?

The screen is doing all kinds of crazy things that are not apparent to your eyes, but that show up clearly on camera.

Some screens are like that, but it's not an inherent property of all visual displays.

For example, you'll see a sort of barber-pole effect when pointing a video camera at a raster-scan digital display whose refresh rate isn't synced to the camera's frame rate. To avoid that, sync them, or maybe use a colour e-ink display.

Alternatively, print a high-resolution version with a decent photograph printer and take a picture of the print with the C2PA camera.

Are we entering a world where if I took a picture with a film camera and scanned it, it would be rejected as not real?

This is ridiculous.

It's not a matter of "rejected as not real", it's "There's no way to know if this is real or not".

When 99% of people are posting pics from their big tech approved smartphones with the "200% real no way to fake this" badge it doesn't matter that absence of the badge technically doesn't mean fake, that's what it will be interpreted as.

The defendant submitted a remarkably high quality video of you saying you generated it with Nano Banana.

And their video was C2PA signed.

You can even do it wholly in an analogue process, do a darkroom print, and it could be seen as fake as well - darkroom techniques are quite flexible as well. And a negative itself? That could be a copy of a prepared printed slide.

Worse than not working, this will likely be used as yet another excuse to attack computing freedom.

> and the entire problem is clearly unsolvable

The problem is not identifying AI generated media

The problem is identifying real media. That it can actually do.

What makes you so certain that this valuable research showing weaknesses in today's systems will render the C2PA concept useless forever?

Yes, software LPEs are a risk -- as they are in every nontrivial computer system. New ones will appear, and old ones will be closed in time, as TFA acknowledges.

Re hardware attacks: The (neat!) glitch injection attack the author describes in the linked "lighter" page only raises the implementation cost of doing image certification properly. For example, if the camera module presented only an interface that dumped raw RGB or JPEG-encoded data plus a digital signature that used a private key known only to the manufacturer, then all that would be required to verify a "downstream" image would be to keep a copy of those original bytes inside the final (potentially cropped, filtered, AI-ed, etc.) image, in the worst case roughly doubling its size on disk (though certainly more efficient schemes could be designed). Any interested third party could then compare the original and final images by eye and decide for themselves whether or not the subsequent processing materially changed the image's "meaning".

Finally: Does the existence of lock picks or bolt cutters render padlocks pointless today? Does it corrode society by encouraging people to mistakenly believe that anything they put behind a $5 padlock will be safe forever? No, and no.

> Does the existence of lock picks or bolt cutters render padlocks pointless today

A padlock shouldn't be the only component of the bigger physical security picture.

A random person carrying/using bolt cutters or trying to pick a lock looks suspicious, and should be noticed by on-site staff or whoever is monitoring the security cameras.

If the padlock is decent, there will also be an inherent delay involved in bypassing it using those tools, which should increase the likelihood of the person being noticed.

If the padlock is used in an unattended/unmonitored location (i.e. a remote vacation house with no neighbours and no security cameras), then the padlock is probably only good for keeping honest people honest.

A lot of physical security => information security analogies are misleading for the same reason. In information security, it's very possible for an adversary to have effectively unlimited time to perform their attack (or to develop the means to perform the attack quickly).

Imagine a scenario where any determined person could e.g. spend a month in their home workshop and develop a pair of gloves containing transducers that would vibrate any padlock open in seconds. They could mimic the action of using the key or entering the combination to avoid looking suspicious. Also, the gloves have a button that instantly creates another pair of the same gloves at no cost. How much value would a padlock have in that scenario? That's the kind of asymmetric playing field one has to consider in information security contexts.

The comparison to padlocks and bolt cutters is not relevant, they are quite different shapes of problems. A certificate that is weak enough that it should not be trusted in any case where it matters is actively harmful, not just mildly less helpful.

(This is true in general: if you add a trust signal that is mainly just a bit of effort to broadcast, you'll find that scammers and fraudsters will show that signal much more reliably than honest actors. For example, every email spammer has DMARC and DKIM set up absolutely perfectly)

I guess you're making the argument that, while padlocks sometimes protect things of low value, thus justifying the use of cheap, easily broken locks in those cases, the times where authenticity of an image is important are nearly always cases of high importance, meaning that there's no case where an assertion that's actually fairly cheap to forge (assuming a similar "lighter" glitch attack works on phones, less than $1000 for a modern smartphone that may get bricked, a soldering iron and an afternoon's work) makes sense? Perhaps so. It's certainly easy to think of images or video clips of very high importance, and outside of kids uploading AI clips to Reddit and pretending they're real, I can't think of any similarly low-stakes cases for image authentication.

What do you think of my digital signature idea?

This is yet another of these absolutely caustic takes that come with a veneer of intellectualism, but actually just ignorantly deconstruct reality.

Each of them weaponizing the rules of the platform that (for sensible reasons) demand you engage with the strongest interpretation of the message/argument you see.

The asymmetry of effort there is unsustainable, and that's exactly the point.

No idea how that could be solved. Maybe a meta comment like this one helps.

I mean if you think about it, it shouldn't be possible for some anon account to drop this and sound like it's a worthy contribution to a debate against some real person with a real name, a track record and multiple thousand dollars of bricked hardware leading up to that assessment. (Nor would it make sense for a non-anon but equally empty account)

It makes no sense, and the guarantees regarding protection of speech and all do not apply to these topics, because it's not an opinion that would get your real name in jail.

What can we do about these social exploits. Someone tell me please. It's driving me up the walls

That's a very long-ass way of saying "I disagree with this comment, I can't be bothered explaining why (too much work), so instead I think this user should be banned". I extracted nothing useful from your comment, which bears no relation to its parents that I can discern, am irritated to have spent time reading it and composing this reply, and also find your attitude pretty rich when you're also an anon account, and with less karma too. If you want to "call someone out" you've gotta be on damn solid ground and bring receipts. As it is you've just decreased the signal/noise ratio of the thread and contributed to a toxic atmosphere. I would just flag and move on but in this case I think feedback is important, and unlike you I can be bothered to explain what it is that's got me riled up.

This is what I'd call the "enabler" archetype that gets unhappy about friction (rightfully!) but takes some.. uh.. mental shortcuts when it comes to tracing back the source of it.

I'm sorry man. The problem is that this exact described dynamic (see also the other reply by me which you seem to have conveniently missed), is that it affects _all_ internet spaces and makes them utterly miserable.

I agree though that this meta stuff doesn't exactly improve SNR, and I am open for better solutions. But those need to actually solve the problem, instead of just silencing the immune response.

___

Interesting side-thought

> with less karma too

I think this might point at (one of the) root cause(s) of the dysfunction I'm pointing at here.

Internet points ceased to be a metric documenting value quite a while ago. Something something Goodhart

Gonna make this real short and simple: it's totally unclear what you're on about. You've described nothing, merely complained, infuriatingly, in the vaguest possible terms.

Don't take this as dismissive or a dunk or anything like that, but maybe an LLM can help with unpacking?

At least I know that they understand my writing and that they can help me understand other writing I don't.

I can assure you that there is a point. Whether it is worth bothering is of course your decision. I'd wager probably not, but I guess that depends. No hard feelings either way.

There is no content in your response. At all. I honestly don't think I've come across a post this devoid of content on this site before.

All I can perceive in it is a generalised hate towards some broader thing that you feel certain I'm consciously aligned with somehow, and that you think is inherently and obviously evil. Because... I don't use my real name on here? (Is your real name "hypfer"?) Because I don't have bricked hardware to back up my opinions?

If you want to convince me or others that something I wrote is wrong, stop hyperventilating and engage with at least one of the specific claims in my post. For your own sake, I'd also suggest editing or outright deleting your original post.

It is correct that I have completely side-stepped your frame and not even attempted to engage with it. This is on purpose, because wanting people to engage with that specifically engineered frame is exactly the bad faith mechanism I've described.

What is interesting is what happens when these tactics are called out in a meta comment like I did, because it's actually a common thing that people start flailing like this once you side-step the script that was supposed to be followed.

Someone acting in good faith would not counter with an attack and a double-down like this, but with intent to resolve the situation (+ probably feeling a bit bad about an exchange having failed and being misunderstood).

I also like the "hyperventilating" claim. Easiest way to get out of a situation is to invalidate the source, and easiest way to do that is to claim emotions.

And the fake concern (for what, even?) of course.

> For your own sake, I'd also suggest editing or outright deleting your original post.

___

Anyway. I think the convincing people is actually working quite well here. Though not in the sense you'd think.

I'm just pointing a spotlight at what I believe is violating the spirit of the rules (while staying within the letter of the rules).

What other people make out of that is their decision to make.

What on earth is happening here? I seriously have no idea what you're even talking about at this point. You seem to be five levels of meta deep and talking in circles about yourself?

Very bizarre.

Reply on news.ycombinator.com