Hacker News Reader

The general public submitting CSAM directly would indeed be highly unlikely, but the scenario we need to consider involves those in positions of authority who can manipulate systems behind the scenes.

Imagine that an unflattering or satirical image of Viktor Orban is circulating in France, and let's say it becomes viral, inciting discussions that the Hungarian government finds detrimental to its international image. The authorities might want to suppress this image, not just within Hungary but also in the whole of European Union.

The Hungarian government - who presumably has access to the EU CSAM database (or can coerce those who do), might attempt to add a fingerprint of a manipulated CSAM image that collides with the fingerprint of the satirical image. The principle here is not for public individuals to submit CSAM directly but for government actors to manipulate systems clandestinely.

Won't that easily be found out when the hash matches the image of Viktor Orban rather than an image of CSAM. I'm not sure the legal system is as stupid as you think it is. Then Hungary would just have their hashes reviewed.

Sure a conspiracy of all the relevant authorities across the whole EU would work, ... but that seems a stretch to enable what, political elites to _cooperatively_ censor images the public hold. There are easier ways, surely.

The point is a real CSAM image can be manipulated such that when hashed it matches the hash of the Orban image. So your phone calls the police saying you have CSAM when you get the Orban image. Your life is ruined and you're financially impacted trying to defend yourself. Even if you're cleared of charges you're fucked. This now has a chilling effect on sharing of the Orban image.

Orban's people won't face any pushback because they submitted an actual CSAM image. It's on you to prove somehow they intentionally poisoned the CSAM image such that its hash matches the Orban image.

I get people's opinions of law enforcement are low, but do you really think that no one would question why a huge number of people have a single CSAM image on their device especially when the hash for that image was just added to the database? Do you think that no one would question that maybe there is something wrong with that hash? Do you think that no one would look at the flagged image on any of those devices?

I understand people's concerns with this tech, but this seems like a rather silly hypothetical to me.

It's really about preventing images from circulating.

Yes, the database maintainers would notice, but it could take them a while to get around to it. And they're not going to be eager to remove a collision, because that would effectively "legalize" the child porn member of the image pair. There are lots of images that could be useful to suppress temporarily.

And if you actually succeed in suppressing the false-positive image, it may not get passed around very fast, and it won't get vastly more hits than real target images, so it will take longer for anybody to notice or care.

There's also a possible end game where the child porn traders start perturbing their images to collide with really common images like flags, corporate logos, iconic movie stills, and whatever else. So now you either have to ban the US flag, or let this or that actual child porn image go.

I don't actually think that the false hits would ruin very many lives in most places. But it's worth noticing that the original article was talking about authoritarian regimes repurposing the system without the consent of the database maintainers. In the Orbán example, it's possible that the system might flag you for child porn, but you might actually get arrested for sedition. And that continues to happen to people until the database maintainers pull the hash.

>Yes, the database maintainers would notice, but it could take them a while to get around to it. And they're not going to be eager to remove a collision, because that would effectively "legalize" the child porn member of the image pair. There are lots of images that could be useful to suppress temporarily.

They may not know it immediately, but the actual CSAM image wouldn't actually be shared in any real numbers which removes much of the concern with "legalizing" the image. You can argue this makes this system ineffectual, but that also means this isn't repeatable since weakening the impact of this system would quickly result in Hungary losing the power to add hashes to the database.

>And if you actually succeed in suppressing the false-positive image, it may not get passed around very fast, and it won't get vastly more hits than real target images, so it will take longer for anybody to notice or care.

Who is sharing the real target image? Is Hungary now an active creator and distributor of actual CSAM in addition to manipulating the database?

>There's also a possible end game where the child porn traders start perturbing their images to collide with really common images like flags, corporate logos, iconic movie stills, and whatever else. So now you either have to ban the US flag, or let this or that actual child porn image go.

Once again, this behavior would get Hungary booted pretty quickly. Also it is important to remember these are hashes. Not all images of the US flag would trigger the system only that specific image that has a hash collision.

>I don't actually think that the false hits would ruin very many lives in most places. But it's worth noticing that the original article was talking about authoritarian regimes repurposing the system without the consent of the database maintainers. In the Orbán example, it's possible that the system might flag you for child porn, but you might actually get arrested for sedition. And that continues to happen to people until the database maintainers pull the hash.

But this isn't a closed system within that authoritarian regime. It leaves bread crumbs of this behavior out for everyone to see. If Hungary is going to arrest people on made up charges, they can do that anyway.

I just think this in a complicated and ineffectual bullet that can only be fired once because it has a fingerprint of the person who fired it. That adds up to make this type of abuse less of a worry.

>> There's also a possible end game where the child porn traders [...]

> Once again, this behavior would get Hungary booted pretty quickly.

I'm sorry; I should have made myself clearer. In that paragraph, I've taken an aside and moved from Hungary (or any other government) as the adversary to child-porn-sharers-in-general as the adversary, and also changed the adversary goal.

No matter what you do, you can't "boot" the child porn sharers, because they're the ones who actually define what images you're legitimately trying to block.

> Also it is important to remember these are hashes. Not all images of the US flag would trigger the system only that specific image that has a hash collision.

They're approximate perceptual hashes, designed to come up with close values on close pictures. The US flag has an officially defined appearance. You'll get the same hash for any two close-cropped, straight-on images of the flag, the kind you might embed in your Web site. They'll be at least as close as two reprocessed versions of the same child porn image.

I was oversimplifying, though. You're not going to be able to tweak just any child porn image to make it hash like just any flag, unless you're willing to distort it into unrecognizability. And flags and logos might be bad candidates in general, because they're going to give DCT output that's wildly different than what you'll get from most photos. But if you had a relatively large library of child porn and a relatively large library of heavily-used effectively unbannable images of whatever kind, you should be able to find a lot of the child porn images that you can tweak to hash like one or another of the heavily used ones.

... and if you're generating fake child porn from scratch using ML, you can probably hack your ML model to bake the hash of one or another unbannable image into everything it creates. You could probably make those matches pretty damned close.

So, once they got the whole thing down, they should be able to force the system to greatly tighten its match thresholds and/or deal with a really high false positive rate. In fact, they could probably make things bad enough that they could end up with a pretty large collection of child porn that the operators would be forced to completely exclude from detection.

Now back on the original authoritarian threat model:

> I just think this in a complicated and ineffectual bullet that can only be fired once because it has a fingerprint of the person who fired it.

For the "authoritarian" purpose, you may be right... although I wouldn't be surprised if it stayed under the radar for longer than you think. If the image you want to suppress only circulates among your own people, and if you're the authority who receives and verifies the reports on your own people, then all you have to do is to keep the overall volume down enough that it doesn't make anybody suspicious enough to demand that you show them the reports you're getting.

If you're a relatively small country, the number of people who share some local meme you care about may be quite a bit smaller than the number of people who share some new real child porn image.

>The Hungarian government - who presumably has access to the EU CSAM database (or can coerce those who do), might attempt to add a fingerprint of a manipulated CSAM image that collides with the fingerprint of the satirical image.

Then what? What does that achieve? There would be a huge spike in images identified as CSAM which would obviously throw up red flags. It seems like this would mostly just be headache for the law enforcement across the EU. It isn't like France is going to arrest a huge number of people without any investigation or thought of how this one CSAM image spread so far so quickly. And if we are talking about the result in Hungary, that government doesn't need this tool to abuse its power. Why go through all that effort? They could just do the equivalent of rubber-hose cryptanalysis.

One idea would be for the government of Hungary to create a list of its citizens that share this inciting, dangerous or whatever you you wanna call it material. If they keep on finding the same persons distributing multiple times, they may pay them a visit, get them fired from their government job, block their bank accounts, put them on the no-fly list or whatever.

And that’s just one idea, probably there’s others.

But why do this flagging out in the open where other member countries will see this obvious pattern of behavior and abuse of this system? Once again, yes this tool can be abused, but there are much simpler and harder to detect approaches that a corrupt or authoritarian government could implement. That is where the rubber-hose analogy comes into play.

I guess it comes down to this black box list of hashes that no one would want to audit for pretty obvious reasons. The authority is only as good as the person paid to automate the hashing process feels that day.

It’s not inconceivable that someone at the authority could put in an image known to be stored by someone else into the hash list just to cause an extremely scary and confusing month for that person.

To be fair, that’s also the same problem of say, someone at your ISP dislikes you. There’s always a chance your supplier has a “bad apple”. Maybe it’s not an issue but I get the idea.

> I guess it comes down to this black box list of hashes that no one would want to audit for pretty obvious reasons. The authority is only as good as the person paid to automate the hashing process feels that day.

Isn't that exactly how it is run today, however? There are actual people at the National Center for Missing & Exploited Children who maintain the database and do in fact get incidentally exposed to the imagery. It's the one organization in the entire nation allowed to legally possess the actual images.

Another version of the attack is to manipulate an innocent image in a way that's flagged by detection systems, and then spread that image somehow. If any of the responses are automatic, then it's a lot of false positives in the system, a lot of contact from police, disabling of the host, things like this.

There's also not really any need to trick the system or manipulate other files to generate collisions. Spreading one of the original law-breaking files to unwilling targets would work as a trigger just the same.

But the recipient would know they had gotten CSAM and wouldn't pass it along.

> You then submit the manipulated version of the found or generated image to the authority.

If this is easy (for some definition of easy), is it not easy to then alter the CSAM images every they trade it to another pervert, such that hashes never match? How much image alteration would that take?, would it be human perceptible? If perceptible, would it be ignorable?

I think the exploit here suggests that the whole system may end up obsolete sooner rather than later.

Abusers who have actual CSAM could intentionally publish the fingerprints to sabotage the scanning scheme. If the illegal fingerprints become known, it will be possible to generate false positives and overwhelm verification/enforcement with bogus matches.

Damn, a bit far fetched but theoretically possible. Jikes.

It's not an "any of us problem" for a bad system to be overwhelmed. Let it be overwhelmed.

My understanding was that you generate a csam with a colliding fingerprint and put it online and trick somebody with an iPhone or whatever into clicking on it. Now they're on a watchlist and will get harassed by the police.

So you have to plant the manipulated csam somewhere that it'll be found by law enforcement and added to the database and just hope you did a good enough job to not be tracked?

Look if I want to host some white noise I can't be held responsible for what happens if people xor it together with some other file hosted elsewhere.

Erm, of course you can, you are then just distributing means for acquiring CSAM, or taking part in conspiracy to distribute it.

Under this logic any image at all on the entire internet is now "conspiracy to distribute" CSAM, since you can make an image diff from anything

>just hope you did a good enough job to not be tracked

Right. Your target uses e.g. an iPhone. You use Kali on clean hardware.

Would this org refuse to accept CSAM from an anonymous source just because they can’t point to its origin?

I think they'd send some police to find out

>Like if an ordinary citizen showed up with csam the excuse "oh I found this JPEG in a trunk in my late grandpa's attic" isn't gonna fly

Just post it to 4chan, it’s actively monitored by intelligence services

Also by unintelligent services

This post shows you either don't know what CSAM is or what an intelligence service is.