Non-Consensual Intimate Image Pilot
newsroom.fb.com
newsroom.fb.com
[A]: human can review the image they already have
[B]: Facebook waits until someone uploads an image that matches and then reviews the image (as normal) but with a marker alerting the problem.
The benefit is that people NOT affected won't have to upload lots of images of themselves to facebook personal for "review" just to be sure.
The problem is that should facebook update it's hashing or find better ways to match images, having the original image would allow them to transition. This point is moot though since facebook claims not to save the image anyway.
Here's what you do: on the client side embed image into a semantic space (using NN or whatever), quantize, _then_ hash the representation. Afterwards you send the hash. If you had to change client side code, you can just ask the user to redo the process.
They can easily scale this by allowing hash uploads, using a trivial client side classifier to guess at whether an image is actually served well by this technology, and confirm positives with CV after a bad actor reposts their media on the site, with human intervention only on the fringes.
I think the point about revealing your fingerprinting system is a good one. But I wonder if the equilibrium point would still be better than what we have now, i.e. unsophisticated posters would get caught, and sophisticated ones might not, vs. what we have now where there is no way to catch any of these posts.
Now that is a job I really would not want to have. Talk about mentally scarring.
I also hypothesise (on the basis of news articles saying how low the reoffending rates are compared to other categories of offender) that the distribution of sadistic narcissism is the same or similar for them as it is everyone else, so when they sit down at the office desk and see a screaming child, most of them will hate themselves if they didn’t already.
I’m not so sure about the second hypothesis though. It’s based on what newspapers say, and I’ve learned to distrust them.
The salient bit:
> A quick note to everybody who says 'calculate the hash locally':
> A) Photo fingerprinting algorithms are usually not included in clients to prevent the development of circumvention techniques.
> B) Humans need to review to prevent adversarial reporting.
Further down the thread, they discuss further that PhotoDNA and similar algorithms are, in particular, not resistant to being tricked. Until they are, it's another layer to prevent adversaries from fully decomposing how they're hashing it, which is a good thing.
It's a trade-off, for sure, but this is a hard problem with no cut and dry answer. Facebook already has the images in question, because you have to be sending them to somebody in order to mark them as NCII. Asking yet another Facebook service to look at them without storing them seems reasonable to me. You may disagree.
Second, security by obscurity is not a good thing. Not running the algorithm locally, not to mention producing the source to their algorithm is not going to help in the long run.
Third, regardless of whether the source of the algorithm is known or not, it would be a simple task to upload a test image of oneself, and then attempt to circumvent the algorithm using dummy accounts.
Fourth, if someone abuses the service by uploading the hashes of a benign images, it can be reported at that time. No need to pre-emptively guard against this.
uhm...this seems like a bit of a flaw in the plan
Humans already need to review the takedown reports themselves to prevent adversarial reporting!
- The user uses a JS solution to hash the images on the client, without the image being uploaded
- She compiles a form with additional information (e.g: capture her account, reasons for uploading, suspect person sharing the picture)
- The picture is saved in the DB as un-verified revenge porn.
- The first time someone uploads a picture that matches the hash, the pic is quarantined and the specially trained individual manually check them
- A scoring system could be used to check the reliability of the submission. If multiple photos marked revenge porn get rejected, the control becomes ex-post. For even more violations, the user get banned from using the tool and should directly contact Facebook. Submitting the same hash that has been rejected, will count as a "red mark"
Now, I understand this system is very complex, what Facebook has done is an MVP and as a product manager, this is what I prefer. But considering the issue (revenge porn, not something I necessarily want to test the impact on retention :-) ). Also, yes, it requires resources, but Facebook has a problem with trust lately, better to do the best...
[edited for formatting]
Alex Stamos (Facebook's CISO) implies this is why they can't do it client-side:
snark-mode off: We use obscurity every day and its a completely valid layer of security.
Not sure I agree with that, most of the time when we do that is because we don't want to spend the time to have better security. And then we get burned. To your example of the 100$ bill: at my parents home with the car parked in the garage? No problem to do that at all. Out on the street in SF? No. I don't trust my glass enough as a security measure. But I don't leave money at all, is not security through obscurity.
But we are going OT. The problem that is raised is that they need necessarily security through obscurity. And we have two problems:
- How really robust are these algorithms? How long before we will see people abusing them?
- Have you thought hard enough about how this system could work? E.g: have a partial hashing made client-side and the final one on the server? Or a situation where the server code is open-sourced without the model to calculate the hash? That would allow for external review without disclosing the hash. Yes, you still need warranties that Facebook is using that code, but you could have a trusted third party certifying the program.
My point is, the person who designed this program didn't really understand the problem. The problem is not revenge porn. The problem is Facebook reputation. And this solution is totally deaf.
Whoever designed this system is probably heavy on security, but low on product.
You have to trust facebook in either case, each time you do it. Either to handle your nude pics properly or to serve you javascript that does what they claim it does, every single time.
On the other hand an open source desktop application only needs to be audited once and then can be validated based on a hash.
In browser crypto is not a solution if you want to minimize the needed trust.
I thought I covered that concern, but I neglected to mention that it should be open source so everyone can audit it.
The cheese is full of holes.
> Once we receive this notification, a specially trained representative from our Community Operations team reviews and hashes the image, which creates a human-unreadable, numerical fingerprint of it.
This clearly implicates that Facebook has, almost without reserve, the ability to read messages from user to user (even though it might limited to messages to oneself), and exposes that ability to its employees.
While I stopped using Facebook a long time ago, Zuckerberg's quote, "they 'trust me'; dumb fucks" seems relevant here. If I were Facebook, I would do this in a different, much more privacy conscious way.
1. Obviously, Facebook is already hashing all images on its platforms and storing the hashes. Given that, let the user, using a frontend only platform, upload an image and generate the hash. Make sure this image is a real photo (not a meme or something heavily photoshopped) in addition to making sure:
1a) The image contains a human[0]. Fairly easily doable for most situations.
1b) If not, let the user know that "we can't automatically detect a human in this photo, do you want to still submit this photo for removal, this could get a strike against your account if (not in some list of reasons for removal)". Maybe the photo has a picture of a credit card or something.
2. Submit the hash to the backend. Given that hash is not used on the platform past a significant amount of views, automatically ban the photo, and allow other users to petition for reinstatement of the photo, knowing that a non-valid petition for reinstatement (the user doesn't have rights to the photo) COULD result in a strike also.
3. Have humans review the petitions, and in cases where the users explicitly allow, the images.
Doing this would limit another human seeing the intimate photo in well over 90% of the cases I'd bet. In addition to that, even if the photo is shared with another human, it would let the end user decide if they'd like a Facebook employee to view the image too. IE, basic user privacy. Come on Facebook. Dumb fucks.
[0] very doable, https://trackingjs.com/
You have a phone, which can communicate with your friends. It's only remotely useful for its near-monopoly on event planning, and imo that's a job for the trustbusters.
You really cannot? Like, can you not imagine one person, who doesn't realize (or worse, and more likely IMO, does not care) about the privacy implications?
It's not hard to understand at all, I think.
I guess there's a whole lot of cognitive dissonance between how fucking stupid People reliably are and how kind and admirable persons can be, though - I should probably just get used to it.
* Cross platform: I don't have to type on a tiny screen when I'm in front of a computer, and won't lose my account or my messages if I lose my phone or phone number.
* Discoverability: most people who I meet are on it, so I can reach out to people I meet at events as well as friends of friends.
I don't love messenger. I use Messenger Lite on my phone to avoid most of the snapchat/gif/events/birthdays features. But I'm not aware of any other cross-platform messaging app with that discoverability, let alone one free of privacy issues.
email isn't cross platform?
>Discoverability: most people who I meet are on it, so I can reach out to people I meet at events as well as friends of friends.
doesn't everyone have an email address?
You can view and search your message history on Facebook, so there is not really any question that they have and can read all messages. I never explicitly checked it but at least I never noticed any gaps, i.e. missing messages because I sent them using the Messenger app.
I believe to remember that Messenger is supposed to use Axolotl Ratchet for end-to-end encryption but that is hard to reconcile with the availability of your message history on facebook.com. So maybe it's - quote - end-to-end - unquote - between the phone and the server?
I never really thought about that. Or is it just not available or disabled in my Windows Phone version of the app?
[1] https://www.facebook.com/help/messenger-app/1084673321594605...
Why do you seem surprised by this? I can't fathom anything about Facebook's design that implies my posted content is hidden from employees.
I’m a FB employee and it’s definitely not possible for employees to easily snoop on random messages. Even if you found some way that isn’t properly protected, you’d get promptly fired if caught.
Why is that surprising? In the absence of end-to-end encryption (not the default for Messenger, and can only be enabled on a conversation-by-conversation basis) where the keys never leave the endpoints, how do you design a system where literally no employee can access particular bits of data?
I'm sure FB doesn't let just any employee read people's messages, and likely such access is doled out on a case-by-case basis, and isn't a global can-read-anything permission.
No, it implies that some Facebook employees have the ability to review messages which are reported as being abusive. Which is... well, kind of obvious. I mean, what did you think was going to happen when you report a message?
(I suspect that some employees have the ability to look at far more messages than just the ones which are reported, but that is not implied by what they're telling us here.)
I can imagine images of people that aren't "non-consensual intimate images" that they'd reasonably still like to be able to block being posted.
One use case I can imagine is doxxing prevention. Say there is a national news story involving some random person in some small town that exposes the persons face and name and rough location. Some group of enraged internet denizens start sharing the person's face and name all over the place as a means to spread their pitchfork & torches mob against them.
Seems like a reasonable feature to me. Recently a professor in my state made some remarks about white privilege and the doxxing of him was so violent and pervasive that he and his family had to leave the state for months, and he still gets threats and needs protection on campus.
If I take a photo of a public landscape and you happen to be there, do you have the right to prevent me from publishing it unaltered?
Keep in mind: you're not the primary subject (and no reasonable person would believe that you were) nor owner of the photo. Your likeness is not reasonably intended to alter the value of the photo.
Does a rhinocerous have a right to be forgotten?
https://www.reddit.com/r/dredmorbius/comments/25ll1v/does_a_...
The law is a tool, not an ends.
Note that I'm not personally convinced of this, I just felt that it was really weak to say that it was fine because the faces aren't an important part of the picture.
This was very intentional, because it is how courts look at the monetization of this kind of content today.
Your likeness cannot be commercialized without your consent (which is why things like photo/video releases exist) - that being said, the line determining your importance to the photograph's value is up to what a reasonable person (a judge) would believe.
This comparison, while not perfect, was used to preempt an argument from the "$ without permission" angle.
--
As a private company, Facebook is free to add this feature for its users.
As a citizen: I am free to publish my photo containing your likeness elsewhere until legislatively or judicially prohibited. And then I am free to challenge that censorship (as many have on the commercialization issue) on obvious First Amendment grounds.
Aside: I won't get into any European arguments about the "right to be forgotten" - we (USA) don't recognize this as a right.
Granted I really wish this was all done on the client so Facebook didn't gain access to the images themselves but I'm not sure of a good way around it to verify the image is something they should remove and not an abuse of the system.
Since that will be very rare case why not make that the human-required step?
Report, report, report, report, report - you know someone will do it, or make a bot to.
Or they would need to put actual effort into developing a novel solution to a new-ish problem. But they're a large incumbent, so...no.
It’s a philosophical question, to be sure.
Also, what happens with these manual reviews that turn up underage nudes? Teen sexting is a thing. For that matter, isn’t Snapchat a massive repository of child pornography?
Then the service makes the worst mistake that they actually upload nudes to their platform, PLUS they have a human look at it, which adds another layer of security leaks.
Did they do this intentionally or did they have such a huge lack of common sense when they decided how to design this and nobody in the project had the idea to address this.
If it already happened, they don't need to upload it again. Presumably this new process is for prevention when it didn't happen yet but they have good reason to suspect that it will.
There is, in fact, a whole world outside of www.thefacebook.com
"We store the photo hash—not the photo—to prevent someone from uploading the photo in the future. If someone tries to upload the image to our platform, like all photos on Facebook, it is run through a database of these hashes and if it matches we do not allow it to be posted or shared."
Also, wonder if basing things on hash will make detecting slightly modified versions of that content almost impossible. E.g. IG / YT can detect if you upload content with a piece of music that's copyrighted regardless of how much you alter it, but this would be fairly primitive.
Edit: Thinking about it a bit, I'd be way more comfortable with the idea of the image hashes being computed on-device and only the hashes being sent to the server. This opens a different door of abuse (by essentially permitting individuals to ban someone from posting an image by uploading the hash of the image), but it's definitely preferable to encouraging people to send all their selfies to the privacy commissioner.
> this would expose the hashing mechanism to analysis
Even just the rate-limiting aspect of keeping the algo on the server (in this case behind a human, but that's even not necessary) is useful: I can't sit offline with the algo and keep beating it with slightly different images until the hash changes enough to be considered unique.
There would be a browser extension to "refingerprint" images within 24 hours of release.
That's how we got good crypto. By not keeping it proprietary.
And a side benefit that we would get some amazingly alteration-resistant perceptual image hashes out of that.
Having the algorithm open for public review will lead to better algorithms, but allow for direct feedback if you are attempting to circumvent it; whilst keeping it proprietary will prevent direct feedback, but might keep the algorithm less clever than it could be.
If Facebook is using just PhotoDNA, defeating it should be trivial¹:
> […] It works by converting the image to black and white, re-sizing it, breaking it into a grid, and looking at intensity gradients or edges […]
That's fairly generic, but effective enough for resizing, colour adjustments, minor changes, etc., but not good enough for porn and an adversary willing to do some experimenting with the photo.
If I were to guess at how Facebook uses this, I would expect that they use a preprocessing routine that strips out anything that doesn't resemble a nude or semi-nude person, to prevent alterations to the person's surroundings from changing the hash — which is what you would do if you wanted to bypass the filter whilst keeping the essence of the picture intact. Keeping that fact hidden would be integral to their strategy I suppose.
Seems like a very awkward flow to forcefully repurpose Messenger to do something it shouldn't be doing.
Given what we know about data retention policies at big tech firms, I'm not so sure I would feel confident taking the press release at face value. I'd really like to see a white paper or similar outlining the specifics of how this is being handled. I'd also have questions around what steps have been taken to prevent rogue Facebook employees trying to obtain the images.
I really wish they had found a way to generate the hashes they require client side and not receive the image at all, especially as this is something that presumably would be really great for revenge porn victims.
Facebooks plan is awful.
https://twitter.com/alexstamos/status/928646228472078336
(You'd reverse engineer the algorithm and permanently defeat it - rendering all fingerprints stored useless)
Technology for searching for similar images (I have no idea how it works interally) already exists and is very widely deployed. You can easily find an image similar (uncropped, cropped, with different filter, greyed, ungreyed, with a logo in corner, etc.) to your input image on https://tineye.com/ and https://images.google.com .
If you had access to the original implementation, you could probably defeat them permanently with minimal alteration via reverse engineering. Which is why you don't have access to the original software, its specifications, or implementation details.
Now, obviously you can't prevent the actual data from being distributed. For example, you could base64 encode the image and send it as a series of messages instead.
The motivation for revenge porn is to hurt the other person. One vector for this is to post it to all of your mutual friends so that the victim is shamed. This adds a barrier so that the mutual friends have to invest work into getting and distributing the porn, and won't accidentally see it flowing through their feed.
Never in a million years would I trust that kind of data residing on Facebooks servers even fleetingly. Years back, Facebook allowed you to have private profile photos. My then-girlfriend-now-wife has a picture of the two of us as hers which for cultural and religious reasons she had private since we hadn’t decided to tell her parents yet. Facebook decided without telling anyone one day that Facebook photos were now public. She hasn’t spoken to her father since that day. Facebook can and will do whatever they want with the data they have on you. Maybe permissions change on messenger (or there’s a bug) and data from this flow leaks. Maybe a year from now this flow still exists but messager makes all picture uploads a soft delete because they want to keep things as training data. Maybe the upload gets replicated to their CDNs in a way that is discoverable and recoverable. Maybe documentation for this manual process outlives the software, someone finds it, and uploads a picture only to find out they did it in the wrong place that now has no guarantees of even an effort for privacy.
That’s a lot of maybes for something so sensitive. Facebook doesn’t deserve your trust with this kind of data. They should come up with something where the hashing happens locally and the user never has to put faith in them to begin with.
At first I wondered "Why?!" Why does a "specially trained" person have to look at the very same intimate images that their users wish to prevent being shared.
Then it struck me that without review, any image could be blocked, simply by submitting it through this process. This would also explain why the offline hashing approaches being suggested aren't practical - they're just too prone to abuse by flagging images that are not of a sexual nature, but that you would like to see blocked for whatever reason.
An ML approach to masking the most sensitive portions of the submitted images would both protect the user, and reduce the burden on those tasked with reviewing the submitted images.
If this could be handled on the client, and sent with a verifiable hash of the original image, that would solve the majority of concerns raised in the discussions here.
E.g. as the perpetrator, open the image of your ex in MS Paint, add a stripe across the bottom in such a manner as to not obscure the primary content at all, ta-dah, upload, image will not have a matching hash.
It seems like HN comment pages might have a critical mass beyond which it's impractical to see if somebody's already said what you want to say so people repeat the same idea - ballooning the comments page even further.
[1] "... a specially trained representative from our Community Operations team reviews and hashes the image, ..."
However, I think a simpler approach would handle >95% of cases here, and that’s to just calculate the image sha and use that. It does nothing for resizes/obfuscations, but if Facebook users are dumb enough to volunteer their nudes to human review, they’re dumb enough to not modify images before reposting them as revenge porn.
Everything else with respect to modified images and elaborate fingerprinting and perceptual hashing can happen at the next tier, which I trust Facebook already has an intricate (and publicly tolerated) system to handle.
Back a long time ago, if you had a Polaroid you might have some reasonable expectation that picture was the only one... but now? Once it is digital you cannot guarantee a single thing that happens to it unless you do it offline.
Why would anyone want to take that risk?
I'm wondering how long it will be before we start seeing cell phone pics of screens with peoples intimate images on them.
If you doubt this will ever happen, look up some images of "1920s beachwear" and ask yourself where the trend is going.
I'm sure they don't have a legal exception for underage photos, but those laws do vary from state to state.
A better idea may be an uploading tool where you can obscure some parts for verification.
The problem is that you presumably want the image to be blocked immediately upon upload.
But facebook is worried that people will abuse the system, and they don't want images to be instantly wrongfully blocked until someone can review it.
Abuse-resistance. Instant blocking. Perfect privacy. Pick two. There's no perfect solution.
All someone has to do is change a single bit/pixel in the image and the hash will be different. No one will notice that and it defeats the hash. Hell, you don't even have to do that. You can just rotate the image, save it, then rotate it back, and re-save it. The odds are that program that you used will most certainly not write the image data in the same exact way.
Hashes would be better for files that cannot be altered without breaking the contents. Although, these could be subject to compression, which would create a new hash.
Either way, I think they need to use something similar to Google's image search that actually examines the photo contents for similarity.
There are perceptual hash techniques which still match after changes to the file, and are tuned in a similar manner to lossy compression algorithms to prioritize comparing information relevant to human perceptions.
https://www.phash.org/ is one open-source example.