Facebook asks users for nude photos in project to combat revenge porn | Hacker News Reader

Facebook asks users for nude photos in project to combat revenge porn

theguardian.com

45 pointstrueduke92 comments

Open articleView on HN

Zuck: Yeah so if you ever need info about anyone at Harvard

Zuck: Just ask.

Zuck: I have over 4,000 emails, pictures, addresses, SNS

[Redacted Friend's Name]: What? How'd you manage that one?

Zuck: People just submitted it.

Zuck: I don't know why.

Zuck: They "trust me"

Zuck: Dumb fucks.

Wasn't he like 19 when he said this?

I sure as hell said some edgy stuff like this when I was 19.

H̶i̶s̶ ̶a̶c̶t̶i̶o̶n̶s̶ ̶a̶f̶t̶e̶r̶ ̶i̶t̶ ̶l̶e̶a̶k̶e̶d̶ ̶s̶o̶r̶t̶ ̶o̶f̶ ̶i̶n̶d̶i̶c̶a̶t̶e̶ ̶t̶h̶a̶t̶ ̶t̶h̶i̶n̶g̶s̶ ̶h̶a̶v̶e̶n̶'̶t̶ ̶c̶h̶a̶n̶g̶e̶d̶ ̶m̶u̶c̶h̶:̶https://www.recode.net/2017/1/5/13987714/mark-zuckerberg-fac...

Edit: Ugh. The witchhunt above was unrelated to those IM messages. Apologies for the mis-step.

I mean, his worries then are still worries now. Except instead of arbitrary actions from a 19-year old, I face profit focused actions from a giant ad company.

Not sure which is better.

I agree, I don't want to sound like a Zuckerberg apologist, but it's simplistic to view the Zuckerberg and Facebook of today as we would a 19-year-old Zuckerberg and his elaborate PHP script.

As a trivial example: In 2005, when Facebook was a non-trivial company, Zuckerberg [0] guest lectured at a Harvard CS50 class. When asked if Facebook would contribute to open-source, he said that he didn't foresee it being worth the trouble (can't find the exact timestamp, so this is all IIRC with a grain of salt). Now of course, open-source is a substantial part of Facebook. Is it because Zuckerberg in the following years had a Road to Damascus experience with Richard Stallman? Maybe, but it's more likely that Facebook evolved into the type of organization where OSS became a benefit to the bottom line, and it was a decision made by people lower than Zuckerberg at that point.

Even if Zuckerberg is still as much a creep as he was in private IM messages as a 19-year-old, he's no longer the sole captain of his tiny boat. Him breaking the law means that many people end up getting in legal trouble, i.e. it doesn't really much matter what he alone thinks is moral when he has dozens of people/potential whistleblowers looking over his shoulder with greater moral concerns.

[0] https://www.youtube.com/watch?v=xFFs9UgOAlE

What about that article suggests that he has or hasn't changed? It seems to seeks to stir up outrage, that it's somehow sinister to expect that when you tell somebody something in confidence, they keep it confident - and if you don't, you get found and fired, because keeping confident things confident is a condition of employment, and not a terribly onerous one.

S̶t̶a̶r̶t̶i̶n̶g̶ ̶a̶ ̶w̶i̶t̶c̶h̶h̶u̶n̶t̶ ̶t̶o̶ ̶f̶i̶n̶d̶ ̶a̶n̶d̶ ̶f̶i̶r̶e̶ ̶t̶h̶e̶ ̶p̶e̶r̶s̶o̶n̶ ̶t̶h̶a̶t̶ ̶l̶e̶a̶k̶e̶d̶ ̶y̶o̶u̶r̶ ̶o̶l̶d̶ ̶s̶h̶i̶t̶t̶y̶ ̶I̶M̶ ̶m̶e̶s̶s̶a̶g̶e̶s̶ ̶i̶s̶ ̶p̶e̶t̶t̶y̶ ̶a̶n̶d̶ ̶i̶m̶p̶u̶l̶s̶i̶v̶e̶.̶ ̶ ̶ ̶W̶h̶i̶c̶h̶ ̶i̶s̶ ̶e̶x̶a̶c̶t̶l̶y̶ ̶w̶h̶a̶t̶ ̶t̶h̶e̶ ̶o̶r̶i̶g̶i̶n̶a̶l̶ ̶I̶M̶ ̶m̶e̶s̶s̶a̶g̶e̶s̶ ̶w̶e̶r̶e̶.̶

Edit: What's above isn't quite right. There was a witchhunt, but unrelated to the leak of those old IM messages. Apologies. My point below, though, stands.

A mea culpa about the messages, and how you've changed personally in the meantime would have been a change.

It wasn't anybody's "old shitty IM messages", it was details about a product under development. The IM messages has been out there since the very early days. The linked article references the "dumb fucks" IM messages exactly not at all.

And looking for (and finding) a specific person who has violated a specific, well-understood rule and sanctioning them is not exactly what a witch hunt is.

Yep. I updated it long before you commented. Apologies for the miss there. Read it wrong.

>A mea culpa about the messages, and how you've changed personally in the meantime would have been a change.

The value of mea culpas is under-appreciated. We all make mistakes, and you can't change that in life. But you can change course after recognizing a mistake has been made.

I personally find that the worst people I have interacted with in life are also those who cannot assert a mea culpa.

> Wasn't he like 19 when he said this?

Isn't the point of Facebook to keep such kind of comments forever in each individual profile? I don't see why Zuck should be immune to something he benefited from in the first place.

I'd argue that's more the point of the internet in general, these days. It's extremely hard to remove something from the internet.

We as rational humans though, can apply context to such situations and realize that teenagers often say insanely stupid things that they later wish they hadn't (and usually didn't mean).

While it's understandable that to get something deleted from Internet is next to impossible that's because there's no Internet Inc. But Facebook is a monolithic network with full control of its walled garden.

It is pretty clear they are not interested and letting us really manage what we posted on its walledgardn. There's no way for me to mass delete/edit posts on facebook. Yes, facebook did not force me to post things 10 years ago but it seems like they are forcing me to hold my own post against me for eternity unless I spend years of my time to go through one post at a time.

Delete facebook profile? well it doesn't really delete it: https://www.quora.com/When-you-permanently-delete-your-Faceb...

Also, privacy cant really be locked down. If you dont want people to see your friends, they will still show up on friend recommendations. Same kind of thing with groups, etc

Ah, the 'young kids often say stupid things" argument.

Zuckerburg continues to only do what's in his best interest, even at an older age. i.e. not combating racism, harassment, or abuse on their website because it'd hurt their bottom line. This a business and everybody is willingly giving him products that exploit his user base.

Sure, but I don't see how that relates to my intended point that things we say as teenagers probably shouldn't be held against us indefinitely.

If you want to say "Zuckerberg is selfish because he refuses to affect Facebook's bottom line to stop harassment" that's fine with me, and I won't disagree with you.

If you're saying "Zuckerberg stole someone's lunch money when he was 12 therefore he's a sociopath" I dunno, that's less convincing to me.

IM transcripts that occurred 12-13 years ago, when he was 19 and when Facebook didn't need a COO or other corporate structure since it had a userbase of ~4000: https://www.theregister.co.uk/2010/05/14/facebook_trust_dumb...

Does anyone have a first-hand source detailing this project, and especially its privacy and accountability principles? This is almost surreal to read:

> A community operations analyst will access the image and hash it to prevent future instances from being uploaded or shared.

How long before the first community operations analyst begins selling the images they access? How long before the first one uses it to blackmail an already terrified victim? And when -- not if, when -- one of these things happen, what price will Facebook pay (I mean, with corporate accountability being what it is, we all know what price Facebook will pay, but it would be useful to at least have a fine print...)?

The hash could trivially be done client side and then only the hash uploaded. If they make it an open source browser plugin, everyone would know if it was trustworthy or not.

There are some relatively easy countermeasures you can put in place to avoid this. Never display identifying information with the images. Never allow analysts to freely lookup images from specific profiles. Make sure the terminals that these analysts work on are blocked from internet access, and have no accessible external drives or USB ports (or run a locked down virtual desktop on a thin client). Maintain a no-camera policy in the workspace, and a policy of always having more than one person on duty in the workspace at the same time.

Considering that we haven't seen any stories of employees abusing backend access to the Facebook servers, it feel reasonably safe to assume that they have countermeasures like these in place.

That is a disaster waiting to happen.

Not really. This is well-understood problem with well-understood solutions. There is no reason to believe that Facebook doesn't have both the resources and the motivation to get this right.

I'm highly skeptical about it. Everyone has motivation to avoid getting hacked or have employees/contractors do bad things, but it happens.

Here is counter measure 1: "Don't upload".

Though I, personally, prefer counter-measure two: terminate everyone who approved/supported/championed this project

First, yes, there are several countermeasures that can be implemented. That is no indication of the fact that they are implemented. There are some relatively easy countermeasures you can put in place to avoid any security incident, and yet hardly a day goes by without reading about one in the news.

There are companies that (supposedly) implement various such countermeasures, and have a very strong incentive to do so, as they handle private, and potentially damning data of important people with millions of dollars to lose in case of a data breach. Just look at Panama Papers, Paradise Papers and the like. If that sort of data can be treated carelessly, I find it somewhat unrealistic to expect better treatment (even from Facebook) of private data belonging to terrified teenagers who effectively have no recourse against misuse of their pictures.

Second, it took considerable time for the fact that former NSA employees used the agency's surveillance tools to spy on their ex-lovers, too. The fact that we haven't yet seen any stories of employees abusing backend access to the Facebook servers does not tell us much. In fact, given how much information is publicly shared (sometimes unwittingly) via a Facebook account, it may be hard to tell when such an event occurs.

Wait, what? I submit my nude pics to FB, so that they can hash it and identify its copies online? That sounds good, but is that the only way of doing this?

Why not use facial recognition + nudity detector to on-demand find potentially compromising videos online? Maybe even send my Facebook notification that says, "Hey we found what appears to be a compromising video with you in it shared to a wide group of people, are you cool with it?"

Am I mis-understanding the concept being proposed?

I like your idea better. For an SV darling like Facebook (and especially since this is Facebook), what the article describes sound incredibly manual and low-tech.

But what really bothers me is

> ... a community operations analyst will access the image and hash it to prevent future instances from being uploaded or shared.

Of course, this community operations analyst will not stash those photos in a flash drive somewhere, right? Because the humans FB hires are 100% trustworthy. /s

What's "SV"?

"Silicon Valley"

What facial recognition technology do you have in mind that would not face the burden of frequent false positives (nevermind false negatives) and could scale to do this efficiently on an on-demand basis?

Then there's the issue of how your proposed system requires FB to show users the porn that other users are privately viewing.

Facebook already uses facial recognition to suggest tags on photos, so I don't think false positives are that much of an issue:

https://www.facebook.com/help/122175507864081

You other points are true though, and I guess a lot of these type of photos don't always contain faces.

I'm not as heavy of a FB user as I was 5 years ago, but my impression was that false positives were at an acceptable threshold, but not nearly a solved problem. And the face recognition tech was heavily augmented by social data, such as the fact that me and a friend were both partying at the same place on the same night and that we had just 3 days ago posted and tagged photos of each other.

That secondary data does not exist reliably in the context of revenge porn videos that go viral. Nevermind that the penalty of a false positive -- which would presumably involve exposing the user to a bit of random porn -- is substantially higher than it ever is with mistagging friend photos.

>> Then there's the issue of how your proposed system requires FB to show users the porn that other users are privately viewing.

I think this can be addressed as "Show me all suspected compromising videos of me that is available to more than 10 users". The answer to that query can be produced with facial recognition and nudity detector, no?

You don't think falsely-identified users would be bothered by notifications to check random porn videos? The same userbase who freaks out (somewhat justifiably) when Facebook suggests friends/connections that the user believes (perhaps wrongly) that FB could not know independently?

The goal is good, but I'd never under any circumstances entrust photos I don't want shared with any of the cloud companies. At the very least the fingerprinting should be done on-device?

Why not just send the hash?

Anyway, does this mean that I can take down other people's legitimate photos by submitting them to this service?

see, that's why they need to "manually" verify the photos. convenient, isn't it?

Or they could use image recognition to detect adult content, like Google's Cloud Vision API

https://cloud.google.com/vision/docs/detecting-safe-search

EDIT: Just read the part that says a "community operations analyst" will hash and analyze the photo. That's terrible!

Have you used that API? According to the documentation, this is the JSON response:

    "responses": [
    {
      "safeSearchAnnotation": {
        "adult": "VERY_UNLIKELY",
        "spoof": "VERY_LIKELY",
        "medical": "UNLIKELY",
        "violence": "UNLIKELY"
      }
      }
     ]

How is that useful for efficient detection of what the OP describes?

edit: "useful" as in, how would that be granular enough for what FB would need? I don't mean that FB literally uses Google's API/endpoint response, but that FB isn't trying to detect and prevent adult content, but malicious content that invariably happens to be adult content.

There's no way around putting a human in the loop. The ML adult filters are nowhere near as good as a person. Try Google Images with safesearch on, it's nowhere near sfw.

If the system allowed users to send the hash, then yes, it would allow users the power to arbitrarily take down photos. The point of the users having to post these images from their own account is so that a FB employee can confirm that the content breaks FB rules and that the content is associated with the user.

In other words: The solution to preventing strangers from looking at your nude photos is to have strangers look at your nude photos? Brilliant!

I've only read the posted article so I don't know if "revenge porn" is the primary or originating motivation for this initiative (as opposed to child porn victims, etc).

But my perception is that victims of revenge porn -- the ones who feel the need to make a complaint (and risk drawing publicity to themselves) have had (or perceive) their images disseminated so widely to such psychologically devastating effect [0] that the thought of a FB employee seeing them is among the least of their worries. Probably because FB employees have already seen them if the user has submitted an abuse complaint.

[0] https://www.theguardian.com/lifeandstyle/2015/feb/06/experie...

I'd assume that any matching photos would be manually verified to be of a nature possibly related to the subject at hand, "revenge porn", if the uploading user clicks the "hey, I should be allowed to post this" button?

No, it should probably work like this:

1. User submits hash.

2. Other user submits photo, which matches the hash, which is detected by the system.

3. Facebook employee is notified. Looks at photo.

4. Photo is removed.

This way, the employee only looks at the photo if it was actually submitted. If (2) never happened, then (3) never happens.

I agree, that sounds like the better solution. Maybe it's not as user-friendly to have the user go through the hashing process client-side (I assume for the average user, it'd require providing a separately-downloaded application, which presents its own security risks), but Facebook hasn't had a spotless record when it comes to manual review workflow. Your suggestion would seem to at least greatly reduce the initial workload (assuming that many/most submitted hashes are matched only much later, if at all).

Because you can't verify a hash as you can a photo.

Otherwise I could hash and submit the Disney logo.

You could rely on hashes if the downed images were themselves analysed for nude content. Given the state of AI/ML, this is fairly tractable (and has been for a decade with reasonable levels of accuracy).

Persons found abusing the reporting system could have their own trust revoked -- their hashes would be considered noncredible absent independent verification. Similar standards apply, e.g., in witness testimony ("impeaching a witness").

So you're saying humans will be verifying nude pics submitted by users? Because that doesn't sound better.

Humans already are tasked on a large scale with manually verifying reports for user-submitted content:

https://www.theverge.com/2016/4/13/11387934/internet-moderat...

> When Dave Willner arrived at Facebook in 2008, the team there was working on its own "one-pager" of cursory, gut-check guidelines. "Child abuse, animal abuse, Hitler," Willner recalls. "We were told to take down anything that makes you feel bad, that makes you feel bad in your stomach." Willner had just moved to Silicon Valley to join his girlfriend, then Charlotte Carnevale, now Charlotte Willner, who had become head of Facebook’s International Support Team. Over the next six years, as Facebook grew from less than 100 million users to well over a billion, the two worked side by side, developing and implementing the company’s first formal moderation guidelines.

I'm not sure there's currently an open source version of something like PhotoDNA[1] that could run locally.

A normal file hash is probably not useful in this scenario.

[1]https://www.microsoft.com/en-us/photodna

I think you're looking for something like the pHash[0] library. I've used it in the past for thumbnail generation and it works very well.

[0] http://phash.org

>>Why not just send the hash?

Is there an app for that? /s

To be honest, this is a great idea. I think its something that should be built into OS's -- select a file, 'calculate WorldHashValue' / 'publish WorldHashValue' should just be a menu-item ..

On the other hand, malevolent people could modify the image until its hash changes (e.g. by cropping, changing colors, etc.)

This should be really difficult for images because different devices saves them different. The iPhone 7 and new use the HEIC format, however they are uploaded as jpeg to my iPad and Mac because they don’t support the new format (no hardware optimization). Now chances are, when I share it on my iPhone, it regenerates the jpeg again, but a bit differently, yet unnoticable by humans. This makes it impossible to use a conventional hashing algorithm for images because an undetectable change for images because an invisible change can change the hash and even alomost every byte and yet it looks completely the same.

Of course, every user can do that knowingling, just crop the image a very tiny bit or change one color of one pixel and the hash is going to be completely different.

You could possibly use a perceptual hash[0] to compare images. While building an image processing application I used the pHashion library to compare thumbnails to their originals to make sure it still looked "the same" to humans. It worked very well and might be useful in the case of comparing "nude" photos to a reference.

[0] http://phash.org/ [1] https://github.com/westonplatter/phashion

I just love the idea of putting the very things you want nobody to see into the hands of the one company that already knows everything else about you. Better hope you never, ever try to engage in a political campaign that fb doesn't approve of... m(

Sounds almost like a joke doesn't it?

I feel like many people won't be super comfortable with Facebook having their nudes on top of everything else.

I wonder if they're doing anything to take into account tiny modifications causing a completely different hash, or if they are relying on users not knowing that changing an image would get around a naieve hashing strategy.

They don't care how bad the strategy is. The goal isn't to combat revenge porn, albeit that is a noble goal, and one that individual actors will care about. The goal here is to partner with governments, 3-letter agencies and others to condition people to accept ideas about their privacy rights that we would normally find exceedingly objectionable.

FTA (emphasis mine):

> The technology was first developed in 2009 by Microsoft, working closely with Dartmouth and the National Center for Missing and Exploited Children to clamp down on the same images of sexually abused children being circulated over and over again on the internet. There was technology that could find exact matches of images, but abusers could get around this by slightly altering the files – either by changing their size or adding a small mark.

> PhotoDNA’s “hash” matching technology made it possible to identify known illegal images even if someone had altered them. Facebook, Twitter and Google all use the the same hash database to identify and remove illegal images.

I expect that the engineers developing this would go for a hash based on an abstraction of the image represented, not the actual bits of the image file.

To prevent your nude photos from getting on Facebook, send your nude photos to Facebook.

    with(image):
       post = facebook.post(image)
       if post.is_deleted():
          image.is_blackmail = True
          otherplatform.post(image)

Hey Zuck, Are you concerned that your nude photos might be used against you at some point? Please post them here so we can know to take them down if someone ever uses them against you. Gee, I guess that doesn't quite make sense does it?

I've got One Weird Trick that Facebook hates, to keep nude pictures of myself off the Internet. You're never going to believe it. Ready for it? It's two easy steps:

1. Don't make nude pictures of yourself or let someone else make one.

If you manage to screw up step #1, then:

2. Don't let other people have those nude pictures or upload them to the Internet.

That's it! Can you believe that this has actually worked for me for a good 20+ years so far? I know it seems too good to be true, but trust me it totally works. I should blog about it.

Submitting your nude photos to a cloud company, what could go wrong.

Facebook? The company that maintains shadow profiles and does not delete anything that its users "delete"? That facebook?!

Every time I think I met the Peak Stupid, Internet proves me wrong.

Assuming they are hashing the images or videos and only comparing the hash in practice, why doesn't Facebook implement a client side JS method for the hash generation. Only thing uploaded is the hash.

Use standard ML Algorithms to validate if the image includes nudity and confirm the hash overlap.

Also need to be hesitant since people at FB will be able to see these images (as recently mentioned in another post on HN where some random folk ended up viewing a Google Doc shared via Messenger)

This can be framed as the requirement for photos to be nudes in order to be blocked from the platform. I'm assuming that there are checks on uploaded images to see if they are actually nudes before they are added to the dataset of images to be banned. Otherwise I could use the system to prevent any photo at all from being uploaded to Facebook by submitting it to the system. This might actually be a good thing, for me, but not for Facebook.

This doesn't really make sense. Isn't machine learning already good enough to detect nude photos? If nude photos are not allowed on Facebook, why do they need hashes of your specific photos? They could identify all nude photos with machine learning and remove all of them.

Or have a checkbox in the profile settings: Do you permit other people to upload nude photos of you? [ ] Yes [x] No

It'll be used for private messages, I assume. I guess the rules aren't as strict there.

They have facial recognition data though. If the uploading user is not the user in the nude photo, and the user in the nude photo has not checked a box that says "I permit other users to upload nude photos of me" then block the upload and flag for review.

It's hard to identify a person when the set of possible values is "every person on the planet" and not "one of these few persons". They could detect if the uploaded looks like one of the uploader's Facebook friends, I suppose, and that would probably be useful.

What would they do if they detect a possible match, though? Ask the person they think is in the photo "is this you, are you ok with John Doe sending this photo to Jane?". That could lead to trouble when the system gives false positives.

Couldn't you just take a screenshot of the original pic to get around this? The new image would have a completely different hash. Or just resize it. Or do some color correction. Or a million other ways to alter the picture.

As absurd as the idea sounds on its face, it seems well-intentioned with the potential to result in a net positive for victims. The main concern is if FB is to be trusted. But what’s the alternative? Is there a way for the user to hash the image in their own machine while allowing FB the ability to discern whether it qualifies as being off-limits for FB content? Without FB verification, the system is likely to become a decentralized and easily-abused version of Youtube’s Content-ID.

edit: Downvotes without suggestions or rebuttals? That's disappointing. I didn't say that this system is without flaws (or worse, unintended consequences). If you think the status quo (whatever that status quo is) for the anticipated victims, then let's hear it.

My solution is for people to be careful with their nude photos or if they weren't, then face the consequences. Either accept it (nude photos have power over you only as long as you're vulnerable to shame & guilt over them), or go through the established LE process to get them taken down and the offender actually punished.

Duplicate of https://news.ycombinator.com/item?id=15648080 and others.

Why don't they just compute hash on user's side?

Data mining for VR content generation?

Got to admire their nerve though

What could possibly go wrong?

Complicated solution for a simple problem.

Don't let your partners take photos when you are intimate. End of story.

And how is this gonna prevent them from taking your photos via hidden camera?

I'll go out on a limb and say that hidden-cameras represent < 1% of revenge-porn.

As such, the parent comment goes a long way towards protecting you.

Think of it like a padlock: it keeps honest people honest.

At that point they are committing a crime with little grey area for them to hide in, compared to a picture taken with consent.

But publishing either is a crime AFAIK. Now it's two crimes.

US: Only in some states. A casual search puts it at around 30 states. Most of these laws are new and relatively untested, legally. Not something I'd want to depend upon to protect me.

On the other hand, unauthorized photos taken of you in a place you'd expect privacy is a fairly well explored body of law; basically unless you're a celebrity figure, it's a no-no.

IANAL, of course.

It's not a simple problem since relationships are not simple.

No, it’s not simple, but it’s a starting point.

And apparently an insufficient one if this is a problem that has affected enough people for FB and gov't authorities to propose a ham-handed initiative. Note that I'm assuming that this isn't all a FB conspiracy to get users to send them their nudes because that is already what FB users are doing via Messenger on a daily basis.

There's a tendency to dismiss partial solutions on the grounds that they don't solve every problem. I find this disturbing.

Surely, not letting people take nude photographs of you is both (a) common sense and (b) likely to make a significant dent in the problem.

Reply on news.ycombinator.com