Hacker News Reader

Comment by jacquesm | Hacker News Reader

I've once spotted a (Latin American) slave trader on a image sharing platform during a technical due diligence. They were pretty shameless about it too, uploading the photos of the girls from their own IP. All the easier for the authorities to go after them.

Please take careful note: If you do not do proper risk analysis of your platform you WILL be used to perpetrate crime and if you find that you did not do what you could have done to stop it you may end up being an accessory to that crime. More so if you ignore warnings or flags by other users of the platform. This goes for anything that allows the uploading and further distribution of user generated content (especially 1-1) and for anything that deals with money.

Your nifty platform or weekend project could easily become the vehicle for someone else's crimes.

View on HN

> Please take careful note: If you do not do proper risk analysis of your platform you WILL be used to perpetrate crime and if you find that you did not do what you could have done to stop it you may end up being an accessory to that crime.

Since we're talking about image hosters (I built one some years ago): how do you solve the dilemma that a) you can be sure that at some point somebody will upload images that are illegal to host and look at, b) in order to identify and remove them you'll have to look at them or pay someone to look at them.

What we did (before ML became widely available/easily usable, not sure if there are ready-to-use models for filtering all illegal images out there these days - nobody would want to train one themselves using illegal content...) was to look at access logs and remove all images that had suspicious hostnames in the "Referer" header. It wasn't enough apparently, it got my successor into trouble and a new dilemma (take down the platform immediately, or operate it as a honeypot to help authorities who asked for it, potentially incriminating themselves for not taking down illegal content after being made aware of it... [police can't give you a carte blanche for operating a honeypot, a DA/prosecutor can still prosecute]).

Edit: it doesn't seem to be available for my country, but a service like https://azure.microsoft.com/en-us/pricing/details/cognitive-... might help nowadays.

PhotoDNA (also Microsoft).

Yeah this is about the best I've seen. Is photoDNA open source? Do you host your own copies of some public models?

You should contact Microsoft if you are seriously interested, there are a number of models depending on your use case.

Wouldn't it be reasonable to assume that if you incidentally stumble upon an image you're not allowed to look at that it is not something that you could be prosecuted over? That is, as long as you weren't going looking for that specific type of image it should be okay.

Possibly, it's for the judge to decide. How can you prove that it was accidental and not intentional? And how to do find an employee who doesn't mind checking questionable material (even pre-filtered/flagged by a ML model) every day?

>How can you prove that it was accidental and not intentional?

I imagine that if you happen across it while moderating then you won't really know what you'll find, because you're relying on reports or random browsing to find offending content. That is, as long as you're following reasonable procedures for moderating then that should be good enough, no?

So, what is a reasonable procedure? Look at content reported to be illegal yourself (possibly already illegal since you had prior notice it was illegal, certainly not good for being able to sleep at night) and then delete it? Or delete it without checking whenever it gets reported and thereby giving users a tool to sabotage other users by misreporting their content (fun for political discussions etc.)?

Not a lawyer, but pretty sure this depends on whether the crime is “strict liability” or requires intent.

> Your nifty platform or weekend project could easily become the vehicle for someone else's crimes.

We might as well all pack up and go home then. God forbid some group coordinates an atrocity using the comments section of our blog, for want of proper risk analysis.

Best leave it all to the oligarchs. They know what they're doing. They'll look after us.

The comment promotes awareness of the problem. Your interpretation does not stand up to that.

If you see packing up and leaving it to the oligarchs as the only workable solution to adress the valid point the parent raised, this says more about you than the argument at hand.

The thing with weekend projects is: they tend to lie around unused and unchecked. If you are aware of this, people can mitigate it by adding propper monitoring etc.

> If you are aware of this, people can mitigate it by adding propper monitoring etc.

And what is an acceptable, reasonable solution? Even FB with their near-infinite resources gets in trouble all the time for illegal content and they still spread all kinds of it. Is the only possible solution to be a big corporation?

Just considering this possibility and asking this question a huge first step.

Ethics in CS is important just like any other field. I think just taking a class to be aware of it helps a lot.

FB doesn't care about illegal content. They may say they do, but they don't as long as they're making money -- that is all they care about.

What would be your opinion in building a nifty side project platform upon keybase.io or other services that provide authenticity verification to a certain degree?

Regardless of what the actual situation is, I wonder what people think it should be, and how far towards either direction (anarchy and whatever the opposite is) we'll go.

For example, What if you sell someone some pencils and they stab someone to death with them?

If I run a forum about gardening and someone is using the PMs as a slave market, am I responsible for not monitoring every single message on the platform, and should I be?

What if I own an office and I rent it out for people to have meetings, and they meet there in person and hold a slave market, am I also supposed to be monitoring their in-person meetings?

If you have trouble telling all these apart I suggest you don't start anything online. One thing is for sure: sticking your head in the sand and pretending there is no problem isn't going to work in the longer term. See TFA.

1. You didn’t have to be rude, did you?

2. If someone asks where to draw the line along the sequence of a|b|c|d that does not mean they cannot tell these apart, it means they are pondering which of these are ok and which are not. Distinguishing things and bucketing things are not the same activity.

As to (1), I've seen these so called thought experiments used more often than not to try to defend inaction. This includes free speech and all kinds of other important principles that then get stretched into the most warped definitions on order to justify acting in a way that benefits the bottom line. See Mark Zuckerberg's recent performances for some samples.

As for (2). It's simple: there is the law and there is ethics. Anything illegal you should work hard to stop and collaborate with LE to get it stamped out on your properties. Anything unethical is up to you where you draw the line. Ethics come at a cost.

1) Your assumption of bad faith is unfair, baseless, and runs afoul of HN guidelines [a]. I see nothing but attempt to discuss and understand where we should stand on these things. Are you reading whitewashing into those questions?

2) Anything unethical is up to you where you draw the line. Seriously? You do not think one should be discussing boundary of ethicals choices with one's peers? Please explain.

[a] Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

> Assume good faith.

Sorry, but you're the one not assuming good faith to begin with.

The key points could be how you react when other users report these activities to you, or you have tangible signs that would be suspicious to an average reasonable person.

These problem would be the same if you had a hot dog stand and customers reported to you a child prostitution ring massively buying your hot dogs for the kids. What would you do ?

you inform the police, and ask them to investigate. You cooperate when feasible, but not at a cost that affects your business's ongoings.

> What if I own an office and I rent it out for people to have meetings, and they meet there in person and...

If you are the property owner, taking care that the people on your property is safe.

As a steward of the land that’s your job. By purchasing the land you are saying: no one else has title to watch over the people on this land, only I do.

So yes, you then have a responsibility to do so.

That’s one of the many reasons I don’t implement a bunch of my side projects, because it inevitably leads to me trying to figure out how to police and remove gross content from whatever website I build, and I don’t want to have to deal with that.

Spot on, it is also the reason I shut down two websites that I ran for years. You really need to be on the ball, preferably 24/7 so many things that would be nice side projects simply can't be left up and running without oversight. It's a rotten world.

Oh gosh I have a project I should check on. If I can find the credentials...

edit: oh no.

Ugh. Need help?

First, thanks for offering, I appreciate that!

But no, I think for the most part it was pretty benign. It looks like they were planning on doing some kind of maybe SEO gaming or something using my de.licio.us clone I made almost a decade ago. I had about 23,000 accounts with .ru and .lv email addresses, many repeats, almost none of which interacted with the app at all.

Many recently created, even today. I think there must be a script that loads in users or something like that.

Ok. Happy to see it is that kind of trouble, and not something far more disturbing. Good on you for taking immediate action after becoming aware of the problem.

This is also reason I abandoned freemium and open C2C for side projects in general.

That seems a bit harsh. If I own a building and somebody sprays an antisemitic or jihadist message on its wall, am I now complicit for spreading racial hatred (assuming in Europe, in US it's probably protected by constitution)?

Are you trying to be obtuse on purpose?

Clean the building.

And if I, for whatever reason (for example, am broke and can't afford it) don't want to? Am I complicit then? It would be interesting to ask lawyers, because it's a very similar case.

Then you are derelict in your management duties. If you are broke then you don't own a building. If you do own a building then you are not broke and presumably can pay for upkeep. Even so, in many municipalities if someone vandalizes your property the municipality will do the cleanup.

What kind of technical due diligence did you do?

I'm wondering if machine learning could be used for this.

We review companies from a technical point of view for investors so they have some idea of the risks associated with the company prior to investment. Very interesting but super intense work.

Reply on news.ycombinator.com