Hacker News Reader

Comment by jacquesm | Hacker News Reader

- anything that allows file upload -> porn / warez / movies / any form of copyright violation you care to come up with.

- anything that allows anonymous file upload -> childporn + all of the above.

- anything that allows communications -> spam, harassment, bots

- anything that measures something -> destruction of that something (for instance, google, the links between pages)

- any platform where the creator did not think long and hard about how it might be abused -> all of the abuse that wasn't dealt with beforehand.

- anything that isn't secured -> all of the above.

Going through a risk analysis exercise and detecting the abuse potential of whatever you are trying to build prior to launching it can go a long way towards ensuring that doesn't happen. Reacting very swiftly to any 'off label' uses for what you've built and shutting down categorically any form of abuse and you might even keep it alive. React too slow and before you know it your real users are drowned out by the trash.

It's sad, but that's the state of affairs on the web as we have it today.

View on HN

FWIW I made a website for our yearly project conference that allowed anybody to create an account and post material. But 1) all postings had to be moderated until the account was verified (either manually by me or by a code sent only to conference attendees) 2) I was pretty active in monitoring new posts and deleting posts and accounts that were clearly spam.

And of course the normal stuff, links all prefixed with "nofollow", only whitelisted markdown content / HTML, &c.

The first year we had a handful of spammers try to post stuff, but it didn't take much time at all to filter it out. This year we didn't have any spam accounts at all.

So my suspicion is that getting ahead of the "spam problem" by doing heavy manual moderation early on is an investment. If he'd just spent 5 minutes a day looking at the accounts and deleting rubbish, he would never have been swarmed by spammers, and the moderation load would have remained relatively low, until he got popular enough that he could afford to actually spend some real effort automating / crowdsourcing the spam-fighting capabilities.

The key here is manual moderation. At NodeBB we also have had our fair share of spam companies trying to build scripts to post things, and the only foolproof solution is manual moderation via a post queue for new users.

The downside, of course, is that they takes effort to maintain, and is a barrier to entry for new accounts.

Unrelated, but seeing NodeBB reminded me of the old days when installing forums for communities. phpBB2, vBulletin, downloading plugins and themes, etc. Oh, the nostalgy for the old web is amazing...

I recently got a raspberry pi and set up phpbb using docker just for a fun side project.

I looked at nodebb might give that ago next

Do you get a lot of people reposting the same thing because they think it didn’t work the first time?

I wonder if you treated new content from unverified users like a temporary shadowban (they see it, nobody else does) how that would affect behavior.

Great idea!

Edit: Why does shadow banning feel like such an elegant solution? Everything has tradeoffs and I feel like shadow banning has tons of upside and very little downside. What am I missing?

The only risk I can think of is that someone will upload the video and then livestream its playback on your platform with your branding using a separate livestreaming tool.

I.e. shadow banning semantically cannot work with a site that is geared toward publishing content, because the content is supposed to be visible to visitors who are not authenticated. If it isn't visible, that is painfully obvious.

Shadow banning only works when only authenticated users can see any content at all; then we arrange for only the offender to see content they have created. This works as long as the offender doesn't create multiple accounts.

(You need something slightly more clever, like allowing the content to be viewed from the same IP address as the last known address for that offender, plus some surrounding range (e.g. IPv4 class C subnet)).

The only one I can think of is that accounts may have extremely objectionable content and it remains there. That could be a legal liability.

There are cases in which explicit and obvious moderation results in retaliation, including DDoS and hacking attacks.

Various soft fail mechanisms, including shadowbanning, degraded performance, errors, authentication failures, etc., may help avoid this.

The question is what adversarial model you're facing: is it some random pr0n / SEO / affiliate / fraudster, or is it a "nice website you gots heyah, be a shame if anyting happen' to it" squad?

Former can be modded away. Latter takes nuance.

It’s kind of easy to check if your posts are visible for not logged in users so you aren’t fooling bots and spammers, just real people. And the bots don’t care about moderation drama but the real people do, no matter if the moderation is proper or not. It always looks really unfair.

That would result in more spam to be moderated. The spammers would see their content themselves, think it was working and mark your site as a target.

One point of manual moderation is that spammers know it is happening.

This invites a couple of questions. One, do the spammers care that the content is actually visible? Two, does spam come from more accounts or the same accounts?

I am assuming checks are rudimentary. Post some spam. See if spam is visible. Add site to you script for mass posting spam.

That's actually a wonderful idea. I'll see if I can get that implemented, and cc you on the commit.

We're always looking for little things that set our software apart!

As for your question: no, with the appropriate messaging, we don't often have repeated submission attempts.

Interesting, thanks for sharing. I'll be launching a service that allows users to upload content and I've been thinking about how to handle anti-spam. I really appreciate you sharing your experience.

Manual moderation in this way could open you up to other forms of liability that you may have easier defense against without it.

I don't think it'd be an issue for a hardware forum, although mass approving comments and finding that one is defamatory about person or company X could get some lawyer's drool glands working for example..

Also section 230 appears to be under threat by the current administration and how things are interpreted - and I just read where Biden has been talking about removing it completely..

So this situation is in a state of flux at the moment I believe.

/not a lawyer / doctor yada yada

This is a common misconception of section 230. Section 230 absolves platform companies from responsibility over speech on the platform.

It does not care about what is moderated out. No one is responsible for speech that isn’t said.

People complaining about "censorship" have been scarily effective in spreading this misconception of Section 230.

I am trying to understand this comment. I do not claim to know everything about 230 - but have read several news stories about it, in relation to it- and have sat in an a seminar by a high dollar lawyer once.. Since I did not stay in a Holiday Inn express I can't claim that I know it all - haha which is also why I put not a lawyer / doctor / yada yada - so people would research it more.

I believe this would be more of an issue for some kinds of sites and much less an issue for others. (I doubt tom's hardware has ever had anything defamatory or libelous in posts to worry about ) - other places that moderate may do so in ways that are over-zealous in protecting the publishers and therefore that would create a different issue, but that's a different discussion.

Anyhow if you are referring to an anti-censor group or some kind of misinformation army like the bots and confangled news stories around the net neutrality debate, I would like to learn more about this/them and see why / how etc. thanks

From what I have heard and seen - there is a difference between moderating each message before it's posted - eg - classified ads person answers phone for local paper tells you that you can not advertise a tiger for sale or put 'foreman' wanted in ad - it must be gender inclusive.. is one type..

Then you have moderating where people can post about anything - but when you get a complaint you research and take it down.

There are other forms and types of moderation between these.. which is why I said "in this way could open"

if you are moderating each message before if goes out, you are not acting so much as a platform as more of a publisher.. you could claim you did not know that an ad you published was not legal.. say it was turtles for sale, or someone offering weed for sale with 'codewords'.. you could claim that you did not know the girl in the ad not in a sex tape, that you just published the info...

But from what I have seen, if you allow everything to publish and take down stuff when notified you have a difference situation in regard to section 230, and claims in general, as compared to having to defend that you ad person could of should of known before publishing..

Indeed - section 230 doesn't care so much about what is moderated out in some ways, but the fact you are moderating out 8 year old's dating profiles, but not 13 year olds could become an issue- and the fact that you moderated out the 8 year olds in the first place could cause holes in certain defenses.

So you would not be responsible for that isn't said - but the fact you allowed one defamatory thing to publish while hiding a defamatory post about someone else - what isn't said could be an issue to prove that you have liability for what is.

Hope I'm saying that right, again this is my understanding of some issues that I have been involved in, and studies other people's issues and listened to their counsel - but I am not a lawyer.. if you have one and you are doing this kind of moderating perhaps have them look into the backpage cases and see how they pierced the "we're a platform, not an editorial publisher' thing. for one example.

100% this. As soon as an IP / Service appears for the first time it is scanned and hammered to look for low hanging fruit. Once the initial wave of exploit bots have found out they don't get far, your 'thing' is marked as 'too much faff' and malicious traffic drops off and is left alone (usually, y.m.m.v etc etc etc).

My experience is different. I manage a wiki. It was a wonderful success in terms of getting people to contribute, however the wiki itself (moinmoin) has lousy authorisation and authentication tools, so I wrote my own. A few versions later changed to moinmoin broke those tools, but by that time most of time content has settled down so I just reverted locking down the system by making the user database read only (literally: "chmod -R a-w"). That worked well enough.

However every so often you would have to create a new account. To do that the chmod had to be undone, but only for a minute or two. In that minute or two typically 2 or 3 spam accounts where created, and maybe 20 wiki pages spammed or defaced.

In short: defending the site had no effect whatsoever for us. The bots were always probing, every few seconds, and it never stopped even years after the site was totally locked down.

PS: SPAM wise the bots were just an annoyance. But every page hit runs moinmoin's Python code, and it's not the fastest thing. We were running on low end VPS's that took a dim view of anybody using too much CPU. Our VM regularly got shut down because of those bloody bots.

Perhaps you could hand-tailor a fail2ban rule that automatically jailed anyone who even _attempted_ to log in, for a whole month, and leave that running for a week prior to the chmod, and disable it during the editing process?

> however the wiki itself (moinmoin)

Right, so I think the key think spammers need to make spam "pay" is to make it scalable: only 0.1% of people will click on your link, but if you can generate a million new "views" per month, then that's 1000 clicks per month. So the key thing for a spammer is to be able to automate as much as possible. One key element in preventing spam, therefore, is to try to make it so that a human has to be in the loop somewhere.

In my case, I wrote the website; I suspect any login bot would have to be customized to my specific website to be effective. I'm sure it doesn't take much time, but you'd still have to start an automatic bot specifically for my site. Why would you do that if you saw that all of your attempts to get spam up had failed?

In your case, you're using a standard tool. Someone's already written a bot that can log into any moinmoin instance; and almost certainly someone's written tools to scan all websites for new moinmoin instances and try to create accounts. It's probably unlikely anyone has specifically thought about your site at all; they'd probably have to write special code to remove it from their automatic scans. Same thing as before -- even if their bot fails on 99% of moinmoin sites, that 1% makes it worth keeping it going.

> PS: SPAM wise the bots were just an annoyance. But every page hit runs moinmoin's Python code and it's not the fastest thing.

Golang FTW. :-D

It's sad and also not always true. I think most people were pleasantly surprised by how the "get into a stranger's car" and "go stay with a stranger for a few days" models worked out for AirBNB and Uber.

Obviously there have been some issues and obviously they have processes in place to minimize and respond to abuse of their platforms, but overall both platforms rely on "trust by default" which is interesting and leads me to think that your comment might lean too far towards pessimism.

AirBnB and Uber are a bit different though, because being online gives you a sense of immunity and relieves a lot of inhibition that you'd have in real life. To some degree, we all have a different sense of morality online and in real life (like most people would never steal even a fruit in the supermarket but see no problem pirating movies and software) and some push it very far.

To say nothing of the fact that you have a credit card and therefore real world identity (presumably) attached to that card and that any serious in-person griefing can easily end up with you tossed in a jail cell.

AirBnB currently requires pictures of your state-issued ID to create an account in addition to the credit card.

"like most people would never steal even a fruit in the supermarket but see no problem pirating movies and software"

A stolen fruit is gone from the supermarket and cannot be sold to a paying customer.

A downloaded movie remains the same.

There is only a loss, if the person who pirated it, would have bought it otherwise.

So no, it is not the same.

Ok, wrong example.

Let's say, most people would never dare to make racist insults to anyone on the streets but many do so copiously online.

I like Mike Tyson's take on it - “Social media made y'all way too comfortable with disrespecting people and not getting punched in the face for it,”

The one that always surprises me is NextDoor. Given the much greater chance that you will literally run into someone on NextDoor in real life, and the fact that NextDoor accounts show your actual identity, you would expect discussion to be more civil there than on other social media.

But, holy cow, every time I go on there, it is a hot mess of angry people screaming at each other. I have to assume those people just don't go outside at all?

I mean, there are plenty of people who would say bad things about others, even if they know it'll get passed on to them, than go up to them and do so directly. Plus you have the usual problems that come with NIMBYs+HOAs all in one app.

Whether it's through moderation, or the real name policy, NextDoor has avoided the worst of 4chat. There are no page long posts of a single repeated epithet, and there's no child pornography, and the only drug talk is in reference to the local homeless problem (though that may also be related to the age range of users on NextDoor). It turns out that adding real names only slightly raises the threshold on what some people will say.

IMO it is NextDoor's hyperlocality that makes it such a hot, angry mess. On Facebook, your cousin that's 5 states away's friend that's straight up posting swastikas unironically? The chances of meeting them, ever, is fairly close to zero if you don't want to, so it's far easier to walk away. But on NextDoor, it's because that the people are in some cases, literally next door to you, that the conversation is almost immediately emotionally threatening. It turns out your neighbors aren't like you, but not in a good way, and it challenges your sense of belonging to the local community. It turns out George down the street who's a sweet old man who's lived there for 20 years and has a pretty dog, is a neo-Nazi. Not something that would come up while exchanging pleasantries about the mail being late today, but thanks to the Internet, his eccentricities are on full display. Good fences make good neighbors.

It's easy to dismiss Nextdoor as a hot angry mess, but there are very sweet things too. My neighbor just recovered their cat, someone is collecting clothes to bring to a women's shelter, but the hot angry mess is very shouty, and impossible to tune out.

Oh, I very much agree to your general point, I just happen to be a free information believer, so I was triggered by your example ...

Pirating movies/software is not in the same moral category of stealing physical goods because it’s non-rivalrous. Yanking something off of Usenet is akin to watching a baseball game or a movie at a drive-in from over the fence.

> Yanking something off of Usenet is akin to watching a baseball game or a movie at a drive-in from over the fence.

This is a excellent analogy that I don't recall having encountered before; thank you.

> I think most people were pleasantly surprised by how the "get into a stranger's car" and "go stay with a stranger for a few days" models worked out for AirBNB and Uber.

Most Americans for the former, maybe. "Get into a stranger's car" is literally an everyday routine in other places in the world, and has been for a long time.

The measure law is also known as Goodhart's law: https://en.wikipedia.org/wiki/Goodhart%27s_law

> It's sad, but that's the state of affairs on the web as we have it today.

Hasn't this been the meta since ever since?

Imagining you'd know all about that with camarades. It was my first exposure to web streaming as a teen way back in the day. Initially, it didn't strike me as a porn platform, but it didn't take long before it felt that way.

At the beginning there were exponentially fewer people and syndicates, so there was much less total volume of abuse.

Virtually all the ccurrent modes of online abuse were present in at least some form on Usenet and dialup networks by the late 1980s / early 1990s.

Content piracy to a much lesser degree as bitrates were so slow and expensive (though texts were fair game), but stalking, harassment, propaganda, bigotry, fraud, spam, etc., yes.

Keyword search of worldwide public content and metadata was not readily available, so the ability to stalk and harass both people and topics was vastly reduced compared to now.

(FidoNet existed, but it was not easy to perform a full-content search of it, compared to eg. mail-archive.com now.)

Not trivial, but possible. Kibo and Serdar Argic managed, at least.

Kibo only grepped the newsgroups he subscribed to, and didn’t have a full UseNet pipe running all groups through grep (unless I misremember the history there).

The problem isn’t full-text search of a forum you’ve already signed up for.

Re: Kibo groups coverage:

Legend has it that if you include "Kibo" in your post to any of Usenet's more than 5,000 discussion groups, the all-seeing Kibo will respond to you.... [James "Kibo"] Parry spends about two hours per day as Kibo - tending a filter that collects any mention of "kibo" in any newsgroup.

https://web.archive.org/web/19961128094925/https://www.wired...

NetMeeting was filled with all sorts of Perverts in the late 90s.

NetMeeting... there’s a word I haven’t seen in decades.

Tell us more!

132-column lineprinter nudes were in circulation well before the web. (and "typewriter art" predates even the IBM 1403)

At one point there was a movie shot at SAIL with rather open minded approaches to computer-human interaction. In the early 1970's (as in the twenties?), "hangups" were for inhibited squares. (If they'd been running unix instead of waits, they'd probably have invoked nohup(1) for the shoot.)

Going back yet another century prior, we find the Victorian equivalent to OnlyFans: https://news.ycombinator.com/item?id=23791112

Yes, that is one of the reasons. Also, helping a number of other platforms to combat all kinds of forms of abuse. Unfortunately, as you correctly surmise, I have some relevant experience which some people find useful to tap into.

Camarades went 'downhill' about 6 months in when it went mainstream and more and more people that had other ideas of where we should go with it joined. Then they brought their buddies and it was 'game over', we had to decide what to do about it. This roughly coincided with the ad market collapse and drove the decision to put the porn behind a paywall to finance the rest of it. Which worked well for more than a decade.

I used to work with a person who used these principles to say no to building anything at all.

There was an article on HN about Disney internet apps. Disney management consider "safe for children" to be a critical part of their brand. Their stance was very clear: no communication between users under any circumstances, ever.

Could Disney take over Facebook? Might be good for the world /s

They actually thought about buying Twitter: https://www.businessinsider.com/disney-ceo-bob-iger-abandons...

While they wouldn't have used the Disney brand for that ;)

That was sort of the idea of their old Touchstone Pictures brand. It let them do movies that weren't on brand for Disney.

Miramax Films, too. Except that was via Disney acquisition in 1993 and not homegrown like Touchstone was.

It's always fun to say that technically Pulp Fiction (released 1994, post-acquisition) is a "Disney" film.

Probably this: http://habitatchronicles.com/2007/03/the-untold-history-of-t...

On HN last year: https://news.ycombinator.com/item?id=20624074

A very good read!

Probably wise. I've seen online interactions in some games be limited to ready-made callouts or phrases you can assemble from pre-made words / sentences (like From Software games).

Of course, points for creativity if they can still make lewd comments like "time for thrust attacks but whole" near a corpse bent over a banister.

Minor quibble, but I think it's "hole" not "whole", so it's even closer. As in "Watch out for " "hole" "!".

The funny thing is that the game has incredibly sensitive username filter as well. I've fought countless players with names like "The #### K###ht" instead of "The ugly knight".

Nintendo is similar. Communicating with other players online is nearly impossible, even if you're friends.

I can't quite recollect how, but on WiiU you could video chat with friends

Doesn't Switch have ingame audio-chat at all?

Separate phone app.

I assume this is why PictoChat never came back :(

It did, as Swapnote on the 3ds. And originally you could send Swapnotes over the internet to your friends. But then it was common to exchange friend codes on message boards with lewds being traded (especially of the apps' mascot, Nikki) so Nintendo axed her and the ability to use anything but local note exchange in response.

Ugh.

I wonder how this policy has changed over the years. Disney used to have a virtual world called VMK that did allow for communication between users. This being Disney, there was a (highly restrictive) whitelist of words that could be used; this being the internet, users found lots of clever word combinations to evade the filters.

"Sufficiently Powerful Optimization Of Any Known Target Destroys All Value" https://thezvi.wordpress.com/2019/12/31/does-big-business-ha...

yes and i'm always looking at HN Entries which show the next file upload etc. and think 'puh holy shit they are fearless' because thats why i'm not building something like this.

I would create a company to have legal seperation for my private assets, i would use a ton of mechnism upfront to make sure i do my best to not support childporn and stuff and when i have analyzed how much work the proper way is, i will just stop thinking about it :)

Are there recommended services where I can point to...say...an s3 object and validate it doesnt fall into some /all of the above categories? Seems like a good business and also something i'd pay for to remove this risk.

Yes, look at Amazon Rekognition. I use it (very successfully) for a website that used to encounter similar issues with file upload abuse.

In my case, image uploads aren't mandatory for users, but they are very helpful in identifying the spammers (for some reason, the spammers almost always try to upload images that get filtered, so it makes it easier to spot them). That, combined with an IP check (getipintel.net) has almost completely eliminated the spam issues on my site.

> IP check (getipintel.net)

I didn't know about this service and it sounds really cool. Kind of sucks that VPN = malicious since I use Mullvad for legitimate browsing purposes. That said, I understand that VPNs are often used a medium for illicit activities.

Microsoft does some of this for the worst offenders, see 'PhotoDNA'. For once something that they did that has no downsides that I can determine.

Was going to say generally the same thing. Back in the day it was anonymous FTP, if you enabled it in your server you didn't have to wait long and suddenly all the things would start showing up.

It reminded me a little bit of the guy that started a "buy a gift card with bitcoin" site, not realizing he was effectively creating a way to convert bitcoin into actual cash while avoiding an exchange. It was wildly successful until he realized that it was really really hard to get large quantities of bitcoin rapidly converted into cash.

Anyone implementing a POST handler should think “what if this is child porn”

Good one, yes, the POST references should all come with pointers to abuse potential.

Would charging for signup, even $1.00, eliminate essentially all of the abuse?

That would depend on the lifetime revenue per user gained spamming your system. There definitely is some sufficiently high signup fee, but it’s different for every spammer. (And for the child pornographers, there may be no fee you could charge to stop them.)

Along with 99.9% of your legitimate users in most cases.

It is a very coarse filter. Your ratio will go up but there will be plenty of people who feel that since they paid you they now really get to do as they please.

ToS / EULA specifying cancel of account and NO REFUND. Usually that keeps them at bay.

Your EULA doesn't help when the credit card reverses the charges.

And when you can't prove that it was the real person who paid for it, you're going to lose that one.

I believe it works for Metafilter, but you also run into problems with Credit Card abuse, fraudsters who create accounts and test which cards are not yet reported as having failed etc.

It is a nightmare.

The bad actors use stolen credit card numbers so then you have to deal with chargebacks and fraud issues.

Don’t forget malware too - a problem even big companies have to grapple with, unfortunately [1].

[1] https://www.zdnet.com/article/mozilla-suspends-firefox-send-...

I predicted that about send the day it was launched. It helped that I had done a very close review of a large competitor in that space so seeing 'send' open up gave a lot of bad people a new playground with predictable results.

github is fairly open and it's not used for porn, yet.

and from that one can infer....

As far as you know...

It's definitely used for pirated stuff.

Put any technical book title into google with site:github.com in it and click the first PDF that shows up.

There's also a font piracy site that simply trawls public Github repos for the font's filename

Ah, so that’s why the site’s so trigger-happy with the excessive-activity bans that it’s nearly unusable unless you’re logged in.

And cloud tokens and other credentials. Those get scraped super fast.

Probably because you have to be somewhat familiar with git to be able to upload stuff to it, there is a barrier of entry.

Not really. Click on Add File -> Upload files on top of the repository view then "drag files [there] to add them to your repository".

"Your repository" being the problem. It's not anonymous, and you don't necessarily want coworkers + world to know about your personal interpretation of Rule 34.

It's just linked to an email. If it's anonymous enough for forum users, it's anonymous enough for people to use it.

The logic of the parent comment would suggest that this hints at a fairly large amount of work going on behind the scenes to keep it that way.

For fun, see https://github.com/topics/pornography

I would guess part of it is that you can't control the mime type of uploaded files.

or compressed files with passwords

There was a project called "Dick ML" though, which I believe was used by some idiots to spam Zed Shaw with invitations.

Well, there are a lots of porn related projects. One of them being Deepfake

I'm sure their community team has to deal with a LOT of shit on a daily basis, even after automated checks and filtering.

I've stumbled upon some, FWIW–I think I was searching for file carvers or something like that.

This is so true. We built Jumpshare (https://jumpshare.com) for file sharing and visual communication, so you can imagine the abuse we got. We had to spend a considerable time and resources to fight this abuse and adds checks and balances which slowed down our product roadmap.

Did you think of the risks as a legal issue, reputational issue, moral issue, or some combination of the three?

Does something like email verification resolve this in some small way.

I ask because we are looking at extending our platform to include enterprise file sharing and rendering for niche file types, bit don't want to expose ourselves to the extra work you've described.

- platform to send money / gateway -> fraud

And money laundering. It even helps the perpetrators when the operator finally decides enough is enough and nukes the platform.

Haven’t heard the word wares in 20 years!

*warez

What is the best way to moderate each such categories?

Maybe this could somehow be remedied by a Blockchain technology.

Reply on news.ycombinator.com