How To Stop Forum Spam
swapped.tumblr.com
swapped.tumblr.com
We were swamped recently with spam when our page rank recently hit 6 (I guess spammers target sites with higher PR). For this we had to implement a block on users with under n posts being able to post URLS. That basically eradicated that wave of spam.
It's amazing how smart and adaptive some of them are though. It only took them 12 hours to realise if they post anything then edit it they can edit in links. We fixed that one promptly and they haven't been back since.
So for that reason your suggestion probably would only be temporary. I'd also be concerned about a lot of false positives.
I also believe a lot of spam is human. If you have people willing to work for a few dollars a day spamming websites that automated processes have difficulty reaching, it could well be a cost effective method for spammers to reach their targets.
We have really good search engine rankings and if you post a topic on our forum you'll be available in all the search engine results within minutes. The funniest thing I've ever experienced was discovering when a spammer will submit a post about a live stream for a football match 15 minutes before kick off that we'll see ~3,000 people browsing the forum all from google because they searched "watch barcelona vs real madrid" and our ranking is being abused by spam bots.
Prior to realising that this happened I'd always viewed spam on forums as a sort of "hope people click the links" thing but after discovering just how much traffic people like those mentioned in my example can drive... these spammers must be making a lot of money. The forums are used as their quick way to get into search results for a term they know is about to blow up, I think it's pretty clever and obvious in hindsight.
However, it is our opinion that the traffic these links generate is fake (and we have some evidence for it as well). I think it's aim is to trick Google into thinking it's a good quality result, or alternatively the links often are very heavy in adverts so the fake traffic routed via Google, your website and finally the spammers website could be used to generate lots of fake ad clicks in a semi realistic way. That's our conspiracy theory anyway, I very rarely see real world examples of content on the web being ranked so well so quickly and driving so much traffic in such a short space of time.
As a small unscientific test when a seemingly popular spam sports event thread was posted, I deleted it and created a duplicate under my account. It didn't get any traffic at all. This in some way helps validate my suspicions that the traffic being driven is not a result of Google's process, but is artificial.
I was going to write a blog post about it but never got round to it!
If it's a small forum you probably don't need any actual moderation features (stuff you do need you can do through a scaffolding interface).
Bots seem to be attracted to phpBBs like flies to shit so not being fingerprinted is a huge bonus.
Also helps to do weird stuff, like instead of having a <input type="submit" /> for a form submit use an IMG tag with an onclick that reads data from the form fields in the most roundabout way possible.
Also make the URL of each thread change when there is a post so it makes it harder for the bots to tell if their posts are working or not. Better yet don't display the results on the post on the site itself for 30 seconds or so but use JS to make them appear to the submitting user.
Some of those tricks can of course hurt usability and accessability though but I managed to reduce spam to 0 on one forum.
If you want to stop spam on a small forum you don't need to break usability because nobody is going to spend time looking at how you're doing things to get around your spam prevention.
We originally used phpBB and applied various anti-spam plugins, including some modifications that were made manually to the php code (including honeypot fields that were hidden with CSS) but the bots kept spamming it regardless.
I guess there is a sense of "we know this is a phpBB therefor we must be able to spam it so keep trying" vs "I don't know what this is","can I POST this form and see instant results?","no?","give up then".
If it had been a larger forum then I'm sure these tactics would not have been effective regardless since we would have suddenly become worthy of having a custom spam bot written just for us.
Those tactics are somewhat extreme, in many cases just giving form fields weird names has good results. Of course there is the problem of "what is bad for spam bots can be bad for screen readers".
Did the number of non-spam posts also get to 0?
</snark> Sorry, had to.
Nothing can be done about targeted attacks I think, besides not being worth the effort, or having moderators and constantly adapting. I wonder how much randomness you can introduce into the posting process, though, before it's no longer worth the effort. Add a csrf value to the form action? Rearrange the order of the fields? Add a captcha based on an average of the edit distance of an IP's previous posts, with a random threshold?
Doubt re-ordering would make much difference either since they probably work on field names and tags.
CAPTCHA can work but you need a difficult one which will also be difficult for humans. Plus spammers can outsource captcha breaking to humans via injecting them into free porn/torrent/movie sites (solve the captcha to get the content) or just by paying someone in india a few cents.
Then again, that captcha might end up being too complicated for regular users to be willing to fill out as well.
For which I find this one of the best providers of up to date lists... oh, and they have an API I use at registration time: http://www.stopforumspam.com/
The last 1%, I let my other users flag, and then I ban them and add their data to the site above.
This isn't a big problem anymore, it just sounds like the author hasn't integrated his forum with the site above.
Theirs is an effective approach, and it works great for dedicated forum sites. But I think it's an overkill for simpler setups for two reasons. First, it creates an obvious dependency on an external service. Second, if I want to allow unregistered posting, it leaves me only with an IP address as a data point and here I wouldn't bet that their by-IP detection is too accurate.
It's a simple, self-contained, virtually maintenance-free way to detect humans trying to post. I don't argue it's a superior to other methods, but it is simple and it helps simplifying the user experience of real visitors.
The cost of a false positive is great... you may lose a customer.
By adding an email field to your form you'd get to use the blacklist and avoid false positives and the risk of offending an already aggravated customer.
It's just that in my own setups I haven't come across for the need to blacklist people.
PS. If they searched the issue, they were presumably having it with my software, so it's quite likely they were on the site before and I can recognize them. That's not to say that direct hits into an appropriate threads from https://google aren't possible. But they aren't that common either.
It misses about 50% of the spammers, there are too many fresh IPs for it to know them all.
It also does "false positives" in that spammers are now using email addresses from legit users, so an email match alone is no longer good enough (if you don't do email verification).
The cookie method described by the post also won't work in the realworld - many times people drop into the very article they are looking for directly from a search engine, read it on the same page, and then leave a comment, so no crawl history.
However, we set up a StopForumSpam plug-in, a time-based plug-in (if you complete the registration page in < X seconds, you're probably a bot), and a custom questions plug-in, and we haven't seen any spam since. If they start figuring out to break through those walls, we'll try a few more... (On the other hand, it' s a small enough forum that I don't know if we're missing false positives.)
Yes, it's a cat and mouse game, but it's one you can generally stay on top of with minimal configuration once you understand what the spammers want and how to use your platform. And I have to admit there's some thrill involved at successfully thwarting their plans...
So I guess there's probably someone going over unknown questions regularly and adding new answers.
I've taken a few steps which has completely rid me of spam (for now):
- IP blacklisting: an obvious one but well worth the few minutes it takes to setup, most of our spam was coming from China.
- Link blocking: our comments don't really require links to work as the comments are generally short and to the point and the blog is not used by terribly tech savvy users.
- Hidden checkbox: add a hidden checkbox to the form. If it comes through as checked you know a human didn't submit the form.
Analysing a visitors progression through the site is a neat idea though - if spam becomes an issue again then I may use this approach (since I'm already gathering this data for custom analytics).
If the filter is unsure, the post can be referred to an administrator for moderation (and will subsequently be added to either the spam or ham corpus, training the filter for similar posts).
In the case of false negatives (spam gets through), a discrete "report spam" button will allow a moderator to add it to the spam corpus (again training the filter against similar occurences).
It might even be possible to use the filter score to reduce "report spam" abuse, i.e. if the filter is fairly certain it's ham, require a larger number of users to report it as spam before bothering an admin with it.
* http://news.ycombinator.com/item?id=4646710 (How to beat comment spam)