A painful tale of SEO, spam and Google's role in all | Hacker News Reader

A painful tale of SEO, spam and Google's role in all

scriptogr.am

203 pointsslaven104 comments

My god, the quote from that "white hat" SEO guy just defies belief:

> There will certainly be webmasters out there who will strip you down to the bone asking for money in exchange of link removals. These are the most soulless snake oil salesmen on earth

To say that about webmasters, already victims of years of abusive SEO spamming, when they then refuse to help an abusive site clean up its own mess for free .. I have no words. Could anyone possibly be more of a self-interested, myopic, egocentric prick?

My rock-bottom opinion of pretty much anyone who has anything to do with active SEO is re-confirmed for the thousandth time.

it's easy to judge

Yes. Yes, it is.

Don't hate on all SEOs just because of the overall clown-ness of the space :( there are some good guys!

Sounds like their job descriptions requires they not be "good guys"

Googles heuristic would work be much more effective if people didn't screw with it.

Not true, there's some very cool stuff you can do that is entirely within Google's guidelines AND gives your site a large advantage over other sites that aren't using them. See http://www.quicksprout.com/2013/01/24/the-advanced-guide-to-...

I'm pretty sure there's no way I can convince you, or any SEO, that you're merely proving GhotiFish's point here.

The ethical disconnect is painful.

Short of subverting human nature of everyone on the Internet it will never be possible to have a leading search engine that people aren't constantly trying to game.

The best we can hope for is gaming behaviors to roughly align with desirable search results.

I don't see an ethical disconnect. Google is the world's largest scraper, making almost all of their revenue by serving ads next to the content of everyone else but them, yet there is an "ethical disconnect" by voluntarily choosing to still abide by their guidelines?

> Google is the world's largest scraper, making almost all of their > revenue by serving ads next to the content of everyone else but them,

That's oversimplifying the service and value Google offers. This is the typical passive-aggressive SEO stance. Deriding Google for the service it offers, but at the same time craving it's attention. Because you know, Google gives you a steady stream of customers.

People use Google because it helps get them to the websites that offer the content that people are looking for. It's (relatively) good at matching seekers with the information they seek. That's what brings people back to Google over and over.

And so, the page they deliver has value to people.

Yes, they scrape content from everyone else. Including themselves (e.g. a search for "Google webmaster guidelines" returns support.google.com).

They also add to that their special ingredients - algorithms that dissect pages, classifying, inferring meaning, deriving synonyms, inferring intent. Weighing, scoring, evaluating every page out there. So when a customer comes a-querying, it uses those calculated metrics to decide which 10 pages get to be mentioned on a search results page, and what snippet/extract of those 10 pages to show.

The value Google adds is the selection of which pages to link to based on the query and what it knows/infers about the pages themselves. That is no trivial thing.

It's nowhere near passive aggressive, just a statement of what Google does. Google does give a steady stream of traffic, but it's my goal to diversify traffic streams as much as possible at the same time - Google's becoming less important as a direct source of traffic from organic SERPs.

I'm not saying they don't add value, I'm saying that their entire business model is built on the back of the world's content.

I'm not sure why HN hates SEO so much, since Google doesn't...they hate SPAM, not SEO.

Also: http://support.google.com/webmasters/bin/answer.py?hl=en&...

You seem to be under the impression that Google is against SEO.

http://www.youtube.com/watch?v=BS75vhGO-kk

Consider reevaluating your views.

SEO is a misnomer. They don't aid search engines, and they certainly don't optimise them in any way.

Goggle doesn't need your optimisation "help". Just create compelling content. That means you need to be open, informative, interesting and innovative.

Google doesn't agree with you.

"Many SEOs and other agencies and consultants provide useful services for website owners"

http://support.google.com/webmasters/bin/answer.py?hl=en&...

I don't know if some SEO actually do that, but following Google's guidelines for webmasters, using Google's webmaster tools instead of building an opaque site with Flash or full Javascript certainly helps Google (otherwise they wouldn't publish these guidelines to start with).

"Google doesn't need your optimisation "help". Just create compelling content."

Sadly, that's still not entirely the case.

It's more true than it used to be that content can carry a site on its own. But I still see dozens of quality websites and writers getting far less readers than they should, purely because they don't understand the basics of formatting pages for search engines.

A little bit of optimisation FOR search engines on your own pages can do wonders for your traffic, readers and viability as a business.

(No, I don't represent an SEO agency.)

> SEO is a misnomer.

As much as a misnomer as cat food. Try to think of it as Search Engine _Placement_ optimisation.

So what if the site requesting the link take downs was a victim of someone else spamming links out pointing at their site? It's not a white/black situation, there are many ways those links could come about and there are legitimate SEOs out there requesting they be taken down.

This is what I was thinking. Not every site that has had its url spammed across forums and in comment-spam did it themselves. The other side of black hat SEO is destroying competitors by triggering Google penalties against them.

I used to run a forum but I shut it down because of spam. It was a shame. It was well ranked on Google and I think this is why spam was such a problem. It was a nice before it was overwhelmed. I did moderator approved sign up but I was getting several hundred sign ups a day. It wasn't possible to determine who was human and who wasn't.

Captchas and security questions decreased spam for a couple of days but it would soon come up again.

I added 10 posts before you can post a link.. but then the forum was filled with non-link spam posts. Stuff like "I agree." "I appreciate your post, it was very helpful."

I added moderator approved first posts but it meant there was a queue of a 250+ posts a day. People stuck in this queue, especially on weekends would often give up on the site.

In the end people started losing interest as moderators were struggling with the battle against spam..

During this time I actually got a few requests for links to be taken down. I refused them all. It wasn't out of anger. It simply wasn't worth my time. I would need to look at each url requested. Determine if it was spam or if it added value to the site. Then choose whether to delete it or not. Each request would take 5-10 minutes to process. Perhaps I should have started charging. I could then have given beer money to the brave moderators who for 18 months tirelessly swept for the forum.

I see nothing wrong with sites charging say $5-10 to permanently remove a link from a website and make it impossible for it to be posted again.

"Not every site that has had its url spammed across forums and in comment-spam did it themselves. The other side of black hat SEO is destroying competitors by triggering Google penalties against them."

That tactic (of spammy linking to your competitors) only became a viable one after Panda/Penguin. I doubt it was practiced by SEO agencies before those algorithm changes. So links added before that change are unlikely to be negative SEO.

It's also just an incorrect use of the term "snake oil salesman."

Well it's written by an SEO consultant, what's the money he has been called a "Snake oil salesman" at some point in the past?

I can understand someone getting frustrated. This forum may have great information, but it is also causing a lot of harm, possibly leading people to dangerous sites. Is it the owners responsibility to do something about that? I think so.

Just as you have lots of machines that have been compromised and are part of a botnet.

A lot of harm caused by the agencies, requesting the service of webmasters to clean up the agencies mess in the first place. That's the issue here.

> Is it the owners responsibility to do something about that?

Reasonably, yes. But, when the volume of spam becomes too much to handle, what can a poor webmaster do?

I quite like that Google have altered their algorithm to penalise this SEO behaviour. Because it now puts the headache firmly on the website that paid SEO people to originally spam those links.

And I applaud and support the webmaster who requested payment for removing the link. The link was placed by a link spammer on behalf of their client, it's only right that that client, after benefitting from that tactic - risk free - now should pay for it to be removed.

Granted, there are negative SEO connotations to this, but factors like link-age can be used to spot who was link spamming before Penguin/Panda, and who is link spamming after.

I would be thoroughly impressed with an SEO agency who was practicing this now negative SEO tactic before links from spammy neighbourhoods were risk free. And thus benefitting by giving their competition a headache post Penguin and Panda.

I have no sympathy for companies that hired SEO agencies who link-spammed their website. They shouldn't have done it (ethic-deprived), and if they didn't know about it they should have known about it (abrogation of responsibility). If there was deceit involved, then the company can take it up with the SEO agency that created that deceit.

Yes, clean SEOers are doing a roaring trade trying to clean up these websites, but they must realise their client caused the initial damage by hiring bad SEOers, and they are not entitled to free link removals.

    "To say that about webmasters, already victims of years of abusive SEO spamming"

I suspect this comment was aimed at webmasters not who were the victims of spam, but those that profited from it. It's those who run directory sites and article farms that would accept cash to add your link, and now that people want the links removed, they're double dipping. And these people are indeed scummy.

If they paid to get listed there in the first place (paid link for the purpose of better rankings), then it's fair game to pay to remove the ranking disadvantage now. I call it an ethics tax.

Paid links were frowned on but not generally penalised. So sites took the risk they wouldn't get caught or penalised. Now they got penalised. So if it was okay to pay for a linkback, equally it should be okay to pay for the removal of a linkback.

Perhaps. I don't have much of an opinion one way or the other on if it's moral to charge for something like that. But the Disavow Link Tool exists for a reason, and webmasters can use it to manually remove those bad links. Though they are encouraged to make an honest effort to contact webmasters first.

That google 'suggestion' is directly leading to the deluge of crap in the OP's inbox.

This is absolutely correct. This guy's forum is the .01% case.

The OP should simply delete these messages and move on. But I guess doing that doesn't drive traffic from HN.

What Google has said is that they just want evidence a reformed spammy site has /attempted/ to remove offending links manually.

See Slide 16: http://www.mattcutts.com/blog/pubcon-2012-slides/

Note how Google wants to see evidence in the Link Disavow file that sites have TRIED to remove links manually:

Here's a sample of a valid file:

     #example.com removed most links, but missed these
     http://spam.example.com/stuff/comments.html
     http://spam.example.com/stuff/paid-links.html
     # Contacted owner of shadyseo.com on 7/1/2012 to
     # ask for link removal but got no response
     domain:shadyseo.com

So this whole post seems like sort of a hissy fit from the edge case of a site that a) failed to protect itself by linking out with followed links from untrusted posters and then got spammed to hell, and then b) gets offended when he reads a comment directed at sites who are trying to profit by removing links when all he really needs to do is hit delete when he gets emailed with link removal requests.

I agree with the comment that this is fair play or an "ethics tax" on people who bought links... f-em I say, since they cheated to begin with and hurt sites who played by the rules and whose content should have enjoyed better rankings and traffic than it did due to the spammers. So whatever hell they're in now is fine by me.

But I also think the OP brought his problems on himself. I ran a huge UGC site for 10 years before I sold it and sorry to say, but if you want to play forum master, you're responsible for defending your site from whatever nastiness is out there. We expect UGC sites to have porn filters, profanity filters, troll filters, and a DMCA takedown procedure, do I don't think it's too much to think that smart webmasters will nofollow links (which is why this site got on the XRumer radar in the first place).

Blaming google for seo spam is not productive - spammers are legion and in constant battle to game google's ranking system, whatever it currently is. This won't change as long as google is used to search for things.

There is a solution to his particular problem: better forum software.

I don't want to trivialize the problem of writing spam-resistant forum software but it's not such an insurmontable problem (this forum being a proof positive for that).

For the reference: I've been running a fairly popular forum (http://forums.fofou.org/sumatrapdf) for several years, using forum software that I wrote.

I don't even require the user to log in, I don't require moderation for posts and yet I had zero automatic spam (I attribute this to my unusual captcha http://blog.kowalczyk.info/article/16fw/Best-captcha-is-exot...).

I occasionally get human spam i.e. someone writes a post with the only purpose of linking to some other website. I just hellban them after I see the post in my rss reader.

And I didn't even write any special anti-spam code (other than hell-banning), because I'm lazy. I can easily come up with simple ideas e.g. putting all posts that contain links in moderation queue.

To reiterate: his problems were caused by a crappy forum software that didn't do much to protect from seo spam.

I don't really know how current best off-the-shelf forums fare in this respect.

I would rather not spend my time maintaining my own forum software so I have high hopes for http://www.discourse.org. I'm sure StackOverflow had plenty of spam problems so discourse people should understand the problem.

I'd attribute it to your custom software. Break phpBB and you can infect thousands of sites. Break your custom software you get to infect 1 site.

I ran a forum using SimpleMachines. Turned on all the spam fighting stuff. Didn't matter. Bots or Mechanical Turks made 10-30 new accounts a day. I tried moderating the accounts but who wants to spend time every day to delete the fefRFGR34@foobar.com account.

I gave up and shut it down.

To be fair, though, SimpleMachines is a very atrocious piece of software (not to mention their equally atrocious licensing, although it seems they're in the process of changing that[0]...), so it's no wonder.

I could say similar things about phpBB, but that's quite a dead horse to beat (hey, at least they use a proper license).

[0] http://www.simplemachines.org/about/opensource.php

If you see Mechanical Turk used for spam like that, you could report it to Amazon as abuse and get the account(s) paying for it shut down.

I also wonder if Amazon has a content filter, to block URL patterns in Turk tasks.

As for bots, IP banning goes a long way.

Not if they really want your site. A place I used to work for was massively targeted by them, and literally the head IT guy gave up banning because they just rented new servers and continued the same stuff.

Essentially, like Jesus and the poor, spam we will always have with us (as long as there's an incentive).

I use modsecurity on my Drupal forum.

At the moment it stops pretty much everything that gets thrown at it, and the rest is stopped with Bad Behaviour and Mollom.

Legit posts still turn up no worries. All the crap just falls away.

I also have fail2ban rules in, so those that trigger modsecurity too many times get added and iptabled. Helps cut down on the repeat attempts.

I don't think it's hard to block spam. I do agree it's time consuming though, keeping everything up to date.

This really only works if you have a very small site. No offense, but Alexa's rank for fofou.org is over a million. No spammer is going to spend time writing custom logic to break your captcha. I bet you don't even need a captcha -- just having custom forum software is enough to stop all automated spam.

If your site was in the top 1,000, you'd have a different experience since the spammers would be adapting to every one of your exotic captchas no matter how many times you change it.

No offense taken. 2 comments, though:

1. fofou.org hosts the forum for my SumatraPDF site, which is 33,279 in Alexa. I think that's enough of a target, more than most forums out there.

2. I don't think it's productive to bring top 1000 websites into this particular discussion. OP's forum wasn't that popular and most forums, by definition, won't be either.

Once you're there you're playing a completely different game and at that level you should have resources to win it. I don't see spam on Wikipedia, StackExchange, Quora, HN etc. When there's will, there's a way to fight spam.

There are Human captcha breakers that you can buy for less than $10 for 1000+ captcha breaking. There are software / plugins that provide you direct access to these people. Once the plugin detects a captcha in a form, it just passes it on to these people.

Your software solutions can do only so much. Wikipedia, StackExchange etc has Human monitoring (volunteers who report spam) in addition to automated stuff.

Its tough keeping up with spam.

I think you have the advantage of hosting the forum on the low ranking site. The forum would be more tasty target if it was on (higher ranking) SumatraPDF site.

P.S. Thank you for SumatraPDF!

Spammers target HN.

If you have showdead turned on you see a variety of obvious spam (handbags and shoes) and non-obvious spam.

And there have been some aggressive voting rings.

I'm responsible for helping to kill spam on Quora, so I'm happy to hear that you've had a good experience. In reality, we get a lot of spam, but we also allow anyone to edit most content so this crowdsourcing approach helps a lot. I'm sure the same is true for Wikipedia/StackExchange.

No matter how large your site, if you give people you trust the tools to stomp out spam/bad content, they will use it and you eventually "win."

I think better software would help but I also don't think Discourse is that software. I'm sure they understand spam but they don't seem to understand how humans communicate.

He's not blaming google for the original blog spam, he's blaming google for the deluge of emails requesting link removal. Since they are directly encouraging these emails by holding out the hope of rehabilitation they are responsible.

Short of evidence that the blog spam was malicious and unaffiliated with the site, there should be no rehabilitation. If the site has valuable content the owner can move it to a new domain and try again. This time without being a scumbag.

One way to combat this problem is to deny new users the ability to post links by replacing the link text with [removed]. HN has a karma system with thresholds that must be passed in order to get additional privileges (like down-voting). If you have a site that assigns karma to users (even if it's a secret number) then you can set a threshold for allowing links to pass through. It's not perfect but it's better than letting brand new accounts post links, IMO.

Another solution is to hide all links when a page is viewed by a user who isn't logged in.

> Another solution is to hide all links when a page is viewed by a user who isn't logged in.

No, this is almost as bad as what medium.com does with all its links (rewriting them to point inwards towards redirects for tracking purposes).

A web link should be respected.

> (rewriting them to point inwards towards redirects for tracking purposes)

I like this idea a lot. It has more than one use. Tracking is one but eliminating the "PR bleed" is another and it should deter spammers who are looking for link weight.

I understand the sentiment you're conveying but a link isn't sacred. If you allow users to post content to your site then you should be taking measures to ensure that new users aren't spamming. I don't think the "go in and clean it up later" method cuts it.

> I understand the sentiment you're conveying but a link isn't sacred.

It damn well should be. The link is the basis of the www.

> If you allow users to post content to your site then you should be taking measures to ensure that new users aren't spamming.

Of course, but breaking legitimate links as a form of spam control is just lazy.

We're going OT here, but...

> eliminating the "PR bleed"

There is no such thing. PR doesn't work this way, and the existence of outward links doesn't negatively impact the linking website in any way.

> It damn well should be. The link is the basis of the www.

I think it is just as much the basis of the www to show a link as it is to not show it.

> It damn well should be. The link is the basis of the www.

Anything on the www is only sacred if it's on your server. If you're on someone else's server it's up to them to extend the privilege of making links (or any content). Having links moderated for new users seems pretty fair--the vast majority of brand new users who make a post that contains links are spammers. The worst case is it will take a little time for a mod to clean up. You can even have your link be removed publicly but still function for members (so no Google juice but people can still answer your question).

If you have ever moderated a forum you will know how bad spammers can be. It's not just posts either, PM spam is really popular and a lot harder to police.

> It damn well should be. The link is the basis of the www.

Surely it is the right of whoever is hosting the website to decide how they want to present content. I don't think you should have some god-given right to expect a url you posted on someone else's site to display however you want.

I've considered rewriting user links to scramble the text and point them inward at a link proxy (which then routes out through an anonymizing service like anonym.to). Anything to destroy their value.

Haven't actually put it to practice yet though and I wouldn't be surprised if it wasn't a good idea. Unfortunately it seems that if spammers catch just a whiff of SEO from you, they'll never stop, like cockroaches.

[edit] I also haven't tried putting it into practice because I can imagine how utterly annoying it would be. I can't stand those "you are now leaving [site x]" pages myself.

I'm not a fan of those either. If you want to avoid having a "now leaving" page while still deterring spammers then you could create your own URL shortener with a separate domain. The link text could be the original link but the URL is the one generated by your shortener. You could even restore the original text with a replace() if you ever felt the need.

Yeah, I like that.

nofollow all external links

point external links through a redirect like /out?to=<url> and put /out in you robots.txt

change your url structure and simply 404 the pages with spammy links

These solutions assume that the spammers have human beings who are actively checking that their spamming effort worked on your forum.

I have a forum where I've been battling spam for years, and since the beginning all external links have been nofollowed... AFAIK this made no difference whatsoever.

That's partly true. But the reason a forum gets on their radar in the first place is by failing to follow those safeguards.

The way these guys operate is they write scripts to crawl the SERPs and find sites that are ranking, then they look for weaknesses (i.e. outbound followed links that don't go through a redirect).

Or they look at their spammy competitor's backlinks with AHrefs, Majestic, etc. and discover the forum that way. Then they help themselves to a link.

I saw the SEO post he's referring to (it was one of the articles in SEOMoz's top 10 monthly email). Site owners don't realize the amount of work and headache they cause forum communities when they contract out SEO work without an understanding of what that work entails. Or they do and just don't care.

Google's Penguin update didn't deter the spammers, either. Here we are nearly a year later and I'm still cleaning out accounts created en masse by XRumer or other bots.

I can't put words in the mouth of the SEO quoted, but (unless you know he was one of the people making removal requests against your site) I suspect he is not talking about you or sites like yours.

I should add that I don't agree with the rhetoric btw, but I think he is targeting a different kind of webmaster.

I think he is referring to webmasters who sold links (knowingly outside the guidelines) for years. They would previously have instantly removed the links if someone stopped paying.

As soon as Google stepped up their game and removing those links was important, those same webmasters started charging to take down the same links.

I would personally point to the irony of this (google creating a market that enriches people who have been abusing their system for years) rather than calling it immoral. Ymmv.

Hope that helps clarify some things and I hope I'm not distorting the guy's real meaning.

(written on my phone. Please excuse typos).

I'm wondering about the graph on Gary's site: what type of business' fortune is linked this tightly to Google's rankings? Is it a particular category, or are there several categories of businesses that suffer from (or leverage) this? That's a pretty frightening situation to be in, IMO.

Oh, there are many businesses at least that exposed. I personally wouldn't particularly relish running one, but that's another story...

How about taking inspiration from D&B[1]?

Don't charge for link removal; charge for priority link removal. Same day service: $1,000 per link. One week: $500. Etc.

Free link removal: First-come, first-serve at your own leisurely pace.

Sell the old forum to someone else and let them handle the requests.

BTW, PocoMail was a godsend back in the day. Thanks.

[1]:http://news.ycombinator.com/item?id=4994246

Props on PocoMail comment :)

So this "SEO" who helping his clients to "clean up their profiles" doesn't even know how to disavow links [1]. Well, it's not surprising that somebody who hired spammers once would hire another idiot later.

[1] http://www.youtube.com/watch?v=393nmCYFRtA and https://www.google.com/webmasters/tools/disavow-links-main (login required)

Well Google requires you to contact the website owners and do everything you can to get the links removed BEFORE you submit a list of disavow links. Google are supposed to check that you really did remove a large part of the links before disavowing.

This is a great post that I'm sure rings true with many 'a webmaster. Time is money. There's no two ways about it. If you want a webmaster to modify their site in your favor, you should be willing to compensate them for their time in doing so.

You could shoot the hostage and add rel='nofollow' to all outbound links. You should be able to programmatically do this.

Then with an auto-reply (or 'link policy' page) inform 3rd party sites that the link-juice (good or bad) is no longer flowing.

Does this constitute a long-term solution to the problem for active communities? That is, default to rel='nofollow' for an untrusted class of users, which presumably includes all the spammers?

that only works if the spammers realize their links are no longer effective; I'm not certain that that will actually happen...

If you have a relatively small site.. curating content isn't too bad... limiting the tail of articles so that comments can only be added for X days helps with that. Only allowing logged in users to comment (and making login easy via oauth sites like facebook/twitter/g+ etc) I kind of like Disqus, but would rather have something more home-grown.

I'd say rel=nofollow ALL user generated content.. but hey, that's just me.

Not at all. I was addressing the fact that when a 3rd party site asks for links to be removed (and are un-willing to pay for 'time' it takes to take links down) and then threaten to turn you into the google police for link spam... you can defend your position and show that you indeed are not participating in link shenanigans.

We did this, it made no difference.

What finally worked was a combo of: custom javascript on the signup form, disallowing new users from posting links in posts or signatures (and auto-banning ones who continually made attempts), insta-banning users who filled out non-linkedin links in the "linked-in link" profile field. We still get spam, but now it's maybe one post a week, instead of 10-20 a day, so it's at a level that moderators/user reports can handle it.

Rel=no follow discourages spam due to PR flow, but won't stop it. It still has an influence, however diminished.

Take an hour or two and write a script that selects all users with less than 100 posts and iterates through their posts removing links. The integrity of actual users is maintained while 99% of spam links eliminated. You spend a total of 2 hours, but don't have to deal with the emails any more - which could save many more hours.

No decent spammer cares about whether or not the links are nofollow. Targeting just dofollow links and then having a backlink portfolio to your site that is 99%+ dofollow links looks completely unnatural.

I'd be tempted to ask for money as well, but I feel like at the very least, you should ask for an apology. They may not have known that their SEO agency was using shady tactics, but since times changed, and it's obvious that they put bad links out there, it seems like an apology would be more than a token gesture.

The problem here is the work. The solution is to write a program. Have the white hat SEO people write a program that spiders the site with admin privileges and removes offending posts. It should come with a "dry run" mode that lets you spot check it. When they get it right, you can run it for them. It's a win-win: your forums get cleaned up and they did the work.

It's not really a win-win. Any time he spends on this is a loss from his point of view since the SEO's client already ruined his forum. It's some chutzpah to get annoyed when your victim refuses to help you hide your misdeeds.

If they do the work to undo the sins of the past, it helps the OP recover the value of his forums for far less effort than if he'd undertaken the job himself. I'd call that a win.

No, removing all the spam is not going to recover the community that fled long ago.

But why would the owner want to do that? By keeping those links up, the author is providing value to the Internet by lowering PR of folks who paid to spam him in the first place. I see no incentive for the author to clean this up.

If your business relies on SEO or is affected by Google search rank in a critical way, it may not be a viable business long-term.

Yes, that means I'm talking about a lot of online businesses.

The intelligent thing to do is sell a product or service that has value on its own and neither relies on SEO nor is it likely to be blames by other sites or companies for lowering their SEO.

I feel bad for these folks, but if you are planning on starting a business that doesn't really provide much value on its own that is identifyable outside of the roach motel of SEO, then you are headed into the ocean in a dingy with a small outboard motor, imo.

I've received similar emails. I've also been tempted to charge a fee for removal. I just ignore the emails.

Link removal requests can also be malicious. A blackhat seo will check his competitors domain to see if there's a spf record, if they're signing their mail, and if there's a catch-all (simply by checking if random mail is accepted). If there's neither they get a list of backlinks, from public web-crawls or sites like ahref, and request these links be taken down by sending mail with spoofed email addresses.

This is sad, sad, sad. I wonder how much awesome communities died that way. I stopped visiting Orkut when communities there got overran with spam. Also the same apply to some Usenet groups and Google Groups I used to like.

The solution is obvious -- write a routine that automatically goes through the entire forum database and disables all the links -- leave the names, but rewrite the links so they're just text, not hyperlinks. Sort of like:

s!http://!!g

The above deletes the "http:// prefix, but leaves the original destination name, in case anyone wants to object that their post has been edited after the fact. So technically, it's no longer a link back to the originating site, but it's otherwise unchanged.

No human intervention required. Problem solved.

There must be a programmatic solution to this problem.

Do some outbound links have value on this forum? If not, then you could remove all links, or remove the "link" part of the link (change @href to text).

If some outbound links have value you need to identify those, and it's more complex, but a Bayesian analysis of posts should be able to score posts on their "spaminess" and remove the links on only the most spam-like comments.

There may be some false-positive doing this, but since no information is actually removed (only the links, not the content) it should be quite ok.

Random links on forums already have pretty low value. I would assume that forum spam has come down now that this is the case.

As the article states, websites are looking the clean up the spammy inbound links as google has threatened to deindex sites that don't make an effort to do this.

There's a difference between buying links, which Google actively started prohibiting at a certain point, and the automated forum spam you are mentioning. The latter is done using software like XRumer, and has always been penalized by Google. The former wasn't prohibited in the beginning, when link deals were often mutual agreements, most of the time involving a traffic component as well.

The snake oil salesmen that are mentioned, are the ones who actively participated in the scheme by selling links and making a buck, and are trying to make another quick buck now that the rules have changed.

Moreover, some links aren't even paid at all, but just look manipulative. For example, if you developed a wordpress theme, and your link is in the footer of tons of blogs, you might get penalized for manipulating the anchor text of your links in a non-natural way. In those legitimate situations, webmasters do have a moral obligation to cooperate.

I don't think anyone would think that of an honest entrepreneur being spammed to death by link spamming software.

On a side note, there are plenty of forums on the web that have survived the spam wave, if it were core to your business, you could have protected yourself.

Wow, so Google penalizes a site if a link to that site shows up in spammy pages? That seems like a new business model for black-hat SEOers: "hey, nice site you have there - it'd be a shame if links to it started appearing all over my spammy network - $$$ will make sure that doesn't happen." Really search engines should just give zero weight rather than negative weight to links from spammy sites.

"hey, nice site you have there - it'd be a shame if links to it started appearing all over my spammy network"

Good way to clue Google in to this network, and then drop it from the index.

"Really search engines should just give zero weight rather than negative weight to links from spammy sites"

That's what was happening before Penguin/Panda - and it wasn't working. There's no risk to link spamming. Now there is.

Here is a question for those who know: is there anything forum regulars can do when we see spam posts beyond letting mods know about it?

I was active in a now dead forum that would get hit once or twice a week. The mods would clean it up with in a half a day but until then those posts would just sit. Does the Google web crawler consider words in replies to the spam post? I used to reply occasionally with words like: scam, fraud, got ripped off. I have always wondered if I was wasting my time. A few times I checked the link to see if it went some where legitimate (Google the base url), if so, I then searched the site for an informational web form. If there was one, left a message that their SEO company was using sleazy methods with a link to the forum post.

Google ought to make a code phrase that forum users can use to red flag spam posts. Though some of the posts were for Japanese and Chinese sites. The spam might not have been meant for Google but other search engines...

I had the same problem myself. In fact I still run a couple of forums. And I get at least 3 "remove my links" mails per day. I guess you can't ask them money, but they already cost you money (to maintain their spam and since the panda update read their emails asking you to remove it) so I guess the best solution is just ignore those mails.

Regarding anti spam I am afraid there is nothing you can do. Real humans will create an account for 0.01$ and they will post anything (I tried adding custom code in the post code as well, but those signups are not necessarily bots) In fact they even post if you don't allow them to post links (they post the urls with no http)

Next solution I'll try is social integration. Maybe that would work for some time but even this way spammers will find their way to create thousands of crap accounts in FB, twitter, etc.

Place a meta noindex, nofollow on the whole forum or at least the infected areas such as memberlist and user profiles. That could be done within 15 minutes and then you can answer with a standard reply that you did that to all emails you receive about removing links.

Does anyone else think that the author should set up something like mailchimp with an autoresponder to deal with the spam? Something that explains his policy, why he doesn't respond to emails, why he won't remove the link without charging them?

Many of the people who will have these spammy links on his forum will have paid one of the more scummy SEOs out there to raise their SERPs. The people emailing may not have originally been aware that the links have been put up. By having an autoresponder address for people to mail to this should alleviate the spam for all but the dumbest of people (that can't read what to sign up to to get a response, or those that fail to read a clearly defined policy via email).

Would love to hear feedback from the white-hat to this article.

Maybe get a decent discussion going from both sides. It does seem that the white^hat's remarks are very harsh, I wonder if he has honestly never thought about it from the other side's point of view.

I use XenForo and it's relatively simple to stop all forum spam for the most part. I just set up a custom captcha that only people who are interested in the topic would know the answer to, and then use XenForo's built in spam catching system for the rest...though I haven't had anything get through yet once I made the changes.

Also deleted forum footprint to dodge people scraping my forum off of Google, so that takes care of about 99% of all spam attempts. Someone would have to custom register accounts on my forum to get in, after which they would be destroyed by the spam catcher :)

Maybe we should not blame search engines, but SEO really has bad effects on the Internet...

PS, due to spams I've closed this forum (http://writingoutliner.com/forum), a online forum for my 'Addon documents organizer for MS Word'.

And I regret I use BBPress as the forum software, because new 2.x versions are not compatible with the old 1.x versions...

Maybe it could be scripted? Site owners would have to place some proof of ownership on their site (to be generated on the site of the forum owner - see Google web admins site verification), then a script could detect the faulty links on the forum and remove them automatically.

Oh man that is some sweet irony.

Just wait till the next waive of spam from SEOs trying to get competitors penalized then the competitors firing back and contacting you. Google stirred up a hornets nest because they couldn't figure out how to effectively devalue these links completely.

There exist very popular forums that receive no such spam.

Hacker News comments also seem free from it.

Any idea why it is that some forums get targeted with 10000's of spam accounts, while others don't? What is the trick to protecting if even capchas don't work?

They mainly use a program called xrumer to spam and it mostly works with your typically install of vbulletin or other popular forum software so custom software won't get hit. Kinda security by obscurity.

Why are people getting bent out of shape over some bad links? Can't you just disavow them in Google Webmaster Tools?

Reply on news.ycombinator.com