Jason Calacanis Knows He's Spamming Google, He Just Thinks It's No Big Deal | Hacker News Reader

Jason Calacanis Knows He's Spamming Google, He Just Thinks It's No Big Deal

smackdown.blogsblogsblogs.com

157 pointsmvandemar83 comments

While it's always a bit annoying to see a meme take up article slots on Hacker News regularly, I don't mind this one. I see sites like Mahalo as doing massive disservice to huge numbers of people --

* thousands of AdWords advertisers that have paid for their ads to be matched with content sites, not scraper pages with a huge ad to text ratio

* thousands of publishers whose work is being scraped, aggregated and outranked without as much as a backlink

* millions of web searchers that are hitting these pages instead of the real sources of the content they were searching for

And calling out companies that harm the fabric of the web for everyone else is worth doing.

Here's one thing to consider : Google's algorithms for detecting the actual value of clicks for advertisers has improved greatly over the past few years.

If Mahalo’s traffic was utter crap, it would be dropped.

"If Mahalo’s traffic was utter crap, it would be dropped."

That's not even close to true. If it were, there would be no AdSense for parked pages program. Their eye is still very much on their own bottom line, not the advertisers.

There's also the possibility that aggregator sites are actually useful for some portion of the Internet users (read: the "facebook login" set) and the advertisers that cater to them. Too bad "we" have to share Google with "them".

Not to be too cynical, but after several years in web marketing I've found that in this case "we" definitely do not outnumber "them." Not by any stretch of the imagination.

That goes without saying.

Here is a link about how search monetizes when compared to content sites (~ 30 to 1!)

http://www.emarketer.com/Article.aspx?R=1007053

As an extension of that, sites which get most of their traffic from search often have the highest ad CPM values because the ad targeting is so relevant on the AdSense ads. And this is true even moreso now that Google is also using the referrer for ad targeting

http://adsense.blogspot.com/2010/02/better-contextual-matchi...

Scraper sites do quite well on the ad revenues front because most of the traffic is search driven, and because the content ends up so bad that the ads look all the more appealing.

"Their eye is still very much on their own bottom line, not the advertisers."

I know for a fact that Google is balancing advertiser ROI against advertising spend and ensuring that they don't upset the apple cart.

However you are absolutely right that Google is maximizing their profit – having the ROI numbers in hand lets them drive down overall sales-from-adsense margins while still keeping the customer.

So my original point is that if Mahalo was flat out non-performing crap, it would be tossed.

An argument could be made though that they use the Mahalo stuff to grow ad inventory, while balancing out the ROI to their advantage since it likely converts on the lower end of the scale.

My conversion rates from the content network have never been as high as search. I try new campaigns targeting both networks every once in a while but always end up either opting out or setting a much lower bid on the content side.

[citation needed]

Its really a shame how low Google's standards are for AdSense. I've tried advertising in the content network several times, but each time I end up wasting so much time blocking MFA sites that I give up.

The rub is, if Google were to block all of Mahalo, a lawsuit would surely be filed.

I do not think anyone is asking for all of Mahalo to be blocked. The major issue is the insane amount of auto-generated pages that are not given a noindex call and are also added to the site map xml file.

These actions appear to be conscious choices to make sure that google will index these pages, which according to several sources, have nothing but "scraped" content and are therefore spam.

And Mahalo would lose. Google penalizes SEO spammers hundreds of times every day. It's very easy to see that Mahalo does not follow the rules.

Google is providing a free service in a field in which there is intense competition. I don't see what basis anyone would have for filing a lawsuit for not being indexed in Google.

Regardless, it still happens all the time.

agreed.

They could argue that Google has a monopoly on online search, and that excluding people from their search results is an illegal restraint of trade. I don't think that such a case would be a cut-and-dry victory for Google if the spammer (as in this case) has the money to fight them.

Ridiculous. I don't know how the law works but I do know that if Google was to somehow lose a case like that, their entire search engine would be worthless incredibly quickly with grey hat/black hat people going crazy.

The question is: why doesn't google do anything about this? There are countless such pages. Why not just blacklist domains? Or artificially give them lower pagerank?

Or is it okay as long as they get ad-sense money?

they do it all the time, as long as it's a small site, Mahalo's existence basically flies in the face of all of Google's rules and regulations

An old boss told me, "When you're the man, the rules don't apply to you."

Not to say that Calacanis is the man, but to illustrate that rules are there to suit the rule makers and in this case, it suits the rule makers to ignore their own rules.

So why didn't Google nip Maholo in the bud when it was starting out? Insider connections?

because when they launched they were being touted as a search engine, not a content farm

They went through a bunch of different iterations and stuff before they established as the site they are now. And look at the funding of Mahalo and Google. Same early people.

they do block domains, at least from their organic search engine results.

i accidentally got one of mine blocked when playing around with some random software that produced a site that was somehow against their TOS. i didn't have it live long, but long enough for the spiders to grab and flag. still waiting on a reconsideration request.

why not mahalo? who knows. if i had to venture a guess, i'd say that the larger the site and the more revenue google sees from that site due to adwords and the like, the higher up the chain the decision to block it has to come from.

I think Google prefers to programmatically detect and blacklist behaviours, without too many innocent casualties - even if working out how to do that takes longer. They aren't interested in playing whack-a-mole.

They do often play whack-a-mole at the lower level too...but they also take human reviewers into account so it is not so expensive on engineering time.

And at the higher level they absolutely do penalize people individually to send a message. Watch the opening of this video if you think otherwise http://blogoscoped.com/archive/2008-07-08-n14.html

Can they then at least pick one scraping site from all the others (to prevent duplicate information)?

Although this is perfectly legal, there is quite a few big sites that scrapes Wikipedia and adds ad-sense. If you are searching for a narrow topic, page 1-2 is WP and WP copies.

As a sidenote - I see that Google Search has upvotes, comments and deletes in the search - yet it does not seem as if anyone uses it. Why? Why can't the user at least blacklist sites?

That's only if you are logged in, and it only affects that user.You cannot up/down vote the Google results for everyone.

>If you are searching for a narrow topic, page 1-2 is WP and WP copies.

It would be good to have a "remove all pages with Wikipedia content" button on searches. Sometimes it seems half the results are simply Wikipedia copies.

They've played whack a mole plenty of times. They do it daily. This isn't some random guess or 'I think' like you're saying. This is fact.

Isn't there a search engine out there that takes into account the advertisements vs. content ratio? It would be trivial to get rid of Made for Adsense sites by scanning for the Adsense code used and seeing how often it's used on a page and the percentage of the page it takes up.

Duck Duck Go tries to do this.

There are a lot of good results that have a similar ratio of ads to "content" unless your definition of content is very sophisticated.

Google allows you to have three ads though. If you had a page with three ads and the best written 500 word article on it's topic, your definition of MFA sites would have it banned even though it is obviously a ton better than most sites around.

For smaller publishers, it's usually three blocks of ads plus two blocks of link units. For those not familiar with AdSense, link units are the horizontal or vertical lists which take the visitor to a search results page rather than straight to an advertiser's page.

I think a strict definition of MFA is always going to be a little grey, but there's clearly something wrong with pages that are nothing other than widgets, scraped content and the like.

I work for Google, and while I don't know anything about this particular situation, one invariant that I've been consistently impressed with is that the ads side of the company doesn't influence the search side of the company _at all_. I've seen well meaning ads support folks emailing people in search quality about an innocent problem with one of their customer's sites, and getting sternly told that they should never again send an email to those mailing lists.

Offtopic but I for one would like to see Google do this for personal information or at least provide the option to do so i.e. info from one G product does not leak to another without explicit opt-in.

Although I understand your points the AdWords and search issues is googles problem. People pay for a product with flaws in it and google is selling, not Mahalo. Mahalo has the freedom to do whatever they want with their pages and they can't be held responsible for what google in it's turn does with them.

Now, stealing other peoples work, that is obviously Mahalos doing and should be a case for the court.

I'm not really sure what the point with this article is. Mahalo spams google - so what? It's up to them to do this and up to google to prevent it. Business is business, even on the internet.

It's really too bad Google doesn't allow you to simply blacklist domains from your search results permanently. The thing that frustrates me the most about these SPAM sites is the fact they're constantly popping up and my down voting of the result seems to do absolutely nothing unless I'm using the exact same search query. Just let me blacklist Mahalo and other sites like it permanently. Better yet make it possible to subscribe to a blocklist so the community can pool it's resources and fight back.

I request this from Google every couple of months. If I could remove an entire domain from all my personalized search results, I'd be soooo happy.

I've requested this, too. I even switched to Yahoo! for a few hours because they let me blacklist expertsexchange.com (since changed to redirect to experts-exchange.com). I don't see the option on Yahoo!'s page anymore.

expert sex change .com?

Yes. It used to show up in Google search results as expertsexchange.com. After being thoroughly razzed, they made it redirect to experts-exchange.com.

Since it was called expertsexchange.com at the time I blocked it in the Yahoo! search results, I took the liberty of referring to it by its old name.

With Google custom search engine you can create your own personalized search service...built off the back of Google, but while delisting any sites you do not want results from.

And it only takes about 30 seconds to set one up. Then you embed the search box wherever you like.

http://www.google.com/cse/

Jason Calacanis is the Paris Hilton of the web.

Mahalo is not particularly interesting, not particularly evil, he doesn't really do anything, and yet - we keep talking about him and putting his shit on the front page of hacker news.

It's a mediocre aggregator/linkfarm, with some mechanical turk style incentives for humans to contribute, and a nice chunk of $ in the bank. Just ignore/mock him for another year or two until the funding runs dry.

You are absolutely right, it is mediocre crap. The issue is that through the spam techniques described in the article, he can now gain undeserved top 10 rankings for many phrases using pages that have no business being there.

If you are not part of the web development community then these discussions will most likely bore the hell out of you. However, if you are, and if you are aware of how many innocent sites Google bans or penalizes on a daily basis, or AdSense accounts that get canceled with no appeal for offenses much less than his, then this stuff actually matters.

One of the things I really miss in google is a persistent blocking preference a.k.a. site blacklist. Mahalo would go straight in there, along with expert-sexchange, sedo parking and a few others.

Depending on the browser you use, this is easy to do. I have experts-exchange and other useless sites blocked using greasemonkey/greasemetal.

Yeah, sedo parking is a bitch tho, since by it's very nature it's tons of different domains. You would need a way to block by app.

a year on, maybe not so crazy now eh? http://news.ycombinator.com/item?id=485618

Heh, got any stock picks you want to share? :)

Nice call.

someones network of contacts > everything

SEO is snake oil. This is further proof that the whole "industry" is ruining the integrity and usefulness of the internet.

Not everything that uses the phrase "SEO" should be painted with the same brush. A lot of SEO best practices are making the HTML markup more semantic and human-readable. White-hat SEO dovetails pretty well with a human-readable web and accessibility standards, and is one of the better business cases for kicking the Flash habit. Just because Calacanis makes his name by abusing loopholes in the algorithms doesn't mean that's the only sort of thing that the name "SEO" applies to.

That is the angle Jason used when he created his steaming pile. It doesn't mean that anyone in the industry agrees with Jason's strategy.

And if you want to place the blame where it belongs remember that Google is the company funding all this content scraping with their ads programs.

I just tried searching for a recent post from an official Google blog (about AdSense using referral data for more relevant ad targeting) and found a scraper site with their ads outranking them for their own content. Pretty sad.

Actually, pretty savvy.

Google gets paid for the scraped site because of ad impressions and possible clicks.

Google does not get paid for running an informative blog, in fact it's a cost to them.

Google cares about getting paid.

>> "Currently when I look, Google tells me that Mahalo has 356,000 pages indexed"

I see 'Results 1 - 10 of about 2,200,000 from mahalo.com'

Have things stepped up a gear or am I misunderstanding?

No, different datacenters will show different results... sometimes very different. It also matters if you are visiting Google.com or one of the country variants.

According to what Jason said in another comment, however, all of their pages are listed in their xml sitemap, and all of those are listed in a master xml sitemap index located here (warning! huge files if you follow the links in the first one!):

http://www.mahalo.com/sitemapindex.xml

Based on what I saw 2 million+ looks like a huge overestimate, if what Jason said is true.

Edit: My bad, frederickcook's answer was the right one. I didn't realize you were doing a regular text search.

There are 12 sitemap-mahalo.xml files, each with 50,000 urls in it. The 12th has 48,661 urls by my count. That gives a total size of 598,661 pages.

Methodology:

  wget http://www.mahalo.com/sitemapindex.xml
  cat sitemapindex.xml |sed -e's/[<>]/\n/g'|grep ^http|xargs wget
  for P in sitemap-mahalo.xml* ; do echo $P ; cat $P |sed -e's/[<>]/\n/g'|grep ^http|wc -l ; done

Edit: Formatting

Search "site:mahalo.com"

Simply "mahalo.com" lists every indexed page with that text on it, such as this one.

Yup that's what I did. Maybe the particular google datacenter I hit had a high estimate there.

It also says 59,300,000 for scribd.com which I can't help but draw comparisons with this.

this is getting really old and we're not interested in doing anything black hat or even gray hat. as such we're doing the following:

1. we're removing (or building out) any page in our system created by our users with under 200 words of original content. This will take a couple of weeks but it's tarted.

2. we're not letting users create stub pages (short pages) until we can noindex them and put them in a different directory (i.e. /stubs/) so google can easily tell the difference between them.

these pages are < 1% of our revenue and low single digits of our traffic. we don't benefit from them materially, and I think we're being targeted by Aaron Wall and other SEOs for my "seo is bullshit" comment from 2005 or so.

I guess that is fine... I gotta live with the ramifications of what I say. however, for the record I don't believe that SEO is BS any more... when i said that it was when we were building joystiq and autoblog and we spent zero time on SEO.

All that being said, we're being targeted by a small group of folks who want to take us down. we're only going to get stronger from this because our hundreds of contributors are rallying around building out the short pages.

Topix, Kosmix, NYTimes and Zimbio are all making quality topic pages and are not getting attacked over it. not sure why there is some double standard.

regardless.... this is not a material thing for us. we're flushing all these pages and moving them to a different directory going forward so that search engines know where they are located (i.e. /stubs/ ).

thanks for the ass kicking.... having a horrible day today over this.

jcal

http://bit.ly/jasondown

we're removing (or building out) any page in our system created by our users with under 200 words of original content. This will take a couple of weeks but it's tarted.

i'm also getting a list of every page under 300 words and having the page managers build them out in 30 days or deleting them.

from: http://news.ycombinator.com/item?id=1143512

is it under 200 words or under 300? are the goal posts moving already?

300 is the goal right now.... we're making plans right now.

.... keep the attacks up, only makes us more motivated.

There is a big difference between stating an observation and an attack. You can (and have) dismissed the Google guidelines as irrelevant. But you shouldn't slag off others as spammers if you are going to do far worse.

You are the one who attacked people. We are simply tracking how well you performed in your efforts.

Hold yourself to your own standards or close your mouth. Period.

Aaron: let's be real for a moment. You're a bit of a troll and you're certainly obsessed with me to an unhealthy level. I'm flattered, but it is getting old.

You pointed out some minor stuff we could do better, we're doing better. Now go back to your basement and hit F9 on Chatroulette.

Jason, either you are not actually reading what aaron's saying or you have an emotional intelligence worse than my mother. Just saying.

Aarons stating facts. You are responding ad hominem. Perhaps you should hold off on the reply button.

Jason, you've always been a total douche. I'm glad the rest of the world is finally catching up.

Jason, did you even read my article? This isn't about the traffic those autogenerated pages get, it's about the fact that through the minuscule amounts of PageRank that they are each capable of grabbing, you are now able to rank your mediocre pages with absolutely zero influence from the rest of the web.

We're not talking about stub pages, it's all the fully automated bullshit that you are generating. They not only need to be deindexed, they need to be nofollowed or removed altogether.

How is it you are out there playing the wounded puppy when apparently you haven't even read the articles or followed the reference links? You can't just skim this one and then craft a rebuttal and think you've addressed the issue. There's a lot of data in those paragraphs you apparently just skimmed over (if that even).

You have over 500,000 pages listed in your xml sitemap, and Google appears to have over 330,000 of them indexed. Click on this link, please, and actually go look at 10-12 of the pages we are talking about here:

http://tinyurl.com/yzmxq7b

Tell me how long it takes you, just by clicking through, to find even 3 pages that have any human interaction in them whatsoever.

Maybe, just maybe, you really don't have a clue what is happening. I personally don't believe that's the case, but if so then whoever it is you have working for you that set this up knows how to spam like a pro.

Try site:kosmix.com or site:righthealth.com

Kosmix always had noindex in their "search results" - Now stop whining like a baby and get your ass back to work instead of justifying your mistakes.

Jason - the "seo is bs" line was heard at every seo conference you headlined when you launched mahalo.

this is getting very, very old. can we move on? please??

It's obviously of interest and importance to a number of people involved in this field or troubled by poor quality material showing up in Google. If you're not one of those people, it's pretty easy to identify these links and not upvote them or visit the articles/comments, etc.

There are countless articles on HN that I have no interest in (e.g., I don't even know what Clojure is), but I just don't click through to them.

Think this thread has gotten way out of hand.

A lot of the comments in here seem like more of a personal attack than anything else. You might as well change the title of this post to "Jason Calacanis ruined the Internet" or something to that effect..

Can we stop the drama already? I think we're going to need a hose to control this mob.

Reply on news.ycombinator.com