Hacker News Reader

Comment by dangrossman | Hacker News Reader

While it's always a bit annoying to see a meme take up article slots on Hacker News regularly, I don't mind this one. I see sites like Mahalo as doing massive disservice to huge numbers of people --

* thousands of AdWords advertisers that have paid for their ads to be matched with content sites, not scraper pages with a huge ad to text ratio

* thousands of publishers whose work is being scraped, aggregated and outranked without as much as a backlink

* millions of web searchers that are hitting these pages instead of the real sources of the content they were searching for

And calling out companies that harm the fabric of the web for everyone else is worth doing.

View on HN

The question is: why doesn't google do anything about this? There are countless such pages. Why not just blacklist domains? Or artificially give them lower pagerank?

Or is it okay as long as they get ad-sense money?

they do it all the time, as long as it's a small site, Mahalo's existence basically flies in the face of all of Google's rules and regulations

An old boss told me, "When you're the man, the rules don't apply to you."

Not to say that Calacanis is the man, but to illustrate that rules are there to suit the rule makers and in this case, it suits the rule makers to ignore their own rules.

So why didn't Google nip Maholo in the bud when it was starting out? Insider connections?

because when they launched they were being touted as a search engine, not a content farm

They went through a bunch of different iterations and stuff before they established as the site they are now. And look at the funding of Mahalo and Google. Same early people.

they do block domains, at least from their organic search engine results.

i accidentally got one of mine blocked when playing around with some random software that produced a site that was somehow against their TOS. i didn't have it live long, but long enough for the spiders to grab and flag. still waiting on a reconsideration request.

why not mahalo? who knows. if i had to venture a guess, i'd say that the larger the site and the more revenue google sees from that site due to adwords and the like, the higher up the chain the decision to block it has to come from.

I think Google prefers to programmatically detect and blacklist behaviours, without too many innocent casualties - even if working out how to do that takes longer. They aren't interested in playing whack-a-mole.

They do often play whack-a-mole at the lower level too...but they also take human reviewers into account so it is not so expensive on engineering time.

And at the higher level they absolutely do penalize people individually to send a message. Watch the opening of this video if you think otherwise http://blogoscoped.com/archive/2008-07-08-n14.html

Can they then at least pick one scraping site from all the others (to prevent duplicate information)?

Although this is perfectly legal, there is quite a few big sites that scrapes Wikipedia and adds ad-sense. If you are searching for a narrow topic, page 1-2 is WP and WP copies.

As a sidenote - I see that Google Search has upvotes, comments and deletes in the search - yet it does not seem as if anyone uses it. Why? Why can't the user at least blacklist sites?

That's only if you are logged in, and it only affects that user.You cannot up/down vote the Google results for everyone.

>If you are searching for a narrow topic, page 1-2 is WP and WP copies.

It would be good to have a "remove all pages with Wikipedia content" button on searches. Sometimes it seems half the results are simply Wikipedia copies.

They've played whack a mole plenty of times. They do it daily. This isn't some random guess or 'I think' like you're saying. This is fact.

Isn't there a search engine out there that takes into account the advertisements vs. content ratio? It would be trivial to get rid of Made for Adsense sites by scanning for the Adsense code used and seeing how often it's used on a page and the percentage of the page it takes up.

Duck Duck Go tries to do this.

There are a lot of good results that have a similar ratio of ads to "content" unless your definition of content is very sophisticated.

Google allows you to have three ads though. If you had a page with three ads and the best written 500 word article on it's topic, your definition of MFA sites would have it banned even though it is obviously a ton better than most sites around.

For smaller publishers, it's usually three blocks of ads plus two blocks of link units. For those not familiar with AdSense, link units are the horizontal or vertical lists which take the visitor to a search results page rather than straight to an advertiser's page.

I think a strict definition of MFA is always going to be a little grey, but there's clearly something wrong with pages that are nothing other than widgets, scraped content and the like.

I work for Google, and while I don't know anything about this particular situation, one invariant that I've been consistently impressed with is that the ads side of the company doesn't influence the search side of the company _at all_. I've seen well meaning ads support folks emailing people in search quality about an innocent problem with one of their customer's sites, and getting sternly told that they should never again send an email to those mailing lists.

Offtopic but I for one would like to see Google do this for personal information or at least provide the option to do so i.e. info from one G product does not leak to another without explicit opt-in.

Its really a shame how low Google's standards are for AdSense. I've tried advertising in the content network several times, but each time I end up wasting so much time blocking MFA sites that I give up.

The rub is, if Google were to block all of Mahalo, a lawsuit would surely be filed.

I do not think anyone is asking for all of Mahalo to be blocked. The major issue is the insane amount of auto-generated pages that are not given a noindex call and are also added to the site map xml file.

These actions appear to be conscious choices to make sure that google will index these pages, which according to several sources, have nothing but "scraped" content and are therefore spam.

And Mahalo would lose. Google penalizes SEO spammers hundreds of times every day. It's very easy to see that Mahalo does not follow the rules.

Google is providing a free service in a field in which there is intense competition. I don't see what basis anyone would have for filing a lawsuit for not being indexed in Google.

Regardless, it still happens all the time.

agreed.

They could argue that Google has a monopoly on online search, and that excluding people from their search results is an illegal restraint of trade. I don't think that such a case would be a cut-and-dry victory for Google if the spammer (as in this case) has the money to fight them.

Ridiculous. I don't know how the law works but I do know that if Google was to somehow lose a case like that, their entire search engine would be worthless incredibly quickly with grey hat/black hat people going crazy.

Although I understand your points the AdWords and search issues is googles problem. People pay for a product with flaws in it and google is selling, not Mahalo. Mahalo has the freedom to do whatever they want with their pages and they can't be held responsible for what google in it's turn does with them.

Now, stealing other peoples work, that is obviously Mahalos doing and should be a case for the court.

I'm not really sure what the point with this article is. Mahalo spams google - so what? It's up to them to do this and up to google to prevent it. Business is business, even on the internet.

Here's one thing to consider : Google's algorithms for detecting the actual value of clicks for advertisers has improved greatly over the past few years.

If Mahalo’s traffic was utter crap, it would be dropped.

"If Mahalo’s traffic was utter crap, it would be dropped."

That's not even close to true. If it were, there would be no AdSense for parked pages program. Their eye is still very much on their own bottom line, not the advertisers.

There's also the possibility that aggregator sites are actually useful for some portion of the Internet users (read: the "facebook login" set) and the advertisers that cater to them. Too bad "we" have to share Google with "them".

Not to be too cynical, but after several years in web marketing I've found that in this case "we" definitely do not outnumber "them." Not by any stretch of the imagination.

That goes without saying.

Here is a link about how search monetizes when compared to content sites (~ 30 to 1!)

http://www.emarketer.com/Article.aspx?R=1007053

As an extension of that, sites which get most of their traffic from search often have the highest ad CPM values because the ad targeting is so relevant on the AdSense ads. And this is true even moreso now that Google is also using the referrer for ad targeting

http://adsense.blogspot.com/2010/02/better-contextual-matchi...

Scraper sites do quite well on the ad revenues front because most of the traffic is search driven, and because the content ends up so bad that the ads look all the more appealing.

"Their eye is still very much on their own bottom line, not the advertisers."

I know for a fact that Google is balancing advertiser ROI against advertising spend and ensuring that they don't upset the apple cart.

However you are absolutely right that Google is maximizing their profit – having the ROI numbers in hand lets them drive down overall sales-from-adsense margins while still keeping the customer.

So my original point is that if Mahalo was flat out non-performing crap, it would be tossed.

An argument could be made though that they use the Mahalo stuff to grow ad inventory, while balancing out the ROI to their advantage since it likely converts on the lower end of the scale.

My conversion rates from the content network have never been as high as search. I try new campaigns targeting both networks every once in a while but always end up either opting out or setting a much lower bid on the content side.

[citation needed]

Reply on news.ycombinator.com