A Duck & a Wiki Team Up Against the Content Farms | Hacker News Reader

A Duck & a Wiki Team Up Against the Content Farms

readwriteweb.com

57 pointsJackHerrick65 comments

Sometimes, whilst DDG's partnerships can be great for getting quality content to the top of search results; it's definitely not the sort of thing I'd want to see a dominant search engine doing. DDG obviously needs to do things differently in a way that can drive them traffic, but if Google were to do this, they'd quite rightly be blasted because such partnerships could massively impact the ability of other site's to grow organically without the engine's blessing.

I agree. Google's prominent ranking of Wikipedia articles usually drives me nuts unless it's the only good article around.

Google's ranking of Wikipedia articles is due in large part to Wikipedia's great PageRank ranking. I don't think Google is giving Wikipedia any sort of singled-out treatment, Wikipedia just ranks highly on Google's relevance metrics.

I know... but it still ranks really high and it's annoying because it's usually not the page that has the useful info when I'm searching.

I guess what works for some people doesn't work for others; most often when I'm looking for something I enjoy the wikipedia article on it and think that it's relevant. Also, a lot of times I don't actually go to google but type my query into the search bar (google is set as my default) so a search happens. It takes me to the first site, and most of the time it's wikipedia I want / get.

But I may just be an edge case. Hm.

No, I'm in your boat too. I almost always find the answer I'm looking for in the Wikipedia article, if there is one.

Me too. I usually include wiki in the search to make sure I get wikipedia heh.

The problem with Wikipedia is that many of their articles are poorly regurgitated versions of other source articles from the Web and that these source articles get no Google credit at all, even if they are linked from the Wikipedia articles, since Wikipedia uses no-follow links. Wikipedia ranks near the top in Google for almost any term for which it has an article, regardless of its quality.

Not always. Just yesterday I was knee-deep in wikipedia for linear algebra stuff and was surprised to find it more thorough than MathWorld.

That rough area of maths is one area well covered, after military and astrophyiscs I would say.

Google has mentioned they're working on the content farm problem. I would expect them to make some moves faster that are smarter than banning content farm X while partnering with content farm Y. Likely algorithmically, coz that's how they roll.

Google would do well for itself to stop horsing around with attempts to find 'algorithmic solutions' (if that's actually what they're doing), and in the short-term doing what's necessary to make a perceivable improvement in the quality of their product. Even if the content farms were changing domains regularly, Google could probably kill them faster than they'll find a way to algorithmically drop the ranking of their results, in aggregate.

For all of the recent noise that Google has been making about fixing this problem, the number of gamed search results in their index is still insanely high. Take plant diseases for example...every time I've made a search for some specific problem related to my houseplants, the results are so loaded with content-farm crap that I have to go at least a page or two in to get to reliable sources of information.

We're not even talking about relevant webspam from plant stores or online merchants...just eHow garbage, which is poorly regurgitated from the other sources that come lower in the results.

Yea, I think you're right there.

Although, perhaps we underestimate the scale they deal with. Plus, manually removing sites that are not clearly evil but on the border may become a PR nightmare.

For the record, the vast majority of our content farm stuff is algorithmically based. In addition, we've made some manual blocks and since the beginning we've been promoting good content in our "Zero-click Info" boxes. Recently it has been a lot of programming stuff, e.g. http://duckduckgo.com/?q=python+split

> Google has mentioned they're working on the content farm problem.

The irony is that is Google's own AdSense program is the incentive for the creation of airburger web sites, which otherwise have no reason to exist.

I'm sorry but I have big problems with calling DDG a real search engine. Their page states "DuckDuckGo is a search engine like Google." BS. It's a meta search engine that relies on other real search engines, such as Bing, to get the results. If ever becomes a threat to Bing, they'll cut them off in a second. DDG doesn't do the hard and resource-intensive work of crawling and ranking the pages, they just tweak the results of others in ways that gets them publicity on tech blogs.

For the record, I've been crawling the Web since the beginning of DDG (and before the BOSS or the Bing APIs existed). Yes, a one FTE, self-funded search engine, does not have the capacity to spend millions of dollars on crawling (does Google spend billions?).

I now focus my crawling efforts where I see they can add significant value, namely on spam removal and zero-click info. Our index actually shows the top result about half the time, which is not from any external API. Given that most people click the top result most of the time, this is not an insignificant addition.

We of course use external sources, but I think your comment is too dismissive of the intelligence we've woven on top and through them. I'm not interested in sharing all of DDG's trade secrets just to prove a point, but needless to say, you can simply compare the external APIs to our results across a swath of different types of queries and see the differences.

Finally, I find the premise of your comment a bit short-sighted, but I'm actually OK with being dismissed as a toy. In the eyes of the average user, they don't care where results came from at all. They just want the right information faster. So quite frankly, I think it is a reasonable move to use external APIs and focus on things like the top result, UI, etc. More recently, Yahoo and Ask seem to agree. Most people are dismissive of them too, but they've been doing a lot of innovative UI things, which matter a lot to end users.

Being dismissed as a "toy" is probably the best thing that could happen to DDG right now, and for the foreseeable future. They're searching for an angle against Google, a way in. They may well find one (although it's a tough battle).

I don't really care whether they're a "real search engine" or not. In my experience, they give better results than Google (less spammy content-farm stuff), and they disclose less of my personal information, so I use them instead of Google.

amen brother.

DDG is a meta search engine now, but this is a step towards being a real search engine someday. Harvesting queries, click through analysis, etc is vital to the development of a real SE. I would call this method SE boot-strapping.

DDG is supplanting labor intensive algorithmic improvements with manually manicured search results. Compared to Google's vast network of resources, it is a logical step for DDG to take. It is, essentially, a fast fix to a real problem.

Google has to do things the hard (and dare I say... proper?) way.

DDG is supplanting labor intensive algorithmic improvements with manually manicured search results.

This sentence is a paradox.

To illustrate what he meant, I'll give an example. Assume someone entered a search, "What are the best dog breeds?". Would it be easier to:

1. write an automated system which could dynamically deduce what the answer to this question is and/or what pages to return in the results, based on dynamically crawling the web, indexing, analying link structure, ranking, filtering, etc.

-or-

2. have a human write an answer to this fricking question and enter it into the database as the result for this search

hint: pick 2

it doesn't scale up to handle bazillions of different searches, but on a smaller scale and in tactical cases it's far simpler and easier to implement and can produce better results.

Of course by manicuring search results I mean blacklisting domains.

DDG has a laughably small market share. This doesn't necessarily make them a non-threat, but it's going to take more than this to change users' search behavior (ie convert them from Google to DDG) on even a minute level. I mean Google has the whole page preview thing now anyway, and the content farm thing should be fixed soon, as mentioned. Honestly, Google Instant is the best thing in search for a long time and it's so ingrained in my expectations now that I couldn't see switching to a search engine without it.

So asking what Google will do isn't really an important question.

their market share may be laughable, but that still doesn't eliminate the fact that many knowledgeable people prefer either blekko or duckduckgo over google.

it can definitely be viewed as a "calm before the storm". or if you're familiar with gentrification, the "cool cats" are living on this side of town.

Yes, many knowledgeable people prefer Blekko or DuckDuckGo. But that's "many" in the absolute sense, where "many" people are part of any arbitrary group on a planet of 7 billion.

Relative to the size of the group we're talking about, even among geeks, using DuckDuckGo as your main search engine is rare. It certainly doesn't look like DuckDuckGo has a glut of tech industry leaders using it exclusively — mostly just the odd geek here and there who wants to try something new or who has just decided he doesn't like Google.

How odd that even a year ago it seemed that the search engine space was closed to new entries, even solved - and now it feels wide open again.

I don't think so, the people using DDG are still a tiny tiny minority. If they did what google did with the browser market when they released Chrome then you would be warranted in saying they blew it wide open, but for what DDG has done with search? I don't think so, not yet anyway. Maybe when they've penetrated beyond the tech "elite" they might get somewhere, but right now if I ask any of my friends (who are relatively in the know about technology) they haven't heard of DDG.

Personally I don't think DDG will show wide success because the majority of people don't care about the added extras it provides, it's just other search engines (I think their major source is the bing api? Not 100% sure [1]) data with pretty labels and positioning, which very few care for. I search a relatively large amount and what DDG adds doesn't make it worth switching for me because I've become accustomed to how Google displays search results and adapted my behaviour to that.

[1] http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go-arc... (potentially no longer accurate which is why I'm not sure)

I think the difference is the attitude is different now - when Cuil launched the world laughed and said nothing could be as good as Google. Almost 3 years later a small, insignificant yet maybe not, number of people are starting to use alternatives.

A good parallel could be Firefox - Internet Explorer was the way to browse the internet, now they're just a way.

If Google wants to be the search engine a decade from now they may have to earn that right all over again.

People laughed at Cuil because it was laughably bad, not because nobody is allowed to make a search engine.

Yes, I think it would be laughed at today too.

Sure it was bad - but was every decision* they ever made so fundamentally wrong that they couldn't possibly have improved? The world wrote them off completely 8 minutes after they launched, and they were a meme like the next day. (* in search, not their ridiculous perks etc)

When I search for online games on DDG I get stuff like this in the first 'page' of results:

- http://onlinegames.innovative-solutions-group.com/

- http://onlinegames.webfreehosting.net/

- http://online--games.com/

Did DDG and Blekko both independently, really stumble upon formulas that beat Google, or are we actually open to the possibility of not using Google for search now?

Anything can be improved — but in Cuil's case, yes, it would have required reversing pretty much every decision they made (or at least implementing them on such a different level they would be unrecognizable). It was just breathtakingly ill-advised. DuckDuckGo is more clear about the benefits it offers, modest though they may be, and it follows through on its boasts. And Blekko is just modest all around.

I said at the time and still say that Cuill released a very early stage experiment that still hadn't worked out what it wanted to do and called it a search engine.

Thx for the specific example, always very helpful! But in this case, I'm having trouble reproducing. Do you have a region set or something? I'm looking at http://duckduckgo.com/?q=online+games and don't see those domains. I'm not saying the other domains are good :), but I just want to understand what you're seeing. We definitely have a problem right now of over-weighted the domain name in the url.

I'm in latin america (Nicaragua) which might hurt a lot if it's trying to pull localized results from this little corner of the world heh.

http://playtomic.com/ddg.png is what I'm seeing.

Edit:

I'm doing a search from one of my US-based servers and seeing those same sites?

http://playtomic.com/ddg2.png

Ahh, thx. Turns out I was the one with the settings set :). Duly noted!

The difference is Matt_Cutts, moultano, nostrademons (Google employees) et al. are all over DDG threads, but weren't on Cuil ones as much, e.g. http://news.ycombinator.com/item?id=258737 http://news.ycombinator.com/item?id=1255122.

I agree though, that DDG is a blip -- a teeny tiny blip in the search engine market -- so, quite frankly, I'm not sure why they care so much, or at all.

They care because maybe tomorrow you're not just a tiny blip and they're not as entrenched as they are today. Unlikely? Sure, but it does happen.

I think the difference in engagement by those guys is that Cuil was trying to be stealth and downplay the hype of being "google 2.0 by ex-googlers" whereas your marketing has been squarely aimed at them.

Intentions aside, Cuil was presented as a "Google killer" to orders of more people than know about DDG. But higher level, I don't think DDG warrants their engagement at all. There are much bigger fish to fry, so-to-speak, and yet I don't see them on all the higher profile search engine stories out there across the Internet.

So we have the website equivalent of spam. A simple solution would be to have a secondary page labeled "We think this is spam" where all the banned links would reside. Anyone who needs to see what's there could always dig in.

Is DDG one person?

Yeah, Gabriel Weinberg: http://www.gabrielweinberg.com/

Don't forget Eli!

yr a worse ddg fanboy than me :P

While I'm kind of frustrated by the anti-Google marketing, I'm impressed by how productive Gabriel clearly is.

Why was this down voted?

google's not going to do anything because they're in the business of making money off of search and duckduckgo is not

Then what is their business?

I actually have a lot of faith in Google to do the right thing for their users. After all they mostly walked away, or at least severely hampered their business in China. They will be more than willing to walk away from sites their users think are low quality.

I'm not saying that this means they will do the exact same thing DDG did. I'm just saying any decisions they make will be primarily based on looking out for their users best interest rather than short term adsense revenue.

what about your statement is supported by their actual actions in the recent years? who are their actual customers, searchers or adwords bidders? They simply need to provide a decent enough search experience to keep users coming back so they can keep adwords bidders participating. It's their entire business model.

In what way is ddg not in the business of making money?

Do you happen to know how DDG is making money, then?

Gabriel has money from previous businesses :p

Yes, but it is not spent on DDG. We live off my wife's salary.

That's the way to do it ;)

I know of a couple ways:

- Adding affiliate codes to the search results: https://duckduckgo.com/privacy.html The relevant part is "we may add an affiliate code to some eCommerce sites (e.g. Amazon & eBay)". It looks like this is where most of the revenue comes from, according to http://blogs.wsj.com/venturecapital/2011/01/26/can-a-start-u... . Relevant part is "The site is self-funded and saw about $15,000 in revenue last year, mostly through Amazon’s affiliate program, which pays whenever a customer buys an Amazon product after clicking on a DuckDuckGo link."

- Sponsorships where the sponsor banner displays on all search result pages: http://www.gabrielweinberg.com/blog/2010/05/duck-duck-go-tra...

Wouldn't this mean DDG is giving amazon information about it's users?

I love DDG, but this seems shady putting the affiliate code into search results. Amazon ranks great, but it seems this would encourage DDG to rank Amazon even higher. Google would get in quite a bit of trouble for doing this.

No, we're not giving any more information than would otherwise flow by clicking on a link in the SERPs. And actually, we're giving less information than if you would click on the same result in Google because we're not sending your search terms via the referrer header. We've stayed away from things like commission junction because then you would be giving away information as it is routed through a third-party. For more info, see the privacy policy: http://duckduckgo.com/privacy.html. In the other direction, Amazon doesn't share any personal info.

Wow, Matt, I'm not sure if I should be flattered or scared that you're paying this close attention to DuckDuckGo :).

The $15,000 is less than my graduate school stipend was. Needless to say, we're not raking in the dough here.

I just added this to the FAQ, i.e. how do you make money (I don't see any ads)? We'd like to show minimal advertising and make the venture more sustainable, for sure, but haven't been able to find a good ad provider willing to work within the DDG privacy principles.

he's apparently used duckduckgo. i can recognize it from a mile away because the stalking/wow factor that all us users experience after using it seems to have also bit him. :)

wait, what? Am I wrong in saying that google is not in the business of making money? They're making a damn killing off of search traffic.

Duckduckgo is not. They're getting press for being defiant.

No, but you're implying that DDG either has no plans to make money (which is possible, if rather a short-lived plan) or will remain small enough to be reliant entirely on Gabriel's day job.

DDG is a search engine that's using its search to make money. Just like Google.

Reply on news.ycombinator.com