Hacker News Reader

"so I found it amusing that they don't include Blekko"

Just took a look at Blekko. It looks good, but you can't honestly believe the layman cares as much about it as he does about search engines coming out of MS or Google. Does the layman (MS' target audience here) even know about Blekko?

Honestly I'm not even a layman; I use DuckDuckGo and Google and read HN (obvs), but I've never even heard of Blekko. Looks nice, but I completely agree: it's a little disingenuous (or perhaps tunnel-visioned) for him to believe that people really know about it.

I first heard about Blekko through HN. I first learned of their API which I used to build most of unscatter.com on through HN too. The 3 card monte the original poster of this thread referred to, once again heard about it on HN. HN has also had links to articles about Blekko's investments from Yandex.

These were all front page. Over all I think Blekko has done a great job of being relevant and covered on HN.

Actually, here's a simple example. 44 pages when you search for Blekko using HNSearch - http://www.hnsearch.com/search#request/all&q=blekko

Heh, perhaps.

DDG is a great product, we are one of their partners and we are providing some of their search results. They are also quite popular on HN I think in part because of the cool things they have been doing about creating interesting one-box kinds of things. But that said, they aren't a search "engine" they are a search "provider." That isn't a bad thing, in fact its a good thing because you can get Google results through DDG without having to get Google's "value added" stuff like G+ friends feeds or being "bubbled" as Gabe is great at pointing out. But the difference is that if the search engines stopped providing results, DDG would stop being able to return them until they built their own index. As would a number of other places like search.com or dogpile.com etc. The weird thing about search is that until Blekko came along and built a third, there were really only two web scale indexes being used, Bing's and Google's. Amazing is it not?

So I think DDG is pretty cool, love the ideas that Gabe comes up with, and am happy to partner with him to provide access to our index.

To rebut your statement that I was being disingenuous however, I suppose it depends on how measure or define 'well known.' From the perspective of various traffic reporting companies blekko.com is a bit more well known than duckduckgo.com which is not a perfect measure either but it does provide a different perspective.

Awesome, now I know what to call Unscatter. I've been calling it a search site, because I don't currently have my own index of anything. I like search provider better. Thanks!

Hmm, we're (DDG) not a search engine? Come on.

Actually we crawl the Web a lot and have a bunch of our own indexes. A more apt description would be a hybrid search engine. More details at http://help.duckduckgo.com/customer/portal/articles/216399

Just to be clear, are you saying you no longer backfill results with Bing?

Also, we don't use any Google results right now.

No, your not. But that isn't a bad thing. From your link:

"we do not expect to be wholly independent from third-parties."

That is the technical difference, search engines do expect to be, if not wholly, then at least materially independent from third-parties, they are the third parties, search providers are all about the experience.

But lets be really clear, that terminology distinction really only matters when you get behind the search bar as far as the world is concerned we're all search engines, even if, like search.com, they don't index anything, and while DDG provides tailored indexes which support key aspects of your site's experience. The differentiation is the experience in that case not where you get your data. And I fully recognize that this distinction is not important for a large chunk of the Internet [1].

For a general audience, yes, we're all search engines.

For a technical audience, no, we're not. And the distinction is whether or not you have a generalized web index or not.

For organic search they present to the end user in a nearly identical way, caveat the 'experience' benefits of one over the other.

But for other things, like "tell me all the sites on the internet that copied this article verbatim." or "give me a rundown on link authority to all inlinks to this page" those kinds of things you need both a web index, and rights to use it like that. That's a pretty objective difference in capability.

So in technical company (and I consider HN to be technical) I try to be crisp about the terminology, in non-technical company I refer to all of these offerings as search engines because it is less confusing and frankly they don't care what its called, they type in words and get results.

[1] My father in law (part of the 99%) thinks he logs into "Google" to get to the internet because Chrome defaults to Google's home page when it starts up.

So the back story, and the source of my amusement, was that we've talked quite a bit to Bing. And in those discussions they got to see the positive benefit of some curation, and got to play the 3-card monte game for themselves (and lose a lot :-). And then they went out and did 'editors picks' and now they've done this 'bingiton' site. So its always nice to be driving the conversation around search, and amusing too.

So do I think the 'layman' cares? Yes and no. I think they care that they get good search results, I don't think they care where they come from. As pg points out building a replacement search engine is a Hard Problem(tm). You can make a competitive search engine with just 15 - 20 million dollars, but displacing the existing player is more nuanced.

This is something Bing knows all too well, they effectively created a clone of Google's search capability and they only pull in about 30% of the market. Blekko has been steadily adding share primarily by not being the same as Google, rather we've added specific features that were valued by different segments of the market (we're still the only search engine that lets you completely turn off ads as a user preference for example) but mostly folks just use what ever the search box provides.

[1] Note that both Yahoo and Bing are effectively the same search engine as Yahoo is 'powered by' Bing results. http://www.comscore.com/Press_Events/Press_Releases/2012/6/c...

(we're still the only search engine that lets you completely turn off ads as a user preference for example)

You can do that with DuckDuckGo: http://duckduckgo.com/settings.html > Layout tab > Advertisements.

[I work at DDG.]

That is awesome! I'm a firm believer in giving folks choices in this regard.

>and that the only viable weapon at the moment for human on human spamming (this is where real humans are working for 5 cents a page to write web pages that draw hits) is human judgement.

You are aware that Google hires thousands of search engine quality raters, all over the world right? But rather than directly curating, i suspect they just use these humans to train their algorithms. There is likely some inductive step going on, in selecting which algorithm (or 'sort' of the web) is selected, and then filtered by keywords.

Secondly, most search engines, actually do also use end-user curation. I have funky enough search queries, that I often get google redirects as search results: they want to know (and record) which link i pick. Also, because of all the google-adds-cookies, google-analytics-cookies, etc, they likely have 'session-bags' full of destination links and search queries that happen to correlate.

For me personally, neither search engine did very good in this test. But that's likely because neither result set is personalized in this test. And i don't know, if Bing does that now, but even if it does, for me to switch to another search engine, means to train another search engine in my contextual world. Hell, i even notice difference in result sets in Google based on my operating system. I get different results in Linux, than i do on my iPad.

People that get a lot of spam in search results, really ought to login, and stop clicking those links.

I used to be a fan of Blekko until you started spamming my favorite download site with a crappy toolbar. Why sell out so early?

This has come up a couple of times and I'd like to address it.

So one of the things that hinders the adoption/growth of search engines is muscle memory. Sometimes people type into their browser's address bar the address of a search engine, and then search, but more often than not they just click into the search box and start typing. If all the results are 'good enough' people don't bother to check who their search provider is, much less try others. You can manually change your search on Internet Explorer away from Bing to something else, you could manually change your search in Chrome from Google to something else, and you could change the default in the drop down box in Firefox to something other than Google. But if you don't know what you are missing, well you don't do that do you?

Now various sites which have bills to pay and no direct revenue stream, like download sites, offer a service to folks like Blekko which says "We can run an offer to our customers, which if they take us up on it you'll pay us for that." Just like Google gets paid by an advertiser if you click through their advertisement to the site and buy something, these services make your offer for you and then charge you if someone takes you up on it.

So that is pretty standard stuff, the down side is that this space has some pretty scammy operators. We have debated quite a bit about this internally. Our goal has always been to introduce people to Blekko who may not have heard about it (see above :-). However, we've also discovered that sometimes that policing these guys is a pain. They get paid if you install our toolbar (which gives you quick access to our search engine). We insist that they make it an option and clearly spell out how to continue on with your download or what not without choosing to install it, and we insist that the vendor of this software make it straightforward and easy to revert the install if folks aren't happy with it. So that leads to situations where lots of people download software and say 'no thanks' on the offer. Sometimes that annoys the site because they aren't getting paid and they "change things" so that it always installs whether you want it or not. When we find out about that we cut those people off. And if the toolbar doesn't uninstall we cut those folks off too. We had one vendor who gave us a 'final' release to QA where it worked easily and the actual release didn't. That really irritated us. Sadly there isn't the equivalent of the better business bureau for these folks, we discover them, add them to our list of bad actors and move on. We've put a number of changes in place to be better at managing the process and it may be that it ends up there are just too many scumbags in this space to make it a credible channel. Time will tell on that, in the meantime if you ever spot a site telling you that you must install a Blekko powered toolbar in order to proceed (which is to say its not completely obvious how to opt out) please report it so support@blekko.com and we'll fix it.

That's noble and all, but you know what, it's still a toolbar. If I see it in an installer it's still going to put a bad taste in my mouth. You know as well as I do that 95% of those installs are not intentional, or are manipulated. The kinds of users that install toolbars are not computer savvy, and when the computer says "you should install this thing", they're the ones to say "accept".

Blekko is pretty awesome. I'll admit. I've found stuff on Blekko that I doubt I'd ever find on Google.

To Everyone: Try it out. Blekko's results are a breath of fresh air.

Actually we (DDG) do have our own indexes (including web links). More details here: http://help.duckduckgo.com/customer/portal/articles/216399

This is a great summary from that link Gabe:

"we've focused on building a better search engine by concentrating on what we think are long-term value-adds -- having way more instant answers, way less spam, real privacy and a better overall search experience."

Now I'd probably word it 'building a better search experience' and remove the duplicate at the end but that's just word-smithing. One of reason we love you guys is that we share the same values in this regard. The more 'engine-like' you get away from 'experience-like' the more dubious Google and Bing get with sharing their index with you :-)

And, like you guys, we like to partner with folks who can provide the 'long tail' as it were.

"or even DDG although DDG is more a search utility rather than a search engine as it doesn't have its own web index."

Looks like you might be wrong about that.

DuckDuckGo gets its results from over 50 sources, including DuckDuckBot (our own crawler), crowd-sourced sites (in our own index), Yahoo! (through BOSS), embed.ly, WolframAlpha, EntireWeb, Bing, and Blekko.

http://help.duckduckgo.com/customer/portal/articles/216399-s...