Bing vs. Google | Hacker News Reader

Bing vs. Google

bingiton.com

100 pointspykello162 comments

Ok, so I found it amusing that they don't include Blekko, I mean we did make this into a fun game with our three card monte game with our /monte tag in search results :-) And I suppose it doesn't really serve their interests to 'lose' to a little guy either.

That said, its an interesting experiment. And at blekko.com we've been running for over a year [1].

What we've found is that there is a huge brand bias, which is to say that if you use some search engine as your primary search engine, you tend to think that is the best one regardless of whether its Blekko, Bing, or Google (or even DDG although DDG is more a search utility rather than a search engine as it doesn't have its own web index).

But 'quality' is also a very subjective thing as well. So if you search for highly SEO'd categories you will find the Blekko and Bing do better than Google mostly because there is a curated input (in Blekko's case it was from day one with it's from slashtags and in Bing's they started doing outsourced curation (putatively after seeing how effective it is in Blekko :-) in some topics with their 'editors' program [2]) At some point Google will realize what Bing and Blekko have which are that the 'indexing the web' problem became the 'filtering the web' problem when the signal to noise ratio started decreasing in about 2005, and that the only viable weapon at the moment for human on human spamming (this is where real humans are working for 5 cents a page to write web pages that draw hits) is human judgement.

If the Bing guys are reading (and I know you are) why not open up your challenge to the new kid on the block, we don't just do Blekko vs Google in our monte results :-)

[1] http://techcrunch.com/2011/06/21/be-the-mark-in-blekkos-3-en...

[2] http://www.bing.com/community/site_blogs/b/search/archive/20...

"so I found it amusing that they don't include Blekko"

Just took a look at Blekko. It looks good, but you can't honestly believe the layman cares as much about it as he does about search engines coming out of MS or Google. Does the layman (MS' target audience here) even know about Blekko?

So the back story, and the source of my amusement, was that we've talked quite a bit to Bing. And in those discussions they got to see the positive benefit of some curation, and got to play the 3-card monte game for themselves (and lose a lot :-). And then they went out and did 'editors picks' and now they've done this 'bingiton' site. So its always nice to be driving the conversation around search, and amusing too.

So do I think the 'layman' cares? Yes and no. I think they care that they get good search results, I don't think they care where they come from. As pg points out building a replacement search engine is a Hard Problem(tm). You can make a competitive search engine with just 15 - 20 million dollars, but displacing the existing player is more nuanced.

This is something Bing knows all too well, they effectively created a clone of Google's search capability and they only pull in about 30% of the market. Blekko has been steadily adding share primarily by not being the same as Google, rather we've added specific features that were valued by different segments of the market (we're still the only search engine that lets you completely turn off ads as a user preference for example) but mostly folks just use what ever the search box provides.

[1] Note that both Yahoo and Bing are effectively the same search engine as Yahoo is 'powered by' Bing results. http://www.comscore.com/Press_Events/Press_Releases/2012/6/c...

(we're still the only search engine that lets you completely turn off ads as a user preference for example)

You can do that with DuckDuckGo: http://duckduckgo.com/settings.html > Layout tab > Advertisements.

[I work at DDG.]

That is awesome! I'm a firm believer in giving folks choices in this regard.

Honestly I'm not even a layman; I use DuckDuckGo and Google and read HN (obvs), but I've never even heard of Blekko. Looks nice, but I completely agree: it's a little disingenuous (or perhaps tunnel-visioned) for him to believe that people really know about it.

Heh, perhaps.

DDG is a great product, we are one of their partners and we are providing some of their search results. They are also quite popular on HN I think in part because of the cool things they have been doing about creating interesting one-box kinds of things. But that said, they aren't a search "engine" they are a search "provider." That isn't a bad thing, in fact its a good thing because you can get Google results through DDG without having to get Google's "value added" stuff like G+ friends feeds or being "bubbled" as Gabe is great at pointing out. But the difference is that if the search engines stopped providing results, DDG would stop being able to return them until they built their own index. As would a number of other places like search.com or dogpile.com etc. The weird thing about search is that until Blekko came along and built a third, there were really only two web scale indexes being used, Bing's and Google's. Amazing is it not?

So I think DDG is pretty cool, love the ideas that Gabe comes up with, and am happy to partner with him to provide access to our index.

To rebut your statement that I was being disingenuous however, I suppose it depends on how measure or define 'well known.' From the perspective of various traffic reporting companies blekko.com is a bit more well known than duckduckgo.com which is not a perfect measure either but it does provide a different perspective.

Hmm, we're (DDG) not a search engine? Come on.

Actually we crawl the Web a lot and have a bunch of our own indexes. A more apt description would be a hybrid search engine. More details at http://help.duckduckgo.com/customer/portal/articles/216399

Just to be clear, are you saying you no longer backfill results with Bing?

Also, we don't use any Google results right now.

No, your not. But that isn't a bad thing. From your link:

"we do not expect to be wholly independent from third-parties."

That is the technical difference, search engines do expect to be, if not wholly, then at least materially independent from third-parties, they are the third parties, search providers are all about the experience.

But lets be really clear, that terminology distinction really only matters when you get behind the search bar as far as the world is concerned we're all search engines, even if, like search.com, they don't index anything, and while DDG provides tailored indexes which support key aspects of your site's experience. The differentiation is the experience in that case not where you get your data. And I fully recognize that this distinction is not important for a large chunk of the Internet [1].

For a general audience, yes, we're all search engines.

For a technical audience, no, we're not. And the distinction is whether or not you have a generalized web index or not.

For organic search they present to the end user in a nearly identical way, caveat the 'experience' benefits of one over the other.

But for other things, like "tell me all the sites on the internet that copied this article verbatim." or "give me a rundown on link authority to all inlinks to this page" those kinds of things you need both a web index, and rights to use it like that. That's a pretty objective difference in capability.

So in technical company (and I consider HN to be technical) I try to be crisp about the terminology, in non-technical company I refer to all of these offerings as search engines because it is less confusing and frankly they don't care what its called, they type in words and get results.

[1] My father in law (part of the 99%) thinks he logs into "Google" to get to the internet because Chrome defaults to Google's home page when it starts up.

Awesome, now I know what to call Unscatter. I've been calling it a search site, because I don't currently have my own index of anything. I like search provider better. Thanks!

I first heard about Blekko through HN. I first learned of their API which I used to build most of unscatter.com on through HN too. The 3 card monte the original poster of this thread referred to, once again heard about it on HN. HN has also had links to articles about Blekko's investments from Yandex.

These were all front page. Over all I think Blekko has done a great job of being relevant and covered on HN.

Actually, here's a simple example. 44 pages when you search for Blekko using HNSearch - http://www.hnsearch.com/search#request/all&q=blekko

>and that the only viable weapon at the moment for human on human spamming (this is where real humans are working for 5 cents a page to write web pages that draw hits) is human judgement.

You are aware that Google hires thousands of search engine quality raters, all over the world right? But rather than directly curating, i suspect they just use these humans to train their algorithms. There is likely some inductive step going on, in selecting which algorithm (or 'sort' of the web) is selected, and then filtered by keywords.

Secondly, most search engines, actually do also use end-user curation. I have funky enough search queries, that I often get google redirects as search results: they want to know (and record) which link i pick. Also, because of all the google-adds-cookies, google-analytics-cookies, etc, they likely have 'session-bags' full of destination links and search queries that happen to correlate.

For me personally, neither search engine did very good in this test. But that's likely because neither result set is personalized in this test. And i don't know, if Bing does that now, but even if it does, for me to switch to another search engine, means to train another search engine in my contextual world. Hell, i even notice difference in result sets in Google based on my operating system. I get different results in Linux, than i do on my iPad.

People that get a lot of spam in search results, really ought to login, and stop clicking those links.

I used to be a fan of Blekko until you started spamming my favorite download site with a crappy toolbar. Why sell out so early?

This has come up a couple of times and I'd like to address it.

So one of the things that hinders the adoption/growth of search engines is muscle memory. Sometimes people type into their browser's address bar the address of a search engine, and then search, but more often than not they just click into the search box and start typing. If all the results are 'good enough' people don't bother to check who their search provider is, much less try others. You can manually change your search on Internet Explorer away from Bing to something else, you could manually change your search in Chrome from Google to something else, and you could change the default in the drop down box in Firefox to something other than Google. But if you don't know what you are missing, well you don't do that do you?

Now various sites which have bills to pay and no direct revenue stream, like download sites, offer a service to folks like Blekko which says "We can run an offer to our customers, which if they take us up on it you'll pay us for that." Just like Google gets paid by an advertiser if you click through their advertisement to the site and buy something, these services make your offer for you and then charge you if someone takes you up on it.

So that is pretty standard stuff, the down side is that this space has some pretty scammy operators. We have debated quite a bit about this internally. Our goal has always been to introduce people to Blekko who may not have heard about it (see above :-). However, we've also discovered that sometimes that policing these guys is a pain. They get paid if you install our toolbar (which gives you quick access to our search engine). We insist that they make it an option and clearly spell out how to continue on with your download or what not without choosing to install it, and we insist that the vendor of this software make it straightforward and easy to revert the install if folks aren't happy with it. So that leads to situations where lots of people download software and say 'no thanks' on the offer. Sometimes that annoys the site because they aren't getting paid and they "change things" so that it always installs whether you want it or not. When we find out about that we cut those people off. And if the toolbar doesn't uninstall we cut those folks off too. We had one vendor who gave us a 'final' release to QA where it worked easily and the actual release didn't. That really irritated us. Sadly there isn't the equivalent of the better business bureau for these folks, we discover them, add them to our list of bad actors and move on. We've put a number of changes in place to be better at managing the process and it may be that it ends up there are just too many scumbags in this space to make it a credible channel. Time will tell on that, in the meantime if you ever spot a site telling you that you must install a Blekko powered toolbar in order to proceed (which is to say its not completely obvious how to opt out) please report it so support@blekko.com and we'll fix it.

That's noble and all, but you know what, it's still a toolbar. If I see it in an installer it's still going to put a bad taste in my mouth. You know as well as I do that 95% of those installs are not intentional, or are manipulated. The kinds of users that install toolbars are not computer savvy, and when the computer says "you should install this thing", they're the ones to say "accept".

Blekko is pretty awesome. I'll admit. I've found stuff on Blekko that I doubt I'd ever find on Google.

To Everyone: Try it out. Blekko's results are a breath of fresh air.

Actually we (DDG) do have our own indexes (including web links). More details here: http://help.duckduckgo.com/customer/portal/articles/216399

This is a great summary from that link Gabe:

"we've focused on building a better search engine by concentrating on what we think are long-term value-adds -- having way more instant answers, way less spam, real privacy and a better overall search experience."

Now I'd probably word it 'building a better search experience' and remove the duplicate at the end but that's just word-smithing. One of reason we love you guys is that we share the same values in this regard. The more 'engine-like' you get away from 'experience-like' the more dubious Google and Bing get with sharing their index with you :-)

And, like you guys, we like to partner with folks who can provide the 'long tail' as it were.

"or even DDG although DDG is more a search utility rather than a search engine as it doesn't have its own web index."

Looks like you might be wrong about that.

DuckDuckGo gets its results from over 50 sources, including DuckDuckBot (our own crawler), crowd-sourced sites (in our own index), Yahoo! (through BOSS), embed.ly, WolframAlpha, EntireWeb, Bing, and Blekko.

http://help.duckduckgo.com/customer/portal/articles/216399-s...

As a Google shareholder, I really want to know whether Bing search results are better than Google. And I really, really want to know what the trend is over time.

However I don't think Microsoft's study is any good. They say, "In the test, participants were shown the main web search results pane of both Bing and Google for 10 search queries of their choice."

So it seems that the study participant had to think of 10 search queries, one after the other. But this is not how search works. In the real world users need to perform one query every so often, and their choice of search terms is based on an immediate need rather than what comes to mind.

A good Bing vs Google study would ask 1000 people to use a split-test search engine for their browser search box, then monitor the results over time. I've asked HN about this before and got nowhere: http://news.ycombinator.com/item?id=3966121

So is there business opportunity here? I imagine that there must be thousands of Google shareholders willing to pay £100 per year for the results of an unbiased, ongoing study into who gives the best search results. In fact, why doesn't Google set up the study and provide such information to shareholders?

You can keep your shares. I picked Google 5 times out of 5.

The only person who would do such a study would be an analyst for a private investment firm.

Google, 4 to 1.

Now, all of my searches were technical; I plan to try again later with more general, "normal people" searches. Bing may do better there.

Also, good results from even google searches have become harder to come by in the last year or so. I have to work harder at my search terms. I don't know if that's Sturgeon's Law at work, or if there's room for a search engine to improve enough to become better than google.

Perhaps targeted search engines are the answer (i.e. engines's who have both their search and crawler algorithms tuned for a specific target audience) are the answer?

This is the thing for me.

Of course it's going to be a draw for most searches (especially the suggested ones: "math games", "coupons", "time zones"...), they're generic and easy to find.

You can use any search engine you want to find those things. Shoot, even AltaVista was good at those kinds of searches, and that was 10 years ago.

The ones I care about—even if I'm not a programmer or technical user—are the difficult searches. What if I need to find a specific car part that only ever existed in the 1972 Ford Pinto? What if I want to find a plumber who specializes in pre-20th-century homes? What if, what if, what if.

I used searches like that and more in this comparison, and it was 100% Google. I could even tell easily why it was Google, and it was pretty clear which one it was. I still chose Google (despite being basically unblinded) because it was fundamentally better at these specific searches.

That's what bing needs to compare. But will they show it? Of course not.

Edit: I did 5 more searches all non-technical but specific and picked only based on result quality. Again 5/5 google. (for those curious, the searches were: "lego mindstorms", "Mars curiosity hi-res", "home developments in myrtle beach", "used bookstores in portland, or", "top baby names in 1982"—the last one was very cool because Google also provided links to the 5 surrounding years right below the 1982 result. It's the little things you know...)

I went 4 to 1 google as well. My searches weren't technical, my name, my job, my hobbies, etc.

I got 3 for Google. The results are surprisingly similar to pick a winner. Probably because I chose some easy searches (Mr. President, person name, place).

Google, 3 to 1.

All searches were related to pages I know and I searched for some keywords or parts of the domain.

Google, 2 to 0

Here's what I searched for -

-my name (Google, I chose Google because it gave more of my results, which may not be fair.)

-extract articles from websites (Google, in this search google's result were much better.)

-python ide for mac (Draw)

-new york time (Draw)

-glusterfs (Draw)

I did a mixture. Google won technical searches easily. Bing marginally better on the others

I've noticed this as well. The closer my searches are to what (I imagine) a typical Google employee does every day (Linux/Python/Java/HTML/HTTP/CSS, English language, American locale, someone in their 20's/30's, etc.), the better it does. Google search can be incredibly good at taking a partial Linux error message, and leading directly to a solution.

The further my search is from this, the worse the results I get. I'd say this is a common theme throughout Google's software. Google Wave, for example, seemed to be aimed squarely at the "I'm a Google engineer" use case. (I assume not many Googlers ride the bus, because that's a weak point of their maps!) The Chrome commercials they're running right now are a great example: they look like something straight out of the old demoscene, that Google engineers might do in their spare time, not something that I could imagine a use for.

I got Bing with 2-0 with 4 draws. I used 1 technical keyword and 1 "normal people" keyword in spanish. It was really tied though, most results were different but pretty relevant, very hard for me to pick a winner.

Google 5:0

3-2 Google. 3 technical searches all went to Google. 2 popular culture searches went to Bing. Interesting.

Google 5 to 0

Maybe I was imagining it, but four out of the five I did I felt I knew which was Google and which was Bing - not from doing searches that I would recognise, but just from the extra non-search information / lay-out.

I did my best to still pick without bias to my current search engine (Google) but ended up picking them 5/5 and Bing 0 times.

The site breadcrumbs using ">" always give away Google unfortunately.

So does the lighter blue of the bing links.

And the collection of similar pages when the result is a forum.

Same here - ignoring that I could sometimes tell which one was Google and just looking at the returned links, Google won by 4 and a draw.

Ditto. The layout of the two's result pages are so different now, it's quite easy to tell them apart.

I have actually found myself using bing lately only because it doesn't try to confuse me into landing on google+.

Seriously, guys; this is a problem. I'm searching for a coffee shop or restaurant or something, and I want to link my friends to the map.

I --DO NOT-- want to go to google+. In fact, I don't really want to google+ ever. For anything. Ever.

Seriously I feel like I'm navigating a maze of accidental google+ links every time I use google anymore. It's really frustrating. Bing at least seems to be just...a search engine.

In fact, google hiccuped all over my 8 years old gmail account, deleting my inbox (yay! Thanks for that guys, oh, and it's just lost, too bad for me!) two days ago.

The upside to this? It somehow also unsubbed me from google+!

I find the Google+ spam annoying too, but my pet peeve is the redirects Google put on search results so they can tell which ones you click on. I hate this, first because it feels invasive and second because it noticeably slows things down. Most of the time it's not that bad, but every now and then it adds a second, or 2 seconds, or 5. I got so mad I astonished myself by switching to Bing for a while. And indeed the Bing experience was much faster. I've switched back to Google for now, mostly out of habit. But I've gradually gone from being a big fan of theirs to a grudging captive. Something could jolt me into a different orbit.

There are extensions to get rid of the redirect: https://chrome.google.com/webstore/detail/dohbiijnjeiejifbgf...

Thanks! I'm going to give that a try.

Edit: holy shit it works! You just made my internet faster. How did I, or rather everyone, not know about this?

The weird thing is they could use JS to fire an analytics event to avoid the redirect.

That's what they used to do. This being Google, I'm sure they had a good reason for switching, especially to something so corrosive of user goodwill. I've often wondered what that reason is. Anybody know?

Other than the logo, and the bar at the top that says "Join Google+" (which of course, may be a slight annoyance to some people), what difference do you find between the old Maps/Places page for a restaurant and the new "Google+ Local" restaurant page that you're afraid of being confused into landing on? I just checked a local pizza joint while not signed in to Google and those two things are the only things that aren't reviews and information about the place (i.e. the only differences of substance between the new pages and the old Place pages).

This entire page is just...cluttered.

https://plus.google.com/107551444336725915386/about?gl=us...

It reminds me a lot of myspace, actually.

Okay, instead of whining about this, I'm going to look into a maps API client :)

Well if your only complaint is the design of the page (which is a fair complaint, a poor design can def. be annoying), then the whole "stop trying to confuse me into using Google+ when I just want to use Places" of your original comment is sort of a red herring, isn't it? Aside from having the name "Google+ Local", it's essentially equivalent to having the Places page undergo a visual redesign which you dislike.

Google deleted your inbox? Could you elaborate on that?

Sure. I logged into gmail.com yesterday morning, and my inbox was missing. I still had everything that "skips the inbox", like mailing lists, my sent mail was all there, but everything that was in by inbox had vanished.

This actually made me pretty sad.

That is sad. Do you use POP or IMAP with GMail? And do you see anything weird in the "Last Account Activity" link at the bottom of the page?

Also, were the messages merely archived (i.e., moved from Inbox to All Mail), or were they completely deleted (i.e., you can't even find them via the search box)?

I do use an IMAP client (thunderbird) with gmail.

No, no strange activity in that window. I see myself coming out of Black Rock City, and the sparse signal I was getting on Monday as a result of it, but other than that it's just my office.

This actually sounds more like your account got hijacked, or the screen was left up and someone deleted the inbox messages. Try this process to recover messages from the trash: http://support.google.com/mail/bin/answer.py?hl=en&answe...

Can you check for unusual forwarding rules or delegation in settings? As a follow-up, do you use two-factor authentication on this Gmail account?

Also, check your

I didn't use two factor before, but am now. The password I was using was unique, though, and also fairly complex, so I find it unlikely somebody brute forced it.

I tried the link you're suggesting, as well as looking through the trash etc. What seemed very odd to me was that the messages which started in my inbox were missing from he trash, but things that were "skip the inbox" weren't.

Meaning anything I had "trashed" before yesterday was intact.

Again this seems to suggest that the account wasn't compromised, but that I fell victim to some sort of bug. It would have to be a very strange/determined hacker to go through and move my inbox to trash, then go to trash and only remove messages that started in the inbox.

Regarding the messages that used to be in your inbox -- are they gone gone, like you can't even search for them, even with the in:anywhere flag that makes GMail check Spam and Trash?

OH! in:anywhere seems to be finding them!!! Raldi! Thank you!

I searched what I /thought/ was EVERYWHERE yesterday. Thank you thank you thank you!

Is it possible that you accidentally typed *a# into a GMail window? Those are the keyboard shortcuts for "select all" and "move to trash".

Fear of doing that sort of thing has led me to disable GMail keyboard shortcuts altogether on my account. (To do that, press ? in a GMail window and click the link on the second line of the overlay window.)

Very strange; I've never heard of a Gmail issue like this.

Matt, they seem to have re-appeared in the trash!

Yesterday was some very odd behavior, so I must have missed them in the morning.

Basically:

1) I logged into gmail, and saw only one email that I had gotten that morning, everything else was gone. I looked in trash, and the things in it were from over a week ago [I've been out of town at Burning Man for the last week], so things hadn't just been moved to trash.

2) Did the "recover deleted messages" request via google, and a few hours got a response that some things should be restored to "All Mail", but still didn't see much.

3) Now they're back!

This is great! :)

Woohoo! Glad that you were able to recover them!

I added plus.google.com to the list of sites blocked from search results shortly after they started spamming them. Too many results were polluted with google+ above ones I was actually looking for.

Are you talking about the results with little people icons, Search Plus Your World, that match pages in your circles? You can turn off that feature in settings, or at least you once could.

Google 4, Draw 1, Bing 0.

Why? Because I tested it with fairly specific queries. A while ago I tried to switch to Bing because after testing it with a few general searches, it seemed just as good. However, when I switched and started searching for compiler errors, etc - I soon switched back because the results were just nowhere near as good.

Here's what I searched for:

    bash: !": event not found error
    gtk calendar tutorial
    python import gtk error
    seed funding in uk

Something to remember by the way is that Google has spent a lot of time and effort on their "bubble", that is, search results personalisation (http://dontbubble.us) and it is a big part of what makes their results so great. That influence won't be present in this test.

Ended up picking google 4-1. I'm somewhat of a bing fan in the sense that I find them good enough for most searches. So I was a bit surprised that the results ended up skewing in google's favor ... guess that kind of backfired for them :-P

5-0 in favor of Google for me. 3 of my queries had 5 words, one had 4, and one had 2. Bing is almost as good at solving the easy short ones, while Google can solve even the long queries.

I got a draw and then a 5-0 for Google. A couple of result for Bing were weird, like displaying "Seattle" when I searched for "a" (the letter), NSFW images when I was searching for a novel ("accelerando"). Google knowledge graph it's just too much ahead ("age of mcgyver") and it has better results on maps and evaluating expressions.

Bing seems to be a bit better at generic, broad searches (games, file extension $something, etc) and Google seems to be better at exact, specific queries (stop error 7B, RPGs announced at PAX 2012).

Bing ended up winning, which shocked the hell out of me, but after trying a couple more times, I noticed that pattern.

And at the end of the day, what I need are exact/specific queries.

Whether to get the job done, to get some certain information or something third.

DuckDuckGo is now a hundred times better than Google for those exact & specific queries. I was looking for some info on my truck engine, and Google sent me to sales pages for completely unrelated parts (and sometimes, entirely unrelated cars!). DDG sent me to deep in an old forum thread, where someone had posted a diagram of exactly what I was looking for.

Every time I see a link comparing search engines, I try them with hope, but always go back to Google. My last search confirms this :

* Google https://www.google.com/search?q=insertObject+atIndex+zero+bu... * http://www.bing.com/search?q=insertObject+atIndex+zero+bug&#... * http://duckduckgo.com/?q=insertObject+atIndex+zero+bug

So, five results at the top from StackOverflow... Why not just skip the search engine middle man and go straight to StackOverflow?

Proper search engines are better at finding content than a site's own search feature.

I usually type in my browser address bar, which searches to Google directly. I tried my query on Stack Overflow... Google still gets a better SO question than SO itself : http://stackoverflow.com/search?q=insertObject+atIndex+zero+...

Because the goal is to have something that finds any particular information I want, not go to every site and hope it may show up in a site search?

Because you still don't know if StackOverflow will have the result you're looking for

I've seen this in shopping results -- I'm probably going to buy from Amazon, but I like to see the top N competitors' results along side the Amazon results, so that I can easily confirm that no one is more than a buck or two cheaper than Amazon and can then feel good about my purchase.

I did a comparison by actually searching the same phrases on google.com/ncr with Incognito. The actual Google results are much better than the Google results on this page.

Search for "Hunger Games" on Google returns the IMDB page as the first result, which is exactly what I want. But neither panes on the Bing's test has the IMDB page as the first result.

5-0 for Google. I tried mostly programming related queries as I find the differences most noticeable there. Especially when searching for specific APIs or objects, Bing tends to give you just the project homepage and some unrelated pages, while Google gives you the actual deep link.

I remember a while back when Bing was shown pretty conclusively to be taking Google results.

So I wonder if this test is really "Google vs. Bing + Google".

EDIT: Source - http://news.ycombinator.com/item?id=2165469

Last time I checked, it looked like Bing was still using clicks on Google search results as a signal in Bing's rankings.

Matt - do Google's terms of use allow users to send the results page clickstream to Microsoft? If not, aren't there legal avenues you could pursue?

I don't think we need any more litigation and definitely not in search. Google search wins by pure accuracy of their results. They should be proud of it and just ignore scavengers.

How does msft have access to google algo click data?

IE and Windows, I believe.

Interesting. The privacy implications are disturbing if true. I'm neither an IE nor windows user but I suspect most users wouldn't opt-in if given that choice. Any refs to research/evidence about the topic would be appreciated.

The IE privacy policy is at http://windows.microsoft.com/en-US/internet-explorer/product... . Look under the section titled "Suggested Sites." Here's the relevant section:

"When Suggested Sites is turned on, the addresses of websites you visit are sent to Microsoft, together with standard computer information. ... Information associated with the web address, such as search terms or data you entered in forms might be included. For example, if you visited the Microsoft.com search website at http://search.microsoft.com and entered "Seattle" as the search term, the full address http://search.microsoft.com/results.aspx?q=Seattle&qsc0=... will be sent."

Most people have little idea that allowing a feature called "Suggested Sites" will result in their Google searches and clicks being sent to Microsoft, or that Microsoft will use clicks on Google search results in Bing's ranking.

MSFT also uses something called the Microsoft CEIP (Customer Experience Improvement Program), and I think that's either opt-out already or they're making it opt-out in Windows 8--it's built into the "Use Express Settings," I believe.

Again, I haven't looked at this very recently, but if you're using a recent version of Windows and IE, you're probably sending your searches and clicks to Microsoft unless you've been very careful about how you configured your computer.

Agreed about "most people have little idea...". I wish more companies were more honest and open about these things instead of burying it in the fine print. I wonder what would happen if you enable "Suggested Sites" but then install IE 9+ (since it appears to support the DNT flag) and select "Do Not Track". The pessimist in me suspects it will just turn off tracking for all non-msft experiments.

In any case, I think you've convinced me that my decision to use a non-windows OS and a non-IE browser was the correct one.

Thanks for mentioning this...

For the longest time, I didn't even realize that clicking a Google result sent the click back to Google (I just didn't think about it).

Then I right clicked one of the results one day and did "copy link" since I wanted to send it to a friend, and got something like this (with a bunch of other stuff in it too that I've yanked out cause I have no idea if any of it would be identifying info, but the end result is that this link no longer works):

http://www.google.com/url?&source=web&url=http%3A%2F...

I just hadn't thought about it before. It makes sense that they want and use that info, but it wasn't obvious at all that they were collecting it in that way, since the address shown in the status bar is the destination site, not the actual link target. I had just always assumed that the results were based on links and such around the web, and didn't think clicks would be factored in.

To answer my own question: http://www.bing.com/community/site_blogs/b/search/archive/20...

Seems it is opt-in users.

The very fact that this site exist says something. To me there's a weird feeling of desperation about it.

Not at all, it's an often used marketing technique, generally by an underdog (even if only a slight underdog) to try and pull in customers who would actually like their product better if they had tried it.

The most famous example would be The Pepsi Challenge [1], but it happens all the time, for example AMD have done it at a few events comparing an AMD machine to a similar spec/price Intel build.

[1] http://en.wikipedia.org/wiki/Pepsi_Challenge

There is nothing "weird" about blind "taste" tests. It's a good way to attempt to filter out at least a little bias, and here you have two competing products that do the same thing (unless your baseline assumption is that no one could ever compete with the current leader).

Seems like a pretty reasonable thing to do if you're the underdog even though you believe your product is better on its merits.

BTW it is best to use Incognito mode.

In my case Bing won 1 out 5 of the rounds. I used searches I had done recently that hadn't given satisfactory results on Google.

In any event Microsoft now at least has searches it can work with and improve. I can't imagine any other way they could get Google users to provide search data.

The study results are not very accurate. They made it with only 1000 US random users, and ~60% chose Bing more often, ~30% chose Google more often (on 10 queries).

See the following fictive example:

1000 people, 600 have the following score: 4 Bing, 3 Google, 3 Draw; 400 have the following score: 10 Google. So, in my example, 60% of the people chose Bing more often and 40% chose Google more often. But, if you look at queries numbers, Google was chosen for 3600+10400=5800 queries, and Bing for 4*600=2400 queries, that's a 2:1 ratio, but in favor of Google.

oops, the two '*' characters were interpreted as Markdown. So, the operation is “3×600+10×400=5800”.

>Bing services aren't available right now

After 5 searches the screen greyed out and was unresponsive.

Guess Google wins by disqualification?

Same here.

Much easier to use blekko's 3-engine monte, since it strips formatting that might give away the source engine. Just add /monte at the end of a query:

http://blekko.com/ws/hunger+games+/monte

>We're Sorry, Google has banned us from retrieving results from them for now

picked google

This is not a real blind test though. (Pardon me for being a web developer, but) I can tell which is which by just looking at the green colors used in the links. The green in Google results is a little bit more saturated than the one in Bing.

Google results also seems to show share counts from Google+ as in: 73,352 people +1'd this

1. survey is organized by Microsoft, 2. I think the survey is skewed because typically when you go there you don't have anything specific in mind to search so you type one-word query (they even propose them!) and one-word queries are hard for google

I had two different runs that both ended 3-2 for Google. Even if Bing came out ahead, they seem so similar that I couldn't be bothered to retrain myself to bing stuff, instead of googling it, especially when I have gmail accounts.

Nice try Microsoft. But if you really need to create a contest to prove your search engine is better you're probably desperate.

Sure, it's a PR campaign, but it's built around proving to you the actual benefits of using the product. Seems pretty clever to me.

You would rather they blanket the city in ads on bus shelters? Or give away $1,000 a day to a random Bing searcher?

It was clever when Pepsi did it to try to let people know that they couldn't tell the difference between pepsi and coke.

Now? Obvious.

The 'pepsi challenge' of search engines?

Tip: try to search for plot lines in movies such as: 'kid that is able to see people already dead'

I was Google 5-0 but with technical queries whose results i did not know for either bing or google but knew what pages should be on top. I think it is actually deceiving. While choosing I was pretty sure two of them were bing. Because one's default address for movie timings was Redmond and in other the video results were all non-youtube. Something does not make sense. Either google is ignoring youtube results and moved to Redmond or they (MSFT) changed it so that people choose other result. (not many people would get the Redmond reference though)

It may be that Google uses location information from your IP address, which, for the Microsoft servers running the competition, is in Redmond. If you search "what is my ip address" in the competition, Google's results show "Your public IP address is 131.253.25.46" (or something similar). This is a Microsoft IP in Redmond. Interestingly, a reverse DNS query gives msnbot-131-253-25-46.search.msn.com as a domain name for this IP.

Well yes that does make a lot of sense. Kind of stupid of me to not realize this myself.

5/5 Google. Two technical queries, three more average ones.

But I could tell which side had which search results in 2 or 3 of the tests, so I can't claim complete blindness in the test. I tried to be impartial, though.

Bing you fool but not we :) it is definite we use google search engine everyday and what search term we will use in this contest is more likely the term we already use to search frequently let say "Design" and almost know the top ten search results of google. So when bing produces couple of different search results people will give definitely give first 2 or 3 clicks to bing just for the sake of change but for long term user most important and refined results belong to Google we know :D

I wound up with 3/1/1 in favor of Bing. I sort of felt like I was choosing the Bing options, but Google still hasn't given me a real reason to switch.

MS has definitely caught up quite a bit, though.

Google 5-0

Google's strength is not in searching for head or torso terms like "facebook" or "2013 calendar". It is in searching out tail terms; those obscure terms which might include a typo or two, and the phrasing may not be correct.

Unfortunately (and I don't have much time to explain this) one simply _can not_ judge a search engine's comparative performance using random users. And the Bing guys are falling into that trap.

EDIT: To expand more on what I wanted to write.

// DISCLAIMER: This is my personal view, and has nothing to do with my employer.

There are 3 large (fuzzy) classes of queries on search engines: head, torso and tail. The head queries are small in number but large in volume: queries like "Facebook", "Google", "Yahoo", etc. (yes, people do 'search' for these on search engines. Just a handful of such queries may make up 10% of the entire search engine traffic, which would be 100M queries per day (or so) on Google. It is depressing to see the numbers). In general, any search engine worth its salt will get the head queries right.

The torso is basically deciles 2-6 or so (there's no hard and fast definition). Here you'd have most musicians, actors, popular restaurants, etc. Bing and Google both would do a mostly adequate job here, for the first result. But: if you want to explore the concept a little, Bing goes astray and Google does not. For example, search for "idle hand tattoo" on http://bingiton.com/ . The first result is the same, to my local tattoo parlor. But the second result on Bing is from Mesa, AZ; while Google sticks with the right result and offers up other, related sites (Yelp, etc.). Google knows that "Hand" in the title is not the same as "Hands", and gives me not only the store, but also their Yelp, FB and Twitter pages; in other words: Google _understands_ your query better.

And finally, the tail. Here we have obscure error codes, weird technical terms, etc.; most of which are seen maybe once or twice a day. These constitute the bottom ~30% of the query stream. This is where Google really shines. It is very rare indeed to look for an obscure term and find that Bing does a better job than Google (I have yet to find an example).

So, as a user, I have a choice of 2 search engines; and one of them I know will do a better job on obscure terms. Guess which one will I pick?

When Bing claims that "most people prefer Bing in our user studies" (or something similar), their conclusion is flawed because the study itself is flawed. You can't sit a person down in front of a computer and ask him to evaluate your performance on tail queries! Where are those tail queries coming from? If they are provided by Bing ("here's an obscure term, search for it and tell us what you think"), then the user has no way to evaluate how relevant the results are, because, by definition, it's an obscure term! On the other hand, if you ask the user to come up with an obscure term, it will most likely be a term they are familiar with and they'll already know the answer, so they won't really hunt for information. So in my "idle hand tattoo" example, they'll see the top result and claim satisfaction; when, in real life, I would like to see the Yelp reviews, maps, etc. for the place.

If they (Bing) really want to compare how well they're doing, here's a suggestion: setup a search engine (like bingiton, but with only 1 pane of results, and neutral branding), and make it the default for a large pool of users (with their permission). Randomly, switch the backend to Bing or Google. Then, monitor the heck out of what the users do (all with their permission): how often do they click on the first result, how often do they re-formulate queries, the time to the first click, how often do they quickly come back to the results page from a bad click, etc. etc. etc.

And this is why I use Google. I rarely need to search for "facebook"; I do have a brain. When I'm searching, its more often than not for something I can describe but don't know the name of, and Google blows Bing out of the water on these. Try describing a movie to see what I'm talking about.

Absolutely agreed. I was also 5-0 Google. Examples: "mdc date" "python math" Google is the clear winner here, as well as many other scenarios where more specific searches yield incredibly higher quality results.

It doesn't surprise me at all that Google tends to handle the long tail questions better. --They're much larger and have had many more years to collect data, build synonym dictionaries, etc.

When I tried to come up with obscure queries, Google does better at the technical queries, and Bing is doing better at the non-technical queries. But the technical queries usually weren't that far off.

Yes, they're much larger, bot not much older. Microsoft was in search business years before Bing, and I also think they now have Yahoo!'s dataset too.

They're just better in search. I generally dislike Google (because I don't like their business model, which is advertising [I don't care for privacy much, just hate ads]), but nothing is as good as them in search, nor could be (for the foreseeable future).

Give http://www.millionshort.com a try for long tail. Set filter to exlude top million sites.

This comment is quite late. Hopefully somebody at Microsoft will see this.

I used Bing for several months. I actually found it indistinguishable from Google except for linux howto questions (Google seems to understand forum websites better).

The dealbreaker for me was lack of SSL support. Switched back to Google.

Nobody who uses Google regularly will switch to Bing: not enough of an advantage. But people who start out with Bing may never switch. That's the long-term game Microsoft has to play.

I ended up with a draw the two times I tried. I made the mistake of trying a Google search I had just done previously and ended up seeing the purple links, which tipped me off to which was which. Bing was the only one that gave me local and mapped results for "Chicago Loop pizza", but Google won out just as often.

Bing is as good as Google on average. The problem is, when you're up against an entrenched competitor, good enough just isn't good enough.

3/2 in favor of Google. I was surprised by how similar the results were, and by what I apparently considered important about search results.

I have this theory Microsoft keeps funding bing as a tax writeoff.

I've yet to see a website where the search referrals are even within 30 points of Google.

"we can make him better we have the technology"

How about we build a community test? We could make a little page showing the results for various search engines (e.g. google, bing, blekko, duckduckgo) and and put them to a similar test. We could open and show the data collected in real time so the community gains from the experience.

Immediately there is a problem: how to get the results.

Naively we could use iframes to show the pages for each results. But it is not possible.

We could use their APIs to get the results and show them. However these are limited to N calls on their free package. That would make the survey limited.

Another way is to screenshot the page (fetched using a simple browser call) and present it. This, however, should also be paid for more than N calls..

Anyway, after resolving this, we should be go to go.

We could gather data in an unbiased way and present it to the world.

This is simple to make, I think. Upvotes if you think it is a good idea (for this to be a join effort probably).

I selected one draw and then Google every other time. It was close for most searches but Google was slightly better. Google seems to be better at more detailed searches whereas Bing is better at searching for broader things (like they suggest 'wedding dress', 'chicken nuggets' etc.).

4 for Google and a draw. Google is still way ahead using my mother language (which is not English).

I found exactly the same thing. Searching in English gave me mostly draws. Searching in Swedish and Google crushed Bing at least 4-1 every time.

3-1 in favour of google for me.

I suspect a lot of this may be down to how challenging the queries are that people put in. Bing's historically been great at generic searches, but I made a point of looking for more difficult stuff, where google has (for me) always been better.

I found that I was not able to take the test fairly, because it was blatantly obvious from the style of the search results which one was Bing and which one was Google. The Bing results use a slightly duller green for the URL displayed underneath each result's title, among other stylistic differences.

So, the comparison is completely useless, as they did not anonymize the style of the results, and thus either conscious or unconscious bias can creep in. I expect that for users that are not aware that the results look different, there will be a significant preference for the results that they see every day (e.g. the engine they normally use).

I got Google 5 - Bing 0. I did 3 math related searches, 1 math graduate student by name and 1 search about a python math library. Looks like I'll only switch to Bing if I leave math, which is not very likely. ;)

5-0 in favour of google. I was quite surprised too since I used fairly generic queries that I thought Bing would do as well as google. - whats on in london - ready player one - bash history across sessions

Anyone else seeing different results on this vs. what Blekko's monte (e.g. http://blekko.com/ws/your+phrase+/monte) produces?

Google 4, Draw 1. Did not pick Bing results once. Methinks their "blind trial" only applies to queries Microsoft specifically gave the testers to make Bing results shine.

Using five technical searches I might actually make on a day by day basis, I chose Google 4 times and a draw once, though the results were close almost every time (once I chose Google because I searched for an organization name and wanted their homepage first rather than relevant news, and twice because the relevant result was from MDN rather than MSDN or W3C since I found that MDN tends to be the best site for actual technical reference, so YMMV).

I am surprised how close the results are in my case. It was often really hard to say this one or this one. Near "draw" for most of them. Trying the Blekko "/monte" way as proposed here it was 50/50 Bing/Google (a bit more than 5 tests).

I am not going to switch yet, but this is good news, the gap is closing and maybe, like we had an IE monopoly, we are going to move slowly in direction of more search engines.

Google for 4, drawn on 1. I like that the blurb at the end was all "whatever, we're still better says a survey". Marketing failure ahoy!

Google 3:2, although the problem with this is that I can always tell which one was google and which was bing based on the formatting.

Apparently like most other people here who tried it, I found that Google won conclusively for me (4.5-0.5), though in each test it won by only a small margin.

My queries: one software-technical python virtual machine, one philosophical argument from evil, one highbrow musical/artistic (this one was the draw), one engineering-technical, one geeky-popular cultural.

Given the results of this group (I was 3 for google, 2 draw), I wonder if Google simply has a stronger technical bias than Bing.

Google always worked for me whatever the result is. Their default queries are just ridiculous and cherry-picked.

Something is wrong, because I chose 'draw' twice, but apparently I prefered bing over google 4 times out of 5.

The CSS gives it away. Bing has slightly different coloured links so you can see it quite obviously.

I'm the obvious exception who's skewing the average but Bing won 4-1. I found their color palette much easier on my eyes, and when half the results were "draw," I sided with Bing, simply because I would prefer the presentation of their results.

Appears to be broken in Chrome on OS X. After my 5th choice I'm stuck on a gray screen.

Broken for me, after choosing first round winner both panels get darker and nothing happens...

FF15

Google 4-1. No surprises here.

5-0 for Google.

This was made by Microsoft??

I did all JavaScript searches and knew which were Bing. Do a search for: array.prototype.filter. The Google result gives you MDN, the Bing result gives you MSDN.

Google 4-1. Backfire

"Bing won 1 rounds"

What bothers me most about this is that the programmers didn't bother to remove the S from "rounds" when the result was singular... sloppy.

Google 4, Bing 1, but I mostly use DuckDuckGo. But Google is unmatched for geographic terms, so I override with !gmaps

they both suck -- for 3 out 5 queries I tried they both return spammy junk or non-helpful shopping links

Man, I completely agree. Both Google and Bing are completely and utterly broken. Both return about.com and ask.yahoo.com and a number of other sites that have been specifically built to game the system. Does anyone else remember when you could search Google and it would return academic institutions and actual interesting blog posts from individuals? It has been a long time.

When your system can be gamed, it is broken. Period.

3-2 Bing. The results can be easily linked to one of the search engines so the outcome can be skewed.

I've noticed, 1st result displayed on their search didn't match mine. Nice try, M$

Google 4-1 for me

2 Draws, 1 Google 2 Bings for me.

the 2-way layout reminds me when I wanted to do the same thing, and combine search result into one (of course removing dups). Ended up with making google gui into bing results of now defunct binngle.com (due to MS charging for their api access)

I would like to see "live" results to help corroborate their claim.

The only one Bing one at was when they suppressed Google Shopping but showed Bing Shopping. I get that they think it is fair to hide ads, but Google ads are highly competitive with organic Bing.

They also suppressed the Knowledge Graph box.

Reply on news.ycombinator.com