Compare & vote for search results of Bing, Google and Yahoo in a blind test | Hacker News Reader

Compare & vote for search results of Bing, Google and Yahoo in a blind test

blindsearch.fejus.com

123 pointsvaksel41 comments

1. This was created by a Microsoft employee (a "Microsoft Developer Evangelist"). Maybe this doesn't matter to you, but considering the $100m marketing budget and the widespread use of astroturfing, it seems a little dishonest to me (certainly treading in a grey area) that there isn't full disclosure. YMMV

2. As someone who cares deeply about research methods, I'd argue that there is little to no value in superficial, subjective and contextless comparisons like this.

Edit: He has now added a statement that he works for microsoft. It was not there before I posted this comment

Well if you question the methods, you can always open bing/google/yahoo and check to make sure the column labels are correct

I didn't say, hint or imply that it was listing inaccurate results. I stated that it's superficial, subjective and devoid of context. Frankly, it means nothing.

For example, someone searches for "bongo". The forth results are a radio station, a page about the animal and an email/calendaring system. Given those results, someone votes for one. Why? We don't know. They might barely now. One thing we do know: it was a one-off search anyway, not something that was part of someone's normal workflow.

the context seems pretty obvious, to let people compare the different search engines without being biased that they are using a search engine other than Google

You are misusing the word "context" - http://www.answers.com/context : 1. The part of a text or statement that surrounds a particular word or passage and determines its meaning. 2. The circumstances in which an event occurs; a setting.

The searches here are without context, the rating of results are without context. All we know is that on arbitrary, one-off searches, a set of people who may or may not be representative of anything clicked the button above a certain set of results for some unknown reason.

And your basis for asserting 'arbitrary and one-off searches' is...?

Because it's a obscure micro app for doing one-off comparison searches that's making the rounds on social sites.

Your argument seems circular, to say the least.

They're comparing search results, not search engines.

Results matter, but so do speed, usability, features, privacy, etc.

I strongly disagree. Incidentally, while this may be new since your first message, there is disclosure on the landing page which I found quite adequate. I notice Google leads by a significant margin. Is Bing looking better than it really is? Who knows, but while I don't plan to switch away from Google, Bing is OK by me - the quality of results are pretty good and its fast. Yahoo not so much (and I got no Yahoo results to my tests at this site) but I'd given up on Yahoo search before Google came along.

You assume it's a one-off search and that the users are simply licking randomly on whatever they see that's immediately interesting. I see no justification for this assumption. Personally, I did what I did with any new search engine: perform 10 searches on topics ranging from topical to deeply obscure, and look at the first 5 results for each search to evaluate the quality of both the returned sites and the text clips for each.

If this was the subject of a paper submitted to the journal of information processing, you'd have a valid point. But as a 'what happens if...' experiment on the web, I'm just fine with it.

<i>2. As someone who cares deeply about research methods, I'd argue that there is little to no value in superficial, subjective and contextless comparisons like this.</i>

I care deeply about research methods too, but I don't understand why this is bad. If the goal is to know which search engine is best -- a subjective question in the first place -- it's gathering judgments from people on exactly that topic.

Of course, you can't take the statistics collected overly seriously, since it's hard to know how seriously different people are taking it and such. But it's probably more true than not.

I tried 5 searches about subjects I knew well. Results: Google 4, Yahoo 1, Bing 0. But in every case I had to look closely at the results to decide which I preferred. None of them returned obviously bad results for any of the queries.

You should share search text so all can appreciate your results.

Do you usually search for information on subjects you know well or subjects you don't know well?

One needs to use any of these search engines over a period of time before making a switch, mostly because the improvements are incremental in Bing.

I searched for Patti Blogovich (intentionally misspelled). All three columns contained what appeared to be relevant results, but after reading the returned articles, only two articles were outstanding (on depth and "objectivity"). One of these was halfway down the column for what turned out to be Google. The other was the top result for Yahoo.

Then I searched for "wriggly wrolly," which I sometimes call my dog. I ended up clicking for Yahoo again, which had all 4 results.

Sure, only two tries, but I was really surprised with how rewarding this experience was. I wish I could use Yahoo but have it look like Google. I'm just so hooked on Google's branding and image, rather than my 90's-based judgments of Yahoo...maybe I can be stronger of that and make the switch...maybe.

This application is weird in cases where spell correction comes in to place. Just search for mike tieson, birack obam, pal graham hacker news and you can see the difference between google results & what's shown on the blindsearch application. There's a whole lot of difference.

The methodology for this is flawed because people will tend to search for things they have searched for before on Google and then pick Google as the best because they find that ranking more familiar.

I don't think its flawed. How can you say which results belong to Google or any other search engine?

For example I searched for "japan", and expected to find "japan-guide.com". Now I realize the only reason I even know about that site is because it was in my Google results before, so now I grew to consider it the "correct" answer.

Looking at the source code you can find which site is used for which column when "engines" is defined: var engines=new Array("YHO","MSF","GOO");

Voting will just reveal that information.

Although if you wanted to game the site you could already do it in different ways.

The vote counter is going up like crazy now, and Yahoo is now trumping both Google and Bing. Yahoo was in 3rd place earlier today when I looked. Seems like someone is hitting it with a robot to goose Yahoo's numbers. So much for the data.

I'm actually surprised - my guess would be something like <10% for yahoo and almost equal results for google and bing.

On the other hand the test can't take one important feature into account - profiled results. After 2-3 searches for anything programming related on someone else's browser I feel there's something wrong with the results... yep - I'm not logged in. Even if it doesn't make a huge difference, it is noticeable. I'd really like to check the stats while being logged in into my google account.

My score: Google 3, Yahoo!! 3, Bing 1.

The test is to a large degree unrealistic. Do a search for "sf restaurants". Google and Yahoo!! (in a recent upgrade!) both have extremely useful OneBox/Shortcut results that aren't captured in the comparision.

Cool site, but not a real comparison. I know google does a lot of personalization, based on being logged in or your cookie, but it can't do that on this. I'm assuming the other engines do something along those lines as well.

For someone who's looking to SEO their site, this tool is exceptionally useful just for that reason.

As others have noted, the site is being gamed. Someone is stuffing votes for Yahoo. Rather unfortunate, because otherwise it was interesting and useful.

I did 10 searches about subjects I was passionate about. Results

  Google: 5
  Bing: 5
  Yahoo: 0

Having read what other people were writing in this thread I thought I would also try and rank the results in order, which would be my first choice, which would be my second choice - so for second choice results, here they are.

  Google: 3
  Bing: 5
  Yahoo: 2

So while the Google/Bing split for first place results were interesting, the second place results were more interesting (to me at least).

I'm going to keep throwing in results later today as I think up more topics to increase the sample size a bit to see which is a bit better, as I think it will trend towards a more even split or even a Google lead, but definitely promising results for Bing.

EDIT - Update, I just checked the results again, seems like it was just hacked in Yahoo's favour.

Around an hour ago (when I first published this comment) I noticed it was about 26.5k votes cast with the split being Google 39%, Bing 31%, Yahoo 30%.

Now having a look the split is Google 31%, Bing 25% and Yahoo 44% with 40k votes cast... I suspect shenanigans.

EDIT #2 - 2 minutes later, split is Google 30%, Bing 24%, Yahoo 46% with 41.5k votes cast, so looks like the fixing is in process. Wonder which Yahoo engineer is doing this...

EDIT #3 - 6 minutes after edit 2, Total is 45k, splits are Google 29%, Bing 23%, Yahoo 49%... Any insight that could be gleamed from this rough poll is now officially tainted, nothing more to see here.

Tried three tests: my own name, "Clojure tutorials" (something I've been looking for recently), and ""ORG directives in assembly code" (something I had trouble finding recently). Scores - Google 1, Yahoo 1, Bing 1. Not a very large sample space, to be sure, but still it was very hard to tell the difference, and by the stats, a lot of other people have had similar reactions (with a slight lead for Google).

Maybe this means we should just stick with the company we like the best? Google it is then ;)

Search itself is simply too hard to judge. Since many of the results in the top 10 (at least in my test searches) were the same but perhaps jumbled a bit, I personally think it doesn't matter that much.

Day to day I use search for two things: very general information and looking up code errors. Code error searches will pull up the same mailing lists and forums no matter which search engine you use, and a general information search will either pull up an official website or a Wikipedia page on any search engine. 9 times out of 10 that's sufficient.

There's no superficial reason to why I use Google over Bing or Yahoo!, just simply that I've always used it and I'm always satisfied with it. Yes, I also like its interface the most and non-search related Google applications are generally amazing, but I wouldn't be any less productive if suddenly I switched to Bing for all my search.

(Although I might simply just be biased towards Google... they did just give me a free phone!)

I think the 40/30/30 results that are currently posted on there show that. For the most part it's relatively even, and honestly Google is probably winning just because some die hards are bent on making sure their favorite wins.

He also needs to have a way to mark the worst result out of 3.

In my personal tests the results are almost identical but the order of results may vary by a few spots. The most relevant searches are appearing in the first page so the order isn't all that important IMO. If anything this proves to me that accuracy is no longer the big deciding factor in search for most people. It's all about the platform -- mail, IM, maps, blogs, photos, etc. This explains why Google has stayed strong since they've made the biggest investment in a complete, cohesive, platform.

I've been using the WebMynd plugin since they posted here a while ago. Been searching for normal stuff + django development. Yet too see a better top 3 result from Bing. The ranking is a lot worse, might be that they need a bit of time for tuning. To me it seems to be something wrong with tokenization / word distance weighting.

This is about as useful as Yuil was - i.e. very. Sure, it turns out to be a little self-serving for the person's employer, but you have to acknowledge the brilliance of the effort and the fact that the results themselves appear to be on the up-and-up.

This is great, but I felt like my mental comparison weighted each individual result more equally than I normally would in a routine search. Another interesting comparison would be to show only the top result, top 3, etc. from each search engine.

I use google almost exclusively.

I only used Bing as a test on the day of its launch.

That said, when I chose based on the results: Bing 70%, Google 30%, Yahoo 0%.

Wow...

Seems like I've finally found a way to Bing for "sex" in India.

Seems like I've finally found a way to Bing for "sex" in India.

Actually I found out you can bypass the filter by adding in a generic stop word at the end which the search engine should ignore anyway (I use the word "the") and it will search for whatever phrase you want just fine.

Discovered this about 8 hours ago - http://news.ycombinator.com/item?id=645812

Yahoo seems to be rising incredibly fast. Its greater than 50% now.

which do you choose?

http://blindsearch.fejus.com/?q=sex+tape

uh it's interesting b/c only one search engine ranks wikipedia first!

I tried camino, jetpack, svg, xul and never voted for bing.

As long as MS tries to move down relevant links to competing technology their search engine is useless.

Reply on news.ycombinator.com