Bing smacks Google in new usability test | Hacker News Reader

Bing smacks Google in new usability test

thenextweb.com

71 pointszuzzo60 comments

Potential confounds:

* Google submits many more searches with it's instant AJAX search-as-you-type

* Google's speed allows users to hone in on better search results by tweaking terms, whereas Bing/Yahoo users don't waste their time with a 2nd search

* Google displays the answer right on the results page with superior semantic analysis, nullifying the use of a clickthrough

* Google's tools (such as "define:", math like "4+1.078", and conversions like "45cm in inches") are well-known and people look to Google itself for answers, not secondary sites in results

* If Google's users are more "tech savvy", they may use search qualitatively differently than Bing users (related to the above explanations)

* Google users click on Google Ads more than Bing users click on Bing ads - are ad clicks tracked?

Of course, these could all be wrong. But worth looking into or acknowledging in the research if they already had.

Another confound I would note is that Bing is highly specialized in certain queries, and that users may select Bing for the queries they know will result in specialized results, and use Google for everything else.

If Google corrects a spelling mistake for a user: did you mean x? than that is counted as a bad user experience. If a user gives up on Bing and goes to Google to refine her search, than that is counted a good user experience for Bing.

Another one based on the logic that Google's users are more "tech savvy" (which I believe they, on average, are):

* A larger percentage of Bing users are doing "time-saving" searches, such as "hotmail" and "facebook", where they know the first result is what they want even before they search.

And extending that logic - a tech savvy audience will be more likely to have more complex results. It's not hard to give searchers want they want when they're looking for "superbowl results" or "justin bieber".

which I believe they, on average, are

Self-selection bias?

Here's what I observe among my non-tech friends: they have Google as their homepage. If they want Facebook, they type "Facebook" into the search box - even tho' typing it into the address bar would do the same thing. If they come across a PC that's not on Google, then they go to Google using the address bar and proceed as before.

Matt's response to this: http://www.google.com/buzz/109412257237874861202/A7d5joZ3tJJ...

There's another possible source of selection bias: users may be going to Google for queries that they know are a hard, because they assume that Bing won't have the answer. Of course such queries will fail more often, because they're hard.

Bing's marketing has centered around the head of the query distribution - common tasks where definite answers exist, like how to book a flight. It makes sense that such tasks would succeed much more often than long-tail ones like looking up an obscure error message.

Well if they are "going" to Google for hard queries then wouldn't they try the queries on Bing first?

Or are you saying that people spend time deciding a query is simple or hard and then choose appropriate search engine for that?

They might try the query on Bing, fail, and then try the same query on Google. A large portion of those are likely to fail as well, just because the queries are hard.

It's less likely that someone will try a query on Google, fail, and then think "Maybe Bing will do a better job with this."

Well in that case wouldn't bing's percentage of failed searches be even larger than Google's? The stats don't reflect that

It would cause Google to have a relatively larger proportion of "difficult" searches. Many "difficult" searches on Bing will be repeated on Google, but not the reverse.

If you assume that all "difficult" searches fail on both engines as a first approximation, that would mean that Google will fail all the "difficult" searches, while Bing will only fail the "difficult" searches made by people who primarily use Bing.

More that they have certain problem domains where they might pick a given search engine, and that those problem domains are biased toward using Bing for easy searches and Google for hard ones.

I'll frequently search for airplane tickets on Bing, for example, while I won't even go to Google for that. But when it comes to looking up obscure programming stuff, I'm not going to bother searching on Bing, I know it'll suck.

"More that they have certain problem domains where they might pick a given search engine, and that those problem domains are biased toward using Bing for easy searches and Google for hard ones."

That makes no sense. Why would you, or anyone, alternate with search engines if one of them too often fails to deliver? Why not just use Google all the time and avoid the cognitive overhead?

Does Google do a poor job, or a worse than Bing job, with airplane tickets?

I use DDG except for local sites, which it doesn't index anywhere near as well as Unca Googs.

A third possible source would be bias in user samples:

If Bing users are on average less sophisticated (plausible given the assumption that less sophisticated users tend to use IE with the default search engine) they are more likely to click through irrespective of search results' relevance or the potential to improve a query.

What does the "success rate" of AOL's Google-powered search look like?

Does someone have a link to the information on how Experian's data was gathered?

It looks to me like 'success' is defined as actually clicking a result - not the ridiculous things like switching to google - but their explanation on the report page is fairly nebulous.

Suppose the user was on search engine A and then goes to site B. I believe Hitwise has no way of knowing whether the user clicked on a link in the search results to go to site B vs. just going to site B on their own.

As far as how the data was gathered, Hitwise gets clickstream data from ISPs: http://www.hitwise.com/us/about-us/how-we-do-it

Here's the gist of Matt's reply: 1. Hitwise's definition of a successful search doesn't actually involve clicking on result.

2. MSN has pages that generate bing search while masquerading as regular pages.

Mr. Cutts provides an example of Kim Kardashian videos on MSN linking to Bing as an explanation for the difference in success ratings between Bing and Google.

However if that were a significant factor, the similarity between the ratings for Bing and Yahoo would be somewhat unexpected without a demonstration that similar searches are directed to Yahoo. It would also seem that there is probably a significant enough behavioral difference between the user bases of Bing and Yahoo Search for Matt's "abandoned search theory" to be consistent with the similarity in success rates between the sites.

A more plausible explanation (but "creepier" in Googlespeak) for Google's results is that Google's sometimes practice of not linking directly to sites but rather to an internal redirecting page [as described by jacques mattheij here: http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+...] lowers Google's success rate. It would also explain why Google questions the value of the metric so vigorously and why their criticism is speculative and anecdotal rather than more scientifically argued.

http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+...

I clearly see where he is coming from and I also believe that the metric Hitwise used is just BS...BUT! the more Matt becomes sensitive about Bing, the more pageviews Bing would get. I bet Bing got quite unexpected PR recently due to Matt's accusation (I don't want to talk about the issue since my opinion is mine). I'm curious to see metrics for Feb. ;)

Very interesting that the first response most HN commenters have when confronted with data that Bing provides better search results than Google is to doubt the data.

This despite that fact that (I'm guessing) most commenters have not actually extensively used Bing day in and day out. Shows the huge barrier to entry that any new search engine has: even if they provide measurably better search results even tech savvy users have an almost unshakeable belief that Google results will always or somehow must be better.

There seems to be some infallible belief that Google provides better search results, always will and that there is some 'magic' Google has for search that no other company could possibly reproduce or exceed.

There is no logical reason that this should be the case and in many ways its more logical to suggest that Google is so wed to its current techniques and revenue models that it has no incentive to disrupt itself. The recent furore over spam (and Adsense which funds its existence) perhaps is the first evidence that this is actually the case. Google made some noise about rectifying the spam issue but really they didn't make any hard decisions in this space.

A personal anecdote - I switched all my browsers and iPhone to default to Bing about a month ago - the experience has been pleasant, the results excellent and haven't found the need or desire to use Google search since.

Very interesting that the first response most HN commenters have when confronted with data that Bing provides better search results than Google is to doubt the data.

The very first reaction I have to most studies is to question first the methodology, then the data. Remember that 88.2% of statistics are made up on the spot.

Yes it's true that people are rightly doubting the methodology.

However if the data showed the exact opposite result (Google having better results than Bing) I think the reaction on HN would be to accept the results at face value. This despite the fact that there is no reason (that I'm aware of) that Google search results should be inherently better than Bing.

In this instance, I think it shows the strength of the Google brand and how the strength of that brand sets users expectations for the quality of the results.

To add to the anecdata - I switched everything across to duckduckgo.com about the same time ago and am incredibly happy. The fact that duckduckgo sends me to google when it runs out of ideas helps, but I'm finding myself needing that less and less as I go.

At this point I use google occasionally simply because I finger-macro Ctrl-T,Alt-D,google.co.uk,Enter before actually thinking "if I hit Ctrl-K I'd have duckduckgo search already".

Here's an interesting comparison:

Google: http://www.google.com/search?hl=en&safe=active&q=Fan...

Bing: http://www.bing.com/search?q=Fantasia&go=&form=QBRE&...

Yahoo: http://search.yahoo.com/search;_ylt=Agt64BoFz7.2ArgJZ_Io2BSb...

What stood out to me is that images and videos are not being served up front and center on Google.

No doubt more and more people are searching for video. Google really looks old-fashioned in this match-up.

My preference:

DuckDuckGo: http://duckduckgo.com/?q=fantasia

I prefer uncluttered and informative results at the top instead of news headlines.

I do too, but I find for non-trivial searches, I often have to make use of !g.

More than once I've spent time doing multiple searches for the answer of a tech question, re-wording the search to try and find a hit, only to have an a-ha moment and say, "oh! hey! '(original query) !g'" and bam, first page, The Answer.

Its amazing how close they all are. One of the key things thats different is the bar at the top - but all same layout, format, color......

My strongly preferred solution to this problem is to search for the desired media by using a keyword, like "!images cat".

I'm not aware of a way to do this from within google, which creates an odd situation: setting my default search to duckduckgo increases google's usability.

But people are forgetting the hover text that Bing returns (see 1st result).For the search "circumference of the earth" http://oi55.tinypic.com/2jfx8ur.jpg

Compare that with Google's results https://img.skitch.com/20110211-r3jmccgi1ejs24p1hbdnu69fwm.j... (courtesy of xentronium). For this query Bing's results are far better that Google's (they actually specify circumference at equator and poles, alerting you that it's not the same; Google doesn't).

Also if you add (in meters|in miles) to the query, you get similar results to Google's.

I initially assumed that everyone around here would have tried Bing, then I realized that may not be true. Have you guys given it a try? When Bing came out I switched my browsers to Bing for a couple of weeks and noticed essentially no difference in usability or results quality. Alas, after reinstalling operating systems and browsers, Google gradually defaulted back to the browser I use, and it's just not worth the effort to change again. I really don't prefer one over the other.

I don't use Bing for search (I'm tempted to start using DDG) but I do use their maps quite often (though not exclusively).

I really don't have any brand loyalty to any of the search/map/whatever providers - if something seems to be consistently better I'll switch (as I did from AltaVista to Google).

Does my 'search' for "skitsophrenia" as a lazy-man's way of spellcheck actually count as a query? IMHO this metric is flawed.

50% of my Google use, as a non native English speaker, is for spell checking, even for tricky native German stuff, USD to EUR conversion etc and I've also never been using a calculator beside Google for some years now. Google just does that fine.

"If you are like most people in tech, the last time you used Bing was never"

Woah. I am a SE and I use Bing's image and travel search all the time!

Are you saying you are most people in tech? He did not say everybody.

A study like this would only yield meaningful results if the user selection for all three search engines were the same and randomly selected from the general internet-enabled population.

In this case, one interpretation of these results is that google users look for something very specific and thus are less likely to be satisfied with the first result that pops up, while bing users are more casual and search for generally consumable content.

>"bing users are more casual and search for generally consumable content."

That would seem to be bad for Google. Bing gets to sell ads alongside casual searches such as "cheap table lamps" and Google has to try to monetize "'Company A' 143rd Ohio Brice's Crossroads."

Wow, Google partisans need to calm down a bit.

The reduction in Big G's search engine monopoly is good for the market and good for tech in general.

I am extremely disappointed with the new-fangledness in the newer Google interface changes.

The 'instant' search is just a jarring user experience in the way it does massive unpredictable updates to the screen at every keypress. Even worse is the new image search results screen - mouseover on any image makes the search field lose focus, it even does this when fetching the first page of results, so you have to click the search field again every time you do a search.

Even the suggestion search field they introduced a few years ago was a dubious value proposition - I personally got better value from the standard text field's 'previous entries' than from the suggestions.

I highly doubt their tactics here. They're grabbing aggregate ISP data for users (http://www.hitwise.com/us/about-us/how-we-do-it) and running scripts to see if the search is followed by an address that appears on the search page. Also, it seems like they don't account for google's instant search, but it's hard to tell as their first "success" statistic was posted in October (Instant Search was released on September 8th). They do however, publish upstream traffic percentages for sites in various categories (http://www.hitwise.com/us/press-center/press-releases/google...) prior to September. Note that in this case, the ratio between the upstream click throughs (75% for google, 15% for yahoo, and 10% for bing for health related click throughs) and the ratios of the aggregate search traffic (71.4% for google, 15% for yahoo, and 9.4% for bing during that month) allow us to extrapolate the "success" rates during this period, and it seems to suggest that everyone is in line for search results (with google being slightly ahead, at least for health. I didn't run the numbers for the other categories as we dont know the search volume for each of those, and as such you couldn't find an average statistic that was relevant). Just my two cents, but it seems like the data is highly affected by google's instant search.

"‘success rate’ .. a measurement of how often a user executes a search, and then goes to a webpage from the set of links"

One simple example where this fails: http://www.google.com/search?hl=en&source=hp&q=circu...

The answer is shown three times in the results. There is no need to click on anything. Did the testers compare these results for Google and Bing?

Bing also shows the answer, however: http://www.bing.com/search?q=circumference+of+the+earth&...

Perhaps Google does this better, but I would be surprised if it accounted for the large difference.

This is one of the rare queries where I'd try Wolfram Alpha first:

http://www.wolframalpha.com/input/?i=circumference+of+the+ea...

https://img.skitch.com/20110211-r3jmccgi1ejs24p1hbdnu69fwm.j...

vs.

https://img.skitch.com/20110211-cfdpqbca86fe1qgwdpbutdu153.j...

Google's are better, however, they both don't show the value for me.

I see it in the second result from Google!

It's even worse than that. As I understand it, the definition is not "a measurement of how often a user executes a search, and then goes to a webpage from the set of links" but rather "a measurement of how often a user executes a search, and then goes to a different website." That is, I believe Hitwise has no way of knowing whether the user clicks on a link from the search results vs. just navigating to a different page.

From my experience back in the pre-google days was that I'd have to open up five or more links to ever see anything worthwhile and this was back in the day when page titles had zero relevance too.

Plus, I haven't seen the google front page to search in months, so that's a definite +10 for usability in my books. If I was smarter (or more pro-procrastination) I'd have a link to google news on my chrome splash screen.

I call sampling error. Bing users are easily pleased with results, take anything they're shown. Google users look harder, and often refine a search.

The explanation just might be the fact that google submits new searches as you type

I also find it interesting that a few defunct microsoft webpages (such as http://popfly.com) redirect you directly to bing. Naturally, you'll follow the wikipedia or equivalent link to figure out what's going on.

I doubt this would generate a large amount of click-through data, but it makes you wonder what other tricks they have up their sleeves.

>It implies that the user found what they wanted

No it doesn't. It will penalize a search engine where the user base like tweaking the search & unduly boost the score of one with sheep for users that click on the first thing they see.

The number of users that use Bing is much lower than the number of users that use Google.

This comparison is similar to saying that book X that has a total of 4 ratings, 5 star each is better than book Y that has 100 ratings, 4 star each.

Given the volume of both though (they grabbed these stats from 10 million searches) the statistical relevance of each should be pretty definite, even if you're using a bayesian average.

Google sucks in results recently.

My greatest search success is when the information I need is in the displayed snippets. Others clearly agree: see DuckDuckGo's 'zero click info' and Google's 'One Box' results.

Next best is when the snippets make it clear I need to refine my search, rather than wasting my time wading through the result sites. Again, no clicks from the results, but a happier searcher.

So the Hitwise 'success rate' metric seems totally bogus to me – inversely correlated to the quality of snippets/OneBox.

I suspect Google's snippets are still the best in the business, though an objective analysis of that would be interesting news.

When powerset launched they had very impressive snippets. Now that they've been bought by microsoft, I wonder how much of the technology has been integrated.

Noticed how Yahoo! has a higher success rate than Google? That alone is proof that these numbers do not correlate to the real world experience.

Must have been wrong shade of blue.

i think they opened google.com because bing was not good enough to come back to!

Reply on news.ycombinator.com