Microsoft attributes $80-$100 million dollars in revenue to a shade of blue | Hacker News Reader

Microsoft attributes $80-$100 million dollars in revenue to a shade of blue

lukew.com

29 pointstbgvi41 comments

This is the frightening reality of what happens when you have a large business where everything scales multiplicatively and then you make a trivial change which is net-beneficial. People want this to be impossible because it strikes you as intuitively wrong that five minutes of work could be worth eight figures. Reality doesn't care what you think, though.

Relatedly, and more prosaicly, if you average one successful A/B test with a 5% lift per month, and you do nothing else, your business will grow at about 80% per year. Multiplicatively effective businesses are a wonderful place to be in life.

Then I look over at Zynga at all where you can do multiplicative effectiveness within the context of a viral loop, which is just so effective I'm almost certain it goes against my religion somehow.

Don't take this the wrong way but on every comment I've read from you, there is "A/B" :-)

Not sure how Zynga's effectivness might be against your religion, but "A/B" is part of you somehow: religiously or not.

A/B = the core of the scientific method.

Not so sure about that...

It looks to me that A/B leads quite easily to a local optimum.

But it's better than no method at all.

That all sounds very deep for something I think it a lot simpler.

  1. They tried some weird colors. Didn't work well
  2. They thought "screw it" and copied the colors Google use. (Give or take)
  3. Users feel more 'at home' and maybe even assume the
     results are from Google.

You can't replicate that situation. You can't copy the colors Google uses again. I guess you can make Bing even more like Google, but only up to a point.

I suspect you're right on. The rest of this story is just some made-up huge number that would get people talking about Bing.

Also, this 'trivial' change isn't so trivial. The actual change itself is relatively minor, however the infrastructure required to perform proper testing, analysics, etc is huge. Not to say the the large number of users required to do proper testing quickly. These huge number of users are a result of decades of MS' brand, multi-million dollar marketing campaigns, etc etc.

Kind of like how it takes a lifetime to be an overnight sensation.

80% per year outpaces Moore's law, btw.

Moore's law isn't about business growth.

No, it's about growth, in general.

No, it's about the number of transitors on an integrated circuit and the resulting computational power.

If one takes the narrow view, yes.

> if you average one successful A/B test with a 5% lift per month, and you do nothing else, your business will grow at about 80% per year.

Aren't there too many variables? How do you know the increase isn't just a correlation?

So the article says that #0044CC is the $80 million color. But if you check bing.com they're using #0033CC. So what's up? Maybe it's a trick to get everyone to switch to the obviously inferior #0044CC color while bing laughs all the way to the bank with the real killer color!!

Bing - #0033CC Google - #2200CC DuckDuckGo - #3068DD Facebook - #3B5998 (what where THEY thinking? ;-)

The point isn't that there's a magic colour, it's that they A/B split tested the new colour against the old one and saw an increase in the number of people clicking. It doesn't mean the process stops right there.

Yes, I was being sarcastic. :-)

I know, but I was being pedantic :)

and I'm being tautalogical :)

These are notes from a presentation by Paul Ray. It's an hour long so I thought these would be better to digest. The $80 million color blue comment is down by "Data Driven Process".

If you're interested in the full video presentation you can check it out here http://live.visitmix.com/MIX10/Sessions/CL06

"A specific color of blue (#0044CC) drove $80-$100 million dollars a year increase over the light blue the design team tried first."

I call bullshit on this statement. There is no way they could have measured this reliably unless the only thing they changed for an entire year was the color of the links.

Unless they A/B tested the colour change.

That's my point - you can't A/B test something while making other changes too.

You could if those changes were replicated in both the colour A test and the colour B test?

Why would they have to change it for an entire year? Maybe it's enough to test it for 1 day, or 10 minutes for all I know. If it improved clicking rate and revenue by 9%(+-1%) and after 1 year they've got $1 billion of revenue, then you could say that 8~10% of that billion likely came from Magic Blue.

You want to sample for 10 minutes or 1 day and extrapolate for 1 year ? The results would be meaningless.

Even if you sample 1 continuous month you could have badly flawed results (for example you might have sampled only early adopters, which might behave differently; and that's just one example of the many, many factors that can make signal indistinguishable from noise). Now I'm sure that the guys working on Bing A/B testing know what they are doing, but I just can't believe a change in link color would bring a "$80-$100 million dollars a year increase" (from a total of ~2000 million $/year revenue which I extrapolate from this http://www.microsoft.com/msft/earnings/fy10/earn_rel_q2_10.m... but which includes all revenue of the Microsoft Online Service Business - I don't know how much of that is generated by Bing)

Huh? The article makes no sense at all. I mean WTF?:

"But something was troubling me with this number. In color theory the thing doesn’t work like that. In order something to be complete red + green + blue should be 255. But in this case we have (0 Red + 68 green +204blue = 272) And 272 <> 255."

That also bothered me. Why should the math matter?

I think you meant for that to be a comment not on this item but on another one, pointing to some guy's rambling deranged blog entry about this allegedly very profitable colour change, full of numerology and nonsense. And yes, indeed, that one makes no sense at all. This one, on the other hand, seems reasonably coherent.

Each color goes from 0 to 255. It's not a sum.

Oh, good. They found the hyper-intelligent shade of blue from Hitchhiker's Guide to the Galaxy.

(Actually, Google does the same kind of color testing.)

>> "Guess what? Microsoft also tested multiple versions of blue for links in their search results. A specific color of blue (#0044CC) drove $80-$100 million dollars a year increase over the light blue the design team tried first."

You can't change a color, see things increase, and claim it's all down to a change of color. Also as bing "increased", Live "decreased". Has Bing been successful?

It makes for a fun PR story, but it's complete BS.

Whatever happened to science? How much money did Live search make? How much is Bing making? Have they been able to add any traffic?

Some hard data would be interesting.

One point to note is that they essentially copied the colors straight from Google.

It's called A/B testing. You serve people one of two versions of a page at random, and test which performs better. With enough data, you can be pretty damn certain that the single change caused the results. It is, in fact, hard science.

There are obviously a couple of caveats. First, you need to run the two tests in parallel as you allude to, and second the separation between the samples needs to be completely random. I would assume that Microsoft can keep these caveats in mind.

It think they'll need to do some A/B/A testing to be sure.

Switch back to the original shade only for those that experienced the new shade. Maybe they'll see another multi-million dollar increase? That way you could further prove that it's indeed the color that made the difference and not simply the fact that the page was changed at all.

Added:

The Hawthorne effect is a form of reactivity whereby subjects improve an aspect of their behavior being experimentally measured simply in response to the fact that they are being studied, not in response to any particular experimental manipulation.

http://en.wikipedia.org/wiki/Hawthorne_effect

Personally I like to A/B test with new users. But I guess on an open website like Bing (instead of one with logins) you won't be able to really know if someone has visited it before.

It's not that simple.

Suppose you ran an A/B test like you say.

User 1 goes to website, it's site A. Dislikes it. Nothing happens.

User 2 goes to website, it's site B. Loves it. Tells friends.

Now how do you track that 'virality' element? How do you know if User 1 was the cause of increase traffic, or user 2?

How do you know if I tell my friend "Hey checkout bing it has this awesome blue color", and how do you know if I was an A user or a B user?

I think there's just too many untrackable elements in a case like this.

And in any case, the story still sounds like there was no A/B testing involved. Just "we changed a color", and later, stuff happened.

> One point to note is that they essentially copied the colors straight from Google

Or their A/B tester implemented as a genetic algorithm evolved the colors towards those used by Google because that increases their chances for survival :-)

Interesting that the url of the search result is also green, like Google. I just opened the two sites side-by-side and their results use mostly the same color scheme:

Blue Title Black description Green URL

I would definitely say MS went with a the "familiarity factor."

In case anyone's interested in pedantic details, the color appears to be #0033CC (not #0044CC) and it can best be described as STRONG VIOLET (according to the Color Curious iPhone app).

So they found the missing shade of blue!

Think I'll stick with DuckDuckGo.

They should name that color "Hooloovoo".

Reply on news.ycombinator.com