Microsoft Cross-browser Test Results Summary | Hacker News Reader

Microsoft Cross-browser Test Results Summary

samples.msdn.microsoft.com

73 pointsvtail82 comments

For all the misrepresentation of facts and existing browser inconsistencies i'm just happy that microsoft has apparently started caring about the browser again.

They may care about the browser, but I fear they care about regaining market share more than anything else. They had a firm lead for a long time, but more and more I see sites making support for non-IE browsers a top priority and the IE market share is shrinking.

If Microsoft uses tests like these to try to divide the browser market by essentially defining its own spec, I think it will be bad for the internet at large.

To be fair, these tests were submitted, and apparently developed with the W3C. They aren't MS-only tests. Merely tests that needed to be developed anyways. At least, that's what I see. Yes, these are tests where MS succeeds. That doesn't mean they are less worthy. Rather, they are just more cases that the W3C felt needed to be tested.

they care about regaining market share

MS is a big beast, and "market share" means different things to different parts of MS. The developer tools division, which makes, among other things, ASP.NET server-side environment for serving web pages, cares increasingly about working on any browser. And this is a good thing.

If you want to know what the IE9 team thinks about HTML 5 and cross-browser issues, try some of these:

HTML5: Cross-Browser Best Practices http://live.visitmix.com/MIX10/Sessions/CL27 HTML5: High-Performance Best Practices for Web Sites http://live.visitmix.com/MIX10/Sessions/CL29 In-Depth Look at Internet Explorer 9 http://live.visitmix.com/MIX10/Sessions/CL28

While this is obviously tailored to things that microsoft is doing I wouldn't be surprised if these features end up in the hands of actual users through other browsers first.

Microsoft's upgrade policy for IE is extremely conservative which gives the other browser vendors plenty of time to add in the features and release them to people before Microsoft does.

True. But still good if only by pushing others.

How does it feel to have a company other than Google create tests specifically designed for the features they implemented?

I'm pretty sure Microsoft have had plenty of practise at creating these sorts of tests :)

Isn't ACID3 independent? If it is why does IE do so badly?

It's one of many tests. While they aren't at 100/100 yet and rendering the page correctly, they have made solid strides since their last release of IE9. They are progressing.

But yeah, ACID3 isn't the only test out there browser makers appear to use.

Fact: IE9 Platform Preview #1 released six weeks ago scored 55/100, IE9 Platform Preview #2 released today scores 68/100 on ACID3.

Thought this http://arstechnica.com/microsoft/news/2010/03/ie9-standards-... was interesting. It paints Miscrosoft as a company that aims to thoroughly support standards, although what they may be doing is looking for under-specified parts of the specification and defining the standard in those areas.

Obviously Microsoft only presented tests that they have passed and at least one other major browser has failed. Still this is free QA for the other browsers so there is not much room to complain.

Not true: Call removeAllRanges() to clear the selection pass pass pass pass pass

They just submitted tests for the features they were working on. Some of which, happen to work on all other browsers as well.

Actually, if you look at the original tests they submitted it was 100% for IE across the board, and if I recall correctly (since they've just replaced the old page) no other browser passed any subset 100%. Clearly they cherry picked the tests, it makes no sense that every test they wrote just happened to pass at that point in time. Either they didn't publish the tests that IE failed, or they intentionally stopped writing new tests some point before they went public to give them time to pass everything fully.

Of course after they published these results it was pointed out that a) the test were wrong and so other browsers were marked as failing, even when they actually passed, and b) IE was getting things wrong but getting marked correct because the tests were wrong. (That's not even touching the fact that they don't even indicate that Firefox and Webkit would pass a whole heap of the tests just by changing a single string in their code to check a different prefix)

They've since scrambled to get their column back up to 100% pass rate and that's why they've updated this page with their new version. Pretty weak all round.

They didn't cherry pick tests. These weren't written by someone else. They wrote tests for the features they decided to implement. Why would they work on tests right now for features they aren't implementing now?

I am pretty sure they prefer to implement features for tests competing browsers fail.

good joke :)

Come on. Given the option, between two equally complex features, it's preferable to implement the one your competitors got wrong because it offers the more bang for your programmer buck.

This is why the Google Sputnik test results not only shows what percentage of tests pass, but also groups browsers together based on which tests they pass in common (i.e. a crude measure of interoperability).

http://sputnik.googlelabs.com/compare

Guess which browser is the outlier in both dimensions?

Clearly they cherry picked the tests

What's the point in creating new test to cover things that existing tests already cover?

What's the point in a summary that ignores literally hundreds of tests? Tests that, as you point out, already exist and so would require absolutely no effort to run.

Why have a row in the table that says SVG IE 100% when the truth is very different:

http://www.codedread.com/svg-support.php

The Google plugin that transforms SVG to Flash passes twice as many tests as the IE9 preview. Opera passes 3 (nearly 4) times as many, yet Microsoft felt it was useful to list a random subset of tests that show them doing well.

Just to be clear, these are tests created and submitted by Microsoft themselves, so it's not so surprising that IE9 passes all of them as it otherwise might be.

Beautiful, I'm happy for MS to make IE9 comply with all this new stantards. Now, I leave you. I have to spend the rest of the evening trying to make a site to work and render correctly in IE6, IE7 & IE8 (at the same time). :(

What about the older versions of firefox, opera, etc...

Wouldn't the users that use an outdated version of Internet Explorer also use an outdated version of an alternative browser, if they had used this browser in the beginning.

I don't know about others but I find much less old version of Firefox, safari and chrome than IE 6 users access my sites. It seems (this is only a personal observation) that users of these alternative browsers are more likely to update frequently.

I think you're right. Another big factor is also the automatic updates of Chrome and Firefox. IE updates have to go through the windows updates and people tend to ignore those as much as possible.

Another factor about IE6 is that it's only used in organizations and companies where people can't install their own programs, so they don't bother calling up IT (sometimes outsourced IT which is expensive) just to install a browser.

The only reason to use an outdated version of, say, Firefox is when your OS isn't supported by the latest version.

Not many people run IRIX these days.

As for IE, it's not trivial to have more than one version of it installed under a single Windows image. That and internal applications that rely on a specific quirky version of IE won't help with keeping your browser up-to-date.

Ok they are comparing beta/preview version of IE with stable version or not even recent versions of browsers?

The current beta/dev version of Chrome is 5 not 4.1. The current FF nightly build is 3.7a5 not 3.6.3, the current stable Opera version is 10.6.3 not 10.5.2.

As usual, more FUD from microsoft.

Where is the FUD?

These are tests submitted by MS. The declare what they are testing, and what they are comparing against? FUD would be MS saying that IE supports standards better than the latest competitors browsers. They aren't saying that. They aren't even implying that. The only way this is FUD is if you selectively choose to be daft about the entire thing and ignore vast portions of the page. It would be if I was selective about your post, and assumed you said:

"I ... Am ... FUD"

Edit: I should note, they are even pretty clear about what they are comparing: "The first table is a summary of the test results with the May 2010 IE Platform Preview and each of the major shipping browsers running on Windows."

They don't pretend to test the latest beta/alpha/nightly release of the browsers. Merely the latest shipping.

Actually, it's called vaporware. It's related to FUD, but it's not the same.

What they are saying is "don't port your corporate applications that require IE to other browsers and don't standardize on them. Our next version will be much better than theirs and it's just around the corner. We promise".

A couple days ago they announced their Courier tablet was canned. If you remember when it was leaked, the timing of the leak was to coincide with an Apple announcement everybody suspected would be of a tablet computer. The message was "Ours will be much cooler. You don't have to consider buying hardware that doesn't reinforce your dependency on Windows".

why all this hate? maybe they are trying to comply with standards, maybe it will help, maybe we will not have to work twice to align the f CSS on IE? At least this is their formal goal and these are very good news.

No their formal goal is to do a half-assed job and pass themselves as the leaders. In hind-sight a hald-assed job is still better than the clusterfuck that is IE, that doesn't change the fact that they are being intentionally dishonest about the test circumstances (Old browser version against IE preview, tests tailored made for them).

I am sure you have memos from MS as to their formal goals

I do. But in hind sight I can't remember MS takes a 30% cut from the app developers selling their own apps.

It's not a standard if your tests are designed to match what your browser currently does.

The real tests of compatibility are those like Acid3 that are designed to fail all browsers equally. This one is clearly designed to fail all non-IE browsers.

I don't think they designed them to fail non-IE browsers, so much as they designed the tests for things they had already implemented in IE 9.

I think they could have avoided most of the criticisms in this thread if they had simply added a note saying something like "We made sure that IE 9 passed these tests before submitting to W3C, which is why IE 9 gets perfect marks below. Note also that these tests do not cover all of the HTML5 specification." (etc.)

Having IE9 at 100% on every test with no explanation really makes it look like they're trying to put one over on the reader.

How many important W3C tests pass on windows XP running IE9?

None... (IE9 wont run on XP)

How many important W3C tests pass on windows XP running Firefox 3.6?

Most...

And according to Wikipedia as of April the XP has a OS market share of 52.94% This value is the median of 7 different data sources. http://en.wikipedia.org/wiki/Usage_share_of_operating_system...

What this means is that by Microsoft refusing to make IE9 available for XP, web developers often can't take advantage of CSS3 or HTML5 and still reach even 70% of the audience.

http://arstechnica.com/microsoft/news/2010/05/chrome-continu...

I would tend to agree with pavs that this is more FUD from Microsoft. http://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt

So we are going to say that a standard is only a standard if it's considered important by you? A standard is a standard. It's either supported, or not. Firefox devs don't just sit there and say "Hey, we have 80% of the standards done, that's enough. The rest isn't important."

As for the XP talk: wtf does that have to do with IE9?

IE9 won't run on XP, Microsoft can release this for Windows 7 and still hinder development of web apps. They look good in the public eye "Hey, we are 99% CSS3 and 99% HTML5 we are standards complaint" but in truth screwing us just as they have for years. If they wanted to stop screwing us they would release a compliant browser for XP and offer it as an upgrade in a windows update.

"IE9 won't run on XP"

Yeah, I know. So what?

"Microsoft can release this for Windows 7 and still hinder development of web apps."

So, instead of building a better browsing experience, they should port IE9 back to XP... despite it not supporting half of what they need it to support to do what IE9 is doing?

I understand what you are asking. I understand why you think it's reasonable. I disagree, however, because it's not reasonable, and it's a very bad idea for many reasons. It's also silly, and ignorant.

Edit: Ignorant might sound strong, but I really do think you just don't know why backporting IE9 to XP is probably a bad idea. I'm not calling you stupid.

it's a very bad idea for many reasons

The only bad reason that I can think of is that they might want to use IE9 as an incentive for XP users to upgrade to Win7. I can't think of any others. (Maybe you could share.)

If XP users want a more compliant browser without upgrading from XP, they'll have no problem, though, so this doesn't sound like a good choice of leverage on Microsoft's part.

It's quite simple. IE9 will be making use of OS features that aren't available on XP, which is after all a nine year old OS. MS have chosen to take advantage of OS features in their OS-specific browser. No mystery.

I was kind of hoping for something more specific. After all, WebKit manages to use modern APIs (CoreAnimation, etc.) on MacOS X, yet Chrome and Safari both ship on XP. Can you think of anything that can't be easily shimmed?

Web Kit isn't a single rendering engine.

http://www.quirksmode.org/webkit.html

There is a reason Safari on OSX performs much better then on Windows. Safari and Chrome are differnet browsers. Safari on Mac is different from Windows, even at the basic level. Web Kit is a foundation, not the end result.

Run over to the IE9 testing site, and run the animated stuff on Chrome and on Safari, and you'll see abysmal results when run on a Windows machine. Then, run the same test with Safari on Mac, and it will be like night and day.

Now, if IE9XP can't support the same features of IE9Win7 like SafWin can't support the same features as SafOSX, then there is arguably more harm in supporting XP from a developer stand point, and therefore, a user standpoint. As a developer, it's like having 2 more browsers to check for IE9XP/IE9Win7.

That's not even getting into the security concerns with XP.

Even Apple doesn't support Safari on Cheetah!

Even Apple doesn't support Safari on Cheetah!

I'll hazard a guess that this might have to do with the success they've had in getting users to update off of old versions. While you might be able to find a Tiger box out in the wild, I bet you'd never find someone running Cheetah. Doesn't over 60% of HTTP traffic come from XP boxes, in contrast?

They are using Direct2D (or DirectSomethingOrOther) for the hardware accelerated rendering. I don't believe this is available on XP. There will no doubt be a bunch of other Win7/Vista specific bits of functionality that would build up to an appreciable amount of hassle when implementing an XP version.

You're missing the bigger, and much more legitimate omission: Windows Mobile 7 is running with IE7. Which effectively means these things won't work their way into WinMo for at least a year after the release of IE9.

Since Windows Mobile 7 isn't out yet, I'd like to note that the current version of Windows Mobile's IE is a hybrid of IE6's rendering engine with IE7's JavaScript engine, a Franken-browser no one is ever truly going to support.

Has IE suddenly become the most standards compliant browser, or is there some subtlety here that's missing?

Well, Microsoft has created their own tests. They argue that tests like Acid3 are testing edge cases and as such are not really useful to test browser behavior in the real world.

What I don't understand is why they don't use the latest Chrome beta to do this tests - e.g. Chrome 5.0.375.29 beta passes all tests for Dom Level 2 Style.

Presumably their goal is partially to make IE look good compared to other browsers. Comparing the latest beta of their browser to stable versions of their competitors skews the results of their tests in Microsoft's favour.

> What I don't understand is why they don't use the latest Chrome beta to do this tests - e.g. Chrome 5.0.375.29 beta passes all tests for Dom Level 2 Style.

That's a valid question to ask and I'd tend to give them the benefit of the doubt on this. As I understand it they chose the most recent released versions of the other browsers.

If they had of chosen the most recently nightly build or beta versions and a particular test didn't work on that browser but had previously worked, ie it was a regression that would be fixed in the next build, then someone would jump all over them for "cherry picking" their competitors versions to obtain results that flattered IE the most.

Picking the most recently released versions seems to make the most sense from this point of view, though I'll allow that you could make the point that choosing the most recent nightlies of the other browsers would also make sense.

It looks like they are cherry-picking HTML5 features that IE supports, but Mozilla/WebKit don't. But I don't see much of the important stuff like tag support, web workers, geo-location, databases... MS is pretty much full of shit.

I agree about cherry picking but I would not call them full of shit. The reason being that every product when being developed starts with a limited set of features and then goes on to start new features once the old ones are complete. May be they are at the point where they have implemented everything they are testing but not other stuff hence no tests for them.

Note that they never say that they are most standard compliant browser. They specify the tests they are running and the % of them being passed.

Disclaimer: I know I might get down-voted for this but I work at MS(not in the IE team though :) ). Of course, my opinions are my own and not my employer's.

You're more likely to get downvoted for predicting that you'll get downvoted than for disclosing in good faith that you work at Microsoft. People at HN have strong opinions about tech companies, but they're usually not uncivil.

No offense. MSFT (as a company) has always been full of shit, throughout their history, so even if they are not full of shit (in this case they are, as they are comparing out of date version of other browsers with their preview version of IE, on a test that has been custom made for them), its hard to take them seriously or trust their words in any way, shape or form.

I see

These are the standards-compliance unit tests that the Microsoft IE team submitted to the WC3. Obviously they're only submitting tests that they already passed, but since they focus on passing standards that they claim are important to web development, the tests are probably worth something when combined with other benchmarks. It doesn't mean that IE is standards-compliant on the whole or compared to other browsers, but it does mean there's at least some evidence (albeit biased) to support their claim that they're focusing on standards they consider important rather than benchmarks. Whether these standards are important for real-world applications is something I can't speak to.

...or is there some subtlety here that's missing?

Nothing subtle about it really. They are only presenting tests in which IE passed.

You are missing things are stated on the page:

IE9 is in "Platform Preview" - meaning "not even beta". It is compared against current released versions of other browsers. They will catch up, and by past experience some will release before IE.

These tests play to IE9's strengths. I don't really have a problem with that since the tests will push other browsers to catch up, but there are other tests like ACID which have different focus and showcase other browsers.

You just used the phrase "catch up" (twice!) to describe other browsers that are far ahead of IE in pretty much every objectively measurable respect.

Clearly publishing a comparison of a "platform preview" (i.e. not even a nightly, it's just the core rendering engine) against the stable, released versions of other browsers is an effective psychological manipulation when even someone who is pointing out that very fact can jump to the conclusion that all these other browsers haven't progressed in the meantime.

Er, the test is an "objectively measurable respect" in which the IE9 Preview is ahead.

Having said that, sorry about the poor wording. And I agree that the released version of IE trails the released version of all other browsers. This has been consistent over several major release cycles. Still, I think MS should be encouraged when they're heading in the right direction.

I'll take this seriously when they support the existing standards properly. Stacks of HTML/CSS/JavaScript books are full of pages saying this/that doesn't work in IE.

There are stacks of books about the IE9 preview just released?

Hilarious!

Such irony. We respect how challenging it is to build one page that works the same in every browser as the technology advances and customer expectations rise.

Then what's stopping them from including IE6,7 and 8 in their "tests" and fixing this whole mess they created in the beginning. They act like those previous products aren't their responsibility... fucking joke

Have a little mercy. All MS is saying is that they're working to get things right.

All browser vendors crow about the functionality of their beta releases.

Using Firefox 3.6.3 here, the same they listed.

I tried the "Call select() on a text field" test because it looked like a simple one I could check myself.

It selects the text field perfectly. When I start typing the current text is replaced, everything seems to be working fine. "Test result: FAIL"

It's only one test, sure, but this page is smelling even fishier than before.

This works just fine for me on FF: http://samples.msdn.microsoft.com/ietestcenter/html5/selecti... I am guessing they put in some tricks in the validation here too.

hardware acceleration is the big one. This will force the hand of the other browser makers to follow suite. Webkit is already pretty far along here but only css animations, not the entire rendering pipe, which seems to be what the IE team has managed to do.

Also, the two-threaded Javascript interpreter is a very nice performance feature for IE9.

Although MS's "Testing Center" is fixed in IE's favor, their IEBlog does compare IE to the latest competitor releases and "showcases" IE's poor Acid3 performance. [http://blogs.msdn.com/ie/]

This is really good news.

Microsoft is excellent at producing software that ticks feature boxes. Anyone who remembers magazine reviews of Excel vs Lotus 123 will remember the relentless way MS would improve in the comparison tables in every release until suddenly they were ahead. The same thing happened in the original browser wars.

The problem is that Microsoft has never been good at choosing which feature boxes need ticking themselves.

But now they have decided to care about list of features called "HTML5" they should be able to push things forward well. (I'm hoping anyway. And I'm hoping "Canvas" makes that feature list)

It says that firefox and webkit browsers fail on border-radius, that's only because they use -webkit-border-radius and -mozilla-border-radius and Microsoft only tested border-radius which is the one supported by IE, not very honnest especially knowing that gecko and webkit had those for years before IE.

But, to be fair, do they support border-radius? I understand what you are saying, don't get me wrong. But either a standard is supported or it's not.

If you want to get technical: Firefox, Safari, and Chrome have 0 support for the HTML5 standard, considering HTML5 isn't official yet. But they still work toward meeting proposed standard.

Competition is good. If IE9 supports border-radius, and that pushes FF and others to support border-radius without the vendor tag, might that be a good thing?

vendor tags are a plague. if it's not in the spec, fine, add them, make your browser appear pretty, but when it hits the spec, you need to pull support for your specific one, or make damn sure you support both.

border-radius isn't in the spec. The CSS3 Backgrounds and Borders model isn't final yet: it's just a candidate recommendation. See http://www.w3.org/Style/CSS/current-work

The whole point of promoting something to candidate status is to get people to start working on implementations.

http://www.w3.org/Consortium/Process/Process-19991111/tr.htm...

"A Candidate Recommendation has received significant review from its immediate technical community (resulting from the Last Call). Advancement of a document to Candidate Recommendation is an explicit call to those outside of the related Working Groups or the W3C itself for implementation and technical feedback."

> ... and that pushes FF and others to support border-radius without the vendor tag, might that be a good thing?

Only if border-radius doesn't change it's semantics; say the standard switches from radii to area. Or perhaps they add some semantics for negative lengths.

If anything like that happens, then we now have two border-radiuses: Microsoft's and everyone-elses.

I think that with border-radii this isn't likely, but for whatever is still under debate, we could still be looking at this sort of thing.

I know. But then you have people saying that X-browser has HTML5 support, but HTML5 isn't even finalized. This means all those sites supporting HTML5 aren't supporting a finalized standard. So you can't use HTML5 now... technically. If you do, you aren't following the standards. Canvas shouldn't be used, nor should the video tags, for anyone who cares about standards. Technically.

Getting a little to anal? =)

Okay, before someone jumps on me and thinks I'm being serious, yes, I agree with you. It's 'dangerous,' but at some point, someone needs to say "Damnit! Let's make this work!" After all, you don't see anyone saying we should use vendor tags for HTML5 markup, even though HTML5 isn't complete, and can, and still does, change.

The great thing about standards is that there are so many to choose from. - Someone else, some other time.

Actually, for the extreme (but sometimes useful) cases, IE9 really shows why the other browsers fail at border-radius: http://blogs.msdn.com/ie/archive/2010/03/19/the-css-corner-a...

Why they are so shy? We all know that it was Microsoft who was invented the Internet, created HTML, Javascript and their browser is the most secure, fast and reliable one.

Reply on news.ycombinator.com