Chromium Blog: Does Your Browser Behave? | Hacker News Reader

Chromium Blog: Does Your Browser Behave?

blog.chromium.org

48 pointspavs8 comments

I really wish standards bodies would create validation tests like this so that implementers can better ensure that they are compliant. It seems that reference implementations and test suites would improve standards compliance immensely.

I couldn't agree more. A good validation test and set of examples are worth a hundred specs.

I've been working on a project for the past couple months that involves using XMPP for client-to-client communication. While there are some good how-tos and libraries, the process would be infinitely easier if XMPP.org's documentation was in terms of reference examples and sample packets/messages, rather than incredibly dense specs written in convoluted techspec language that takes some time to comprehend, even when reading intently.

A refreshing counterexample is the RubySpec project. While Ruby began with only a reference implementation, the process of developing a comprehensive test suite that is able to measure conformance to the reference implementation has created an invaluable metric for other implementers.

While JavaScript has a very thorough specification, Sputnik's project of building a test suite that measures the conformance of an interpreter to the spec will be helpful in understanding how far we have to go, and in which direction we're moving.

I believe that such test suites would rival the complexity of the practical implementations. Reference implementations sound nice, and may work well for small things, but become large engineering efforts where they are really needed.

What would a reference implementation of a Linux distribution look like, for example?

I think you are right for particular cases, but CSS, HTML, and Javascript are all feasible to create test suites for. Either in the form of unit tests, or in the form of written test cases. From this input the expected output looks like this. And, unlike the spec itself, these example cases can grow over time to be more complete. A partial test suite is still better than no test suite. I am all for pragmatism and if a particular domain being spec'd doesn't lend itself to a particular method of validation, then don't do it, be there are plenty of specs that could benefit that don't have them.

Perhaps something like this? http://www.linuxfoundation.org/collaborate/workgroups/lsb

Now we need something like this for CSS/html to test _all_ the standards as opposed to some of the standards that most benchmarks do (AFAIK).

There are official test suits to go with CSS available from the W3C - here's the latest snapshot of the one for CSS2.1:

http://www.w3.org/Style/CSS/Test/CSS2.1/20100127/

This version currently has 8777 tests, covers most areas of the spec but isn't complete.

http://sputnik.googlelabs.com/compare

I'm surprised to see Opera 10.50 beat out Chrome and Firefox.

Reply on news.ycombinator.com