Hacker News Reader

Comment by tedunangst | Hacker News Reader

"Together, these two algorithms form the core of the parser and consist of over 10,000 lines of code."

Anybody else think that if your markup language requires a 10k line parser, somebody took a wrong turn at the complexity vs simplicity fork in the road?

View on HN

Pay attention to this part:

  All browsers that implement the HTML5 parsing algorithm
  should parse HTML the same way, which means your web page
  should parse the same way in Firefox 4 and the WebKit
  nightly, even if it contains invalid markup.

Parsing perfect HTML5 would be easy, but one of the features of HTML5 spec is that it does that no spec did before: it defines how parsing should work exactly, even in the case of invalid markup. Also, the parser hast to deal with deprecated elements no longer in the spec (such as infamous <font>). I assums most work went into this "how to parse tag soup" part.

Deciding that invalid markup should work would be the wrong turn I alluded to.

It would have been very easy to drop all the back compat bs by saying "An HTML5 document is one that begins with the 6 bytes '<html5' and if the document is invalid, reject it. Anything else, parse however you want." Browsers that support HTML5 add text/html5 to the Accept header.

So, how many different parsers did you want to see in the browser again?

You'd still need to support HTML4 somewhere. Supporting HTML5 separately just means duplicating the common parts of the parser. The simplicity boat has already sailed.

Addition is simpler than combination.

    HTML4 + HTML5 < HTML4 * HTML5

They're extremely similar. You don't get an explosion of code size. Unless you're suggesting that HTML5 should also be radically different syntactically as well?

I'm suggesting that a strict HTML5 parser that doesn't have complex recovery code is radically simpler than one that does.

I'm suggesting that a HTML4 parser that has complex recovery code, and a mostly-copy-and-paste HTML5 parser that doesn't isn't a big win.

Browsers are going to have a complex, ugly, recovery-enabled parser in them either way, and the effort to add HTML5 to the recovery-enabled parser isn't very big, comparatively speaking.

Not really true in this case. WebKit's previous HTML parser is similar in complexity to the HTML5 parser. Adding a second HTML parser would have been more code, more complexity, and a more complex test matrix.

Complexity does not increase linearly with number of lines of code. 110k lines can easly be 5x as complex as 100k lines of code.

1 + 1 > 1 * 1

Alas, your six bytes would drop older browsers into quirks mode, which makes is not nice. The only reason HTML5 has doctype at all is because it forces browsers into standards mode. What you propose sounds a bit like the way XHTML2 was intended to go, we know how that ended. There is and will be tons of invalid documents on the web—the wast majority of it, so browser rejecting them has no future, the blame will be on the browser, not the authors. Having at least some consistency in dealing with invalid documents is A Good Thing™.

The problem with this approach is the same problem people serving XHTML as application/xhtml+xml ran into: You don't control your page any more these days.

If you allow people to leave comments on your page or if you are serving ads, you lose a bit of control in what gets put on a particular page of yours.

In case of comments you could try and sanitize them, but with ads that's hardly possible.

I tend to think that very few things that see widespread real-world use are "simple" by any reasonable metric.

The basic idea may be simple, but by the time you have covered a reasonable fraction of real-world use cases and error conditions for a sizeable population, the resulting code is anything but.

And 10k lines still counts as "moderately simple" - 10k lines is well within scope for one developer working alone.

This must be your first time hearing about HTML, welcome!

Just imagine how large "quirks mode" must be.

You make it sound like it was a conscious decision. Most of the complexity comes from the need to tolerate the janky markup that older browsers processed in a reasonable way.

Just for kicks, I am going to try and implement these two algorithms using using the State Machine Compiler (http://smc.sourceforge.net/). I expect that my "code" will be much less than 10,000 lines, although I don't really know how long the code SMC itself generates will be.

Is there any advantage to SMC vs. Ragel for parsing? The code looks similar but a bit larger.

Heh, just what I need, a new toy to play with!

I spent a few hours looking into Ragel over the weekend, and, honestly, I can't envisage how a html5 parser mostly written in Ragel would look. But I'm going to give it a crack.

What's the magic number of LOC that any markup language should be implemented under?

To put 10k in perspective, that's substantially more than the pcc C compiler front end, and closing in on the compiler's total line count.

When I consider what I can accomplish using C vs what I can accomplish using HTML, that doesn't appear to be the expected result.

C compilers don't have to deal with users that expect purely random garbage thrown at the parsers to have a "valid" parsing that's consistent across every implementation. Unfortunately, making a language simple to use by not requiring that users get everything right makes it very hard to implement.

Considering the number of people that can accomplish anything useful with HTML vs the number of people that can accomplish anything useful with C (or even XML), it's well in line with my expectations.

It's fairly easy to get a simpler implementation by shifting complexity from the parser to the user. But from a "total society productivity" standpoint, having a handful of developers spend a few months to implement a resilient parser is a cheap price to pay to have umpteen million people save hours of debugging for every page they create.

Back when HTML was invented, it was based on SGML. In retrospect, this was a fundamental design flaw. SGML has all sorts of fillips that were intended to make documents easy to type (back in the pre-GUI era) but which in practice make them hard to parse. SGML parsers, even parsers built into expensive commercial software, were famous for not reliably interoperating. The HTML5 spec abandons SGML entirely because, as it turned out, even those fillips of SGML did not make it possible for browsers to consistently agree on how to parse HTML in the wild.

It would have been better for everyone if HTML had used a bog-simple Lisp-like format from day one, but without a time machine we’re stuck with these kludges of history.

Most markup languages (indeed most languages) don't have error discovery and recovery built in and specified.

HTML5 does, and most of the spec (and probably the parser) is about dealing with error cases.

Plus it's not like XML parsers are small. And HTML parsing is far more complex an affair (due to not just blowing up on error)

Having this complexity ensures that nobody will ever endeavor into writing any more browser engines, those ensuring that there will be no more browser bugs. The rest of the job is fixing the behavior of the already existing browsers and then the web will be perfect.

HTML already tried the "let's switch to a simple parser" route, and failed:

http://dig.csail.mit.edu/breadcrumbs/node/166

Now HTML5 has to deal with two parsers.

Reply on news.ycombinator.com