The way I see it, it's ultimately about tradeoffs. I can only imagine what things would be like today if web browsers implemented a strict parsing of HTML and refused to render invalid pages. One possibility is hindered adoption of HTML by the masses. Another is that two vendors would disagree about the HTML spec and cause pages to be browser-specific. (Turns out this happened anyway :-))
Is the HTML5 spec better in terms of interop and compatibility than the previous ones? http://www.tbray.org/ongoing/When/201x/2010/02/15/HTML5
You could argue that we would have been better off new if all browsers from day one had only rendered valid html, but you need a time machine to fix that.
Some authors might use the newhtml doctype (because they have read somewhere it is better) but only test in a browser which dont support newhtml mode, so they still don't discover that the html is invalid. So we are back to square one.
The various HTML strict modes turn off "quirks" mode as well.