A testament to one of the worst decisions in computing history - not to fail displaying a web page with an error message in case it is not a valid HTML document.
Compare that to JavaScript, which will happily fail if you use new syntax or a missing function, and thus web pages which rely on JS often show up as just a full screen of white when something goes wrong, which it frequently does. That's not to say JS should be as flexible as HTML is here, but it provides an interesting contrast.
Why push the complexity onto the user? Someone who just wants to make a working website doesn't care about your pedantry.
Do they want their page to fail to render entirely when PHP outputs a warning? Do they want their website to be completely broken because they forgot to convert some of their text from Latin-1 to UTF-8 before pasting it into the document? Should we really expect them to have to modify their blogging software to validate custom HTML snippets, lest the entire page become unusable? Will they be pleased when their style of code falls out of favour in future, is deprecated, and then their page doesn't work at all later?
Moreover, strictness can backfire when you have such a diversity of implementations.
> Which creates room for bugs, exploits, and unexpected situations like OP's post.
The OP is not so much an unexpected situation as a carefully engineered one that's completely within the constraints HTML sets.
Displaying warnings is fine. Invalid XHTML, less so.
> Do they want their website to be completely broken because they forgot to convert some of their text from Latin-1 to UTF-8 before pasting it into the document?
The encoding of the content has nothing to do with the document markup.
> Moreover, strictness can backfire when you have such a diversity of implementations.
On the contrary: this prevents subtle bugs in the interpretation of invalid data by different implementations.
Because this approach has historically resulted in people who "just wanted to make a working website" making websites that only work in specific browsers (or worse yet, specific versions of specific browsers on specific platforms). And then those sites stuck around and infrastructure got built around them that made them hard to fix.
We've spent the best part of 00s fixing that mess, and there are still some pockets that haven't been properly cleaned up. If that's not a lesson to learn from, I don't know what is.
Isn't that more due to failure to handle exceptions and display errors to users?
You could describe it that way.... or you could describe it as failing to have reasonable default logic for handling faults & gracefully degrade.
I often wonder how different the internet would be if Postel's prescription never gained traction and fail-fast behavior were the norm instead.
Instead, it felt like we had a bunch of people who though that big lofty standards were so obviously correct that everyone else would take care of those boring implementation details, and 99.9999% of web developers correctly realized that there was very little measurable downside to sticking with something which was known to work.
1. Simple examples: namespacing is a good idea but it leads to gratuitous toil in most tools – e.g. a valid XML document which has <foo> should just work if you write a selector for /foo, as present in the document, rather than requiring you to do kludgey things like have to lard up every parser registering the same namespaces which are also declared in the document and writing fully-qualified selectors like /mychosenprefix:foo or /{http://example.org/fooschema/1.0}foo for every tag, every time.
Similarly, getting XPath 2.0 support to actually ship in enough tools to be usable would have made one of the better selling points for using XML actually exist as far as the average working programmer is concerned.
By the way, there's SLAX, it's isomorphic to XSLT but with nicer syntax. Nice approach but anyways, the standards are horribly complex and stiff.
Maybe it would be so bad that somebody else would make a new, more permissive standard that took off instead.
Maybe all of that has already happened.
Maybe that line from Battlestar Galactica was right - All of this has happened before; all of this will happen again.
They started making browsers more lenient since there was so much poor malformed HTML being produced
In a world where invalid HTML documents aren't rendered at all we could have had the evolution of the format dictated by Microsoft because of their market position.
The decision to allow this was made early and the liberal accept/strict transmit paradigm has in general made the web a mess.
On the plus side, the consistent failure of browser vendors to apply strict controls to input means that as an application security person I will probably never be out of work :D Even though this pattern of behavior is starting to change, legacy support means that I will still be dealing with these issues well into my retirement!
How many security flaws are the result of malformed HTML?
To take this article as an example, according to the HTTP specification, the `Content-Type` header is supposed to have the final say in what media type is being served. Internet Explorer decided it would be better to use heuristics. I think the idea was that if a web host was misconfigured, rather than have the web developer fix their bug, it would try to guess its way out of the error.
Which kinda worked. The problem was, it opened it up to abuse. If you had a web host that allowed untrusted people to upload images (e.g. profile photos), you could construct an image that tricked Internet Explorer into thinking that it was an HTML document, even if the server explicitly told clients that it was an image. The main difference between images and HTML, of course, is that HTML can contain JavaScript, which would now execute in the security context of your web page.
So all of these web hosts, thinking they were only giving people the ability to upload images, were now letting people execute JavaScript on their domain – simply because Internet Explorer tried to be lenient.
The workaround ended up being forcing downloads with `Content-Disposition` headers instead of displaying inline. That's why, for example, visiting the URL of an image on Blogger directly triggers a download instead of showing the image.
Other examples that spring to mind:
Netscape interpreting certain Unicode characters as less than signs. People were correctly escaping `<` as `<` but the Unicode characters slipped through and caused XSS vulnerabilities in that browser.
Browsers ignoring newlines in pseudo-protocols. Want to strip `href="javascript:…"` out of comments? No problem… except some browsers also executed JavaScript when an attacker placed a newline anywhere within the `javascript` token.
Being lenient in what you accept has caused security vulnerabilities over and over again and there's no reason to think that it will stop now.
Which kinda worked. The problem was, it opened it up to abuse. If you had a web host that allowed untrusted people to upload images (e.g. profile photos), you could construct an image that tricked Internet Explorer into thinking that it was an HTML document, even if the server explicitly told clients that it was an image. The main difference between images and HTML, of course, is that HTML can contain JavaScript, which would now execute in the security context of your web page. So all of these web hosts, thinking they were only giving people the ability to upload images, were now letting people execute JavaScript on their domain – simply because Internet Explorer tried to be lenient.
This is an interesting example, though I think this is a fair bit different than being lenient on HTML interpretation.
The topic was strict HTML. Accepting malformed HTML doesn't seem to pose much of a problem. Blindly executing a non-executable file seems like a much different problem.
> Netscape interpreting certain Unicode characters as less than signs. People were correctly escaping `<` as `<` but the Unicode characters slipped through and caused XSS vulnerabilities in that browser.
This doesn't sound like being lenient. This just sounds like a bug.
> Browsers ignoring newlines in pseudo-protocols. Want to strip `href="javascript:…"` out of comments? No problem… except some browsers also executed JavaScript when an attacker placed a newline anywhere within the `javascript` token.
Huh? I don't understand the scenario being described here. It again sounds like a bug rather than lenient acceptance of data, though.
It's not. There are two areas where the leniency was a problem here. Firstly, the leniency in rendering one media type as a completely different media type because the browser heuristic thought it was being lenient. Secondly, the leniency in parsing HTML out of an image file – you can't do that with valid HTML.
> Accepting malformed HTML doesn't seem to pose much of a problem.
I've literally just given three specific examples of it causing security vulnerabilities.
> This doesn't sound like being lenient. This just sounds like a bug.
No, it was intentional. It was specifically Unicode characters that looked like less than and greater than signs, but weren't.
> I don't understand the scenario being described here.
Somebody noticed that href="java\nscript:…" wasn't being parsed as JavaScript, and it was causing some malformed pages to fail to work properly. Rather than let it fail, they tried to fix it by stripping out the whitespace, and caused a security vulnerability.
If these three examples aren't enough, take a look at OWASP's XSS filter evasion cheat sheet. There's plenty of examples in there of lenient parsing causing security problems:
https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_She...
I think you can argue the first is a problem. You have an example demonstrating as much. Arguing that the second is a problem is much harder. Lenient HTML acceptance been hugely advantageous to the adoption of the web. There may have been some issues from this, but it's valuable enough that the effort to "fix" it was abandoned and the W3C and WHATWG returned to codifying what leniency should look like.
> I've literally just given three specific examples of it causing security vulnerabilities.
Well, at least one example. Coercing a file served as an image to HTML isn't an issue of accepting malformed HTML, nor would I agree that the JS example is a problem with leniency.
> No, it was intentional. It was specifically Unicode characters that looked like less than and greater than signs, but weren't.
Okay, I reread your last comment. I initially thought you were saying that Netscape was treating '<' as '<'. So Netscape decided to treat some random unicode chars that happen to look kind of like the less-than symbol (left angle bracket: ⟨, maybe?) as if they're the same as the less-than symbol? This seems amazingly short-sighted and pointless. How was this issue not seen, and was this even solving a problem for someone?
> Somebody noticed that href="java\nscript:…" wasn't being parsed as JavaScript, and it was causing some malformed pages to fail to work properly. Rather than let it fail, they tried to fix it by stripping out the whitespace, and caused a security vulnerability.
So the issue here is incompetent input sanitization. I don't think the browsers being lenient here is the issue.
> If these three examples aren't enough, take a look at OWASP's XSS filter evasion cheat sheet. There's plenty of examples in there of lenient parsing causing security problems: https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_She...
A few of these are interesting in the context of browsers being lenient. e.g. This one requires lenience as well as poor filtering:
<IMG """><SCRIPT>alert("XSS")</SCRIPT>">
Wait, that's a completely different point. The argument here is that it caused a security vulnerability, and it did. If the lenient HTML parser didn't try to salvage HTML out of what is most certainly not valid HTML, then it wouldn't be a security vulnerability.
> This seems amazingly short-sighted and pointless. How was this issue not seen, and was this even solving a problem for someone?
You could ask the same of most cases of lenient HTML parsing. It's amazing the lengths browser vendors have gone to to turn junk into something they can render.
> So the issue here is incompetent input sanitisation.
No, it isn't. That code should not execute JavaScript. The real issue is that sanitising code is an extremely error-prone endeavour because of browser leniency – because you don't just have to sanitise dangerous code, you also have to sanitise code that should be safe, but is actually dangerous because some browser somewhere is really keen to automatically adjust safe code into potentially dangerous code.
Take the Netscape less than sign handling. No sane developer would think to "sanitise" what is supposed to be a completely harmless Unicode character. It should make it through any whitelist you would put together. Even extremely thorough sanitisation routines that have been worked on for years would miss that. It became dangerous through an undocumented, crazy workaround some idiot at Netscape thought of because he wanted to be lenient and parse what must have been a very broken set of HTML documents.
This is not a problem with incompetent sanitisation. It's a problem with leniency.
Thanks for providing actual, concrete examples.
(Yes, the output should have been escaped, but that is sadly not always the case)
The cost of a malformatted HTML document rendering despite the errors is not that severe compared to the benefits it provides, as we have seen.
It's not as if many of the web's security holes are related to whether a page displays valid HTML markup or not.
Ultimately it's less about saving keystrokes and more about amateur enthusiasts having the opportunity to start with the browser rendering their unformatted document rather than an "Error at line 1" warning, and changes they introduce being considerably less likely to break the entire page
The real security due to leniency problems in the web platform revolve around the handling of data and how JavaScript works, which were not addressed by XHTML. In that sense it's not surprising it went nowhere.