Couldn't this be fixed by rebranding? Instead of IE8, release a totally new browser that just happens to reuse a lot of code from IE. No hacks get applied. It either works or it doesn't, and if it doesn't it also doesn't work in Firefox, Opera, etc.
I suppose it would be a nightmare from a marketing perspective, though.
You and I tend to forget that such sites exist, because the environment of the public web, which teems with Firefox and Safari users, has rendered them mostly extinct in the wild. But in captivity, behind various corporate firewalls, where every single user has been running IE for a decade, such sites still exist, people care about them, and those people just happen to be Microsoft's core customers.
If, as you suggest, Microsoft ships a totally new browser -- which, let's face it, might as well be called "Firefox 3.0", although I suppose it could just be IE8 with a different User-Agent heading -- you and I will be happy, but lots of internal corporate sites will break, with no upgrade paths other than "redesign your site's HTML to be standards-compliant" or "tell your users to run two browsers: one for the intranet and one for the modern world". And corporate customers don't want to fork over license fees in exchange for that.
That's the basic contours of the dilemma: MS can either blow off their installed base or blow off the standards movement. The installed base has the money. The standards movement has the influence. Spolsky is right: The IE team is well and truly damned. They're just going to have to weasel out of it somehow.
On the other hand, Microsoft's piecemeal approach to web standards is starting to hurt it. The de facto web development best practice is now "Develop for a standards-compliant browser first, then hack it until it looks right in IE." This hurts Microsoft because it means that most new webpages only work incidentally in IE; if a corner case of some new feature is accidentally missed in the IE rewrite, the workaround might well be: "Try using Firefox until we get a fix out."
If Microsoft goes through with what they've promised, one day you've got a web site which works on the latest version of IE, and the next day (when MS releases IE 8) you don't.
Since all web pages aren't going to be fixed in a day, practically nobody will upgrade.
The only solution is to do what they did for version 7, and 6, and 5.5. Maybe they could have prevented the problem by releasing a more standard-compliant IE 5, but since then, they're stuck.
The good thing is that we're slowly getting better at abstracting the problem away.
Having users press a button for IE7 mode whenever they see a broken page is just asking for trouble. Now you've equipped every Solitare-playing cube dweller in your company with a "break this page" toggle switch that needs to be constantly moved back and forth. The call volume to the IT department help desk just went up by an order of magnitude.
So you're stuck with stupid hacks like "enter IE7 compatibility node if the URL matches this regexp". Fragile, fragile, fragile. And what if you actually have the budget to slowly migrate your ancient intranet site into the modern world, one page at a time? How do you serve half your pages in IE7 and the other half in IE8 without a special tag to distinguish them?
Most likely these intranets keep IE7 on their computers and slowly migrate and phase out.
It's not like these sorts of things are alien to the entreprise world - they spend an enormous amount of time dealing with legacy incompabilities - but consumers don't want legacy browsers and that's what Microsoft are afraid of.
Change that and be done!
One possible solution is to identify as IE <= 7 to intranet sites (which can be designated via Group Policy) and IE >= 8 to non-intranet sites. Of course, this means admins have to properly define intranet/non-intranet, which means much rioting will ensue.
Poor IE team. You're getting screwed because management decided to rest on their laurels.
...There must be a good reason why this would never work, because I don't recall hearing it proposed before.