How about instead of a full web runtime you change the problem to be implementing (or inventing) server and client protocols for common web services? (Vblogging, Q&A forums, micro blogging, social bookmarking, wiki's etc.)
Essentially every kind of service (e.g. email, blogging, q&a, live news) is available without JavaScript, thus, using a pure html through http interface. The problem with a-standard-protocol-per-service is that new uses arrive in a distributed, unplanned manner.
Looking at instant messaging history is instructive: there were 3 protocols in major use (aim, msn, icq) about 20 other in common use. The “standard” committee was sabotaged by the major players for years and eventually disbanded, culminating in the only open option in some use (not major use, just some use) - XMPP - to win by default, except the providers explicitly chose to NOT interop (Facebook, WhatsApp when it was independent, Google chat).
Of course you're right that this would all be hardcoded and would not allow new types of sites to work right away.
I don't know that you'd even need a protocol or service for each category of site. It would probably make more sense to use the same architecture for all types of services with something like a manifest to determine the mode. I think the challenge would be making the APIs public in a way that would be practical for the servers implementing them.
For example: Instead of an API for microblogs and another for blogs and another for news sites it could just be one API with flags or something that determines which other calls are used and how.
Along comes Tumblr, and … the APIs don’t fit, in the sense that they significantly limit the planned functionality.
Now what? A new API! Or a committee to update the existing API!
Central planning doesn’t work for these things. And when you do finally have agreement on something (e.g. email), you either stagnate because no one can improve things, or get incompatible implementations because a significant player decides they will break the standard for some reason (e.g. recall and TNEF on outlook, which only work on outlook).
The internet started the way you describe (with finger .plan for “social” status, Unix talk for one-to-one, IRC for chat, email, nntp, gopher, ftp, Xwindows, RDP, etc etc). All of these is are 98% web based these days, because the protocol+client+server model cannot adapt quickly enough. The modern web with presentation layer + runtime code delivery + transport protocol does allow lightning fast evolution.
And now we have dozens of popular implementations of roughly the same functionality layered on REST/HTTP or JSON/WebSockets or whatever.
I suggest that the complexity is basically that same, from a programmers point of view. You define messages and a state machine and a bunch of event-handling …
The UI is now universal (mostly) but the programming model for HTML/CSS isn’t simpler than say Xaw/Xt: it is more capable, but putting together a decent UI for a browser-based email client is not substantially easier than doing it in Xaw/Xt.
With one exception: our programming languages are better, and the ecosystem of libraries and frameworks makes what would once have been weeks of work an import statement.
We could do the same things in the same time using custom protocols over at TCP as we do using JSON over WebSockets, using modern tooling, but the world has moved on. The entire ecosystem of libraries and services and network infrastructure channels you into using a vastly more complex stack.
The web is the best application runtime for making clients. However, I don't think it's existence invalidates the creation of these kind of protocols and server APIs. In fact some web standards such as RSS feeds could be described as such.