Broken Links
tbray.org
tbray.org
1) HTTP GET http://twitter.com/some_account [~500ms for me]
2) 302 redirect -> HTTP GET http://twitter.com/ [~600ms for me]
3) HTML tells browser to download some JS -> HTTP GET bundle.js [~500ms for me] (concurrently here we start getting CSS)
4) JS reads hashbang & request actual data -> HTTP GET data.json [~500ms for me]
... only after about 2 seconds can we start(!) rendering data. Now there's about another 2 seconds for all json data & CSS calls to complete. It takes upwards of 4 seconds for a twitter page to render for me (the Load event lies as it fires well before actual data shows. Try it yourself with your favourite browser inspector).
When not using hashbangs, a single HTTP request can get all the data for the page and start rendering it. One blocking CSS call (possibly cached) is all that's needed for styling.
Hence when I see an external link with a hashbang it frustrates me (barely perceptively) because I know that when I load the page it's going to take a longer than a normal HTTP request. Significantly longer. While subsequent page loads are faster, it's not these you want to optimise for if you care about bounce rates. This issue affects every new link you click into a website, so it affects an even larger number of requests than normal bounces.
Hashbangs are a good solution to an important problem, but I don't see them as a tool to build entire websites upon. Fortunately I see the performance issue as one which will result in people voting with their browsers and choosing sites which only use hashbangs when they genuinely improve the user experience - especially since they're easily visible in the url.
Unfortunately, web applications and web pages are growing increasingly divergent. It is simply not feasible to take the performance of web apps to the next level without doing away with full page loads. This is why Facebeook, Twitter et al are going the #! route. That's the cold hard truth.
You follow a canonical link to the resource, get a real page back, with real links, but js click handlers to enable AJAX-goosed speed for those with javascript enabled? And given that they imply a fallback for those times when javascript fails, aren't they actually better?
And GMail is a bit different than Twitter. It handles inward-facing data; content that no-one particularly wants crawled and wouldn't benefit much from caching.
Admittedly, you need a modern browser for that. But you can always present full-page-load HTML to users with older browsers and then provide AJAXy history-ified goodness to everyone else.
Twitter is very different. If you're signing in yourself to update your own feed, that's one thing. But when you are trying to view the feed of someone else, that's a horse of a different color.
The question really is: should Twitter be an "app" or a "site"? If you allow links in from the rest of the web, it should be a "site".
For example, browsers could implement partial caching. Here is how it could work. The first time the browser requests a page, it gets all the content in the response. However, some fragments of the content are identified as cacheable and marked with unique ids. When a browser requests a page for the second time, it sends a list of identifiers for the cached fragments to the server. The server then doesn't render those fragments, but places small placeholders/identifiers where they should be substituted into page content.
---
First Request
GET index.html
---
First Response
[cacheable id="abc"] [h1]This is twitter[/h1] bla bla bla, header content [/cacheable] ... Page content ... [cacheable id="xyz"] footer content [/cacheable]
---
Second Request
GET index.html Cached: abc, xyz
---
Second Response
[fragment id="abc" /] ... Page content ... [fragment id="xyz" /]
See <http://dev.w3.org/html5/spec/Overview.html#dom-history-pushs...; for spec details.
It's in Safari, Chrome and Firefox. While Opera and IE don't have it yet, it would be easy to use conditionally on browsers that support it. I'm a little surprised that more sites don't use it.
EDIT: This chart shows what browsers it will work in: http://caniuse.com/history
And your server could easily send a fully rendered page on the first page load when it receives a full URL (one which was made by pushState and linked elsewhere) and still subsequently load pages via XHR. So it wouldn't have to parse any JS on first load -- subsequent loads would, but they'd be saving time from not downloading as much and not refreshing the entire page.
Apart from not being supported in IE - the browsers that do support it still have quirks e.g. your code has to manually track scroll state.
The AJAX approach to Web apps does provide a genuine user interface benefit. A full page load is very disruptive to application flow, and being able to have new data appear without incurring that penalty is great. Most of the time you only need to load a little bit of data anyway, and it's wasteful to reload all the markup that wraps it.
AJAX solves that problem, but it creates the new problem that your address field no longer properly reflects the navigation you do inside a web app. #! URLs are one approach to fixing it, and pushState will do even better. At that point, the user won't even have to notice that the Web app they use is built in an AJAXy style, other than the navigation being smoother and faster.
Not every website is a web app. Just show one article or item or whatever the site is about under one URI.
On many examples I don't see any real disruption to application flow with just using normal links, though there are more full-fledged webapps (like gmail) where I would agree. Playing around with old v. new Twitter, the old one actually has considerably faster navigation performance, at least on my setup (and I'm using a recent Chrome on a recent Macbook Pro). Sure, some HTML header/footer stuff is being retransmitted, but it's not very big.
Within an app, my current context (which control has the focus etc) is important and a full page reload loses all of that.
Within a web site, as it is less interactive, this stuff doesn't matter so much.
As to whether New Twitter is a site or an app is debatable (I say site and therefore shouldn't be using #!). And as for Gawker...
1) don't use AJAX in response to actions that alter the page content in a significant way. This of course forces page reloads and prevents the cool emerging pattern that is to not serve dynamic HTML but just have a REST API and do the rendering client side.
2) you do the ajaxy stuff but you don't touch the URL. This leads to a nonworking back button and prevents users from bookmarking or sharing links to specific views. You can work around this google maps style with some explicit "link to this page" functionality, but I would guess, people just don't get that.
3) you do the fragment change thing which allows for ajaxy page content changes but also makes the back button work and keeps links inherently shareable and bookmarkable at the cost of that one redirect, at the cost of screen-scrapability and maybe confusing to techies (normal people probably don't care either way)
pushState can look like a salvation, but keep one thing in mind: to keep the page working for browsers without JS (and screen scrapers), you will have to do your work twice and STILL render dynamic content on the server which is something people are now beginning to try to avoid.
Finally, as pushState is yet another not widely deployed thing, for the next five to ten years, you would have to do all of this three times: dynamic HTML generation for the purists. pushState for the new browsers and fragment change for IE.
Personally, I really feel that fragment change is a good compromise as it works with browsers and even in IE while still allowing the nice pattern of not rendering anything on the server and keeping the URLs shareable.
Maybe this current uproar is based on a) techies not used to this (normal people don't notice) and b) badly broken JS that sometimes prevents views from rendering AT ALL, but this is not caused by an inherent problem with the technology: if I screw up the server side rendering the page will be as empty as it is if I screw up on the client side.
This would be a problem for search engines as well, if it wasn't for the awful translation Google said they'd do. It's just breaking the meaning of fragment identifiers completely, and that really makes me worried.
Take New Twitter, for example. If I click on a tweet in my stream, it shows related tweets. If a drill down a few of those, at some point it becomes impossible to represent the address of the current state in a sane manner.
I think URLs are particular to the web (desktop apps don't have them) because the web is traditionally about content. Web applications are increasingly breaking that. Perhaps web applications and URLs don't go together all that well.
Don't get me wrong--I love URLs, and it's crazy for content sites like Lifehacker to break them for so little benefit. But maybe the reason for this hashbang trend is that URLs aren't expressive enough for some of these sites.
But most of the popular content accessible via the web now fits this description. Look at Google's own homepage, it's a complex Javascript application that's completely opaque. @bruceboughton is right, the problem isn't that people aren't respecting the WWW specification, it's that the specification is no longer adequate to describe what the Web has become.
First I thought they were referrer log spamming, then it dawned on me that fragment identifiers get stripped out of HTTP referers, so making hash-bangs useless as a means of joining up distributed conversations on the web.
Somewhere on those two Gawker media sites there's a conversation going on about the use of hash-bangs. But nobody outside knows about it. It's a big black hole.
If you visit http://mysicksite.com/article/1 javascript changes all the links to the #! format. Then when the user clicks the links they enter #! land.
Now the user copies a link from their address bar and puts it into the wild. Someone gets that link, http://mysicksite.com/#!/article/1, and visits it. Rewrite with htaccess or whatever method you employ to serve the content at http://mysicksite.com/article/1, using javascript to change all the links to the #! format.
I posted this in the reddit thread about the Gawker/lifehacker problems recently, but was too late for anyone to really give me a response. For those of you that have worked with these kind of systems before, would this solve the problem the original link was describing?
EDIT: Ahh I think I get the problem now, of course after I post it. Server doesn't get the data from the uri trailing the #! I think?
Yes, the old, conservative model of HTML is very simple, but when people use AJAX well, the user experience is enormously and materially improved. We're still early in the development of this medium, and many people will do it wrong. But even the people who do it right will probably seem inelegant and kludgey by the standards of the old model.
And yes, you can get both AJAX and clean URLs via (still poorly-supported) HTML5 History API and/or other progressive enhancement methods, but these may require a significant amount of additional effort. Maybe worth it, maybe not.
This topic reminds me of when sound was added to movies. "Tight coupling" and "hideous kludge" sound a lot like the arguments that were made against that too. The conventional wisdom was to make your talkie such that the story worked even without sound; one can still sometimes hear that, but it isn't, I think, a standard that we associate with the best movies being made today.
http://code.google.com/web/ajaxcrawling/docs/specification.h...
So when you see the lifehacker URL in the article, you know that there's an equivalent non-AJAX URL available with the same content at:
http://lifehacker.com/?_escaped_fragment_=5753509/hello-worl...
There's no need to execute all the JavaScript that comes back from the server - if they're following the spec, all you have to do is escape the fragment and toss it over to a CGI arg.
Another option is progressive enhancement, where you make every link point to a valid page and then add onclick event handlers that override the click event to do whatever JavaScript you want it to. I think this is a far superior option in general, but it has various issues in latency and coding complexity, so a good portion of web developers didn't do it anyway.
In other words the spec treats one of the symptoms, not the original problem.
May be I'm missing something, but it seems to me that there is a way to have your cake and eat it too in this case.
Say we have a site with a page /contacts/ which lists various contacts.
On this page there are completely normal links like '/contacts/john/', each link preceded by/wrapped by an anchor tag - <a href="john"> in this case.
If you visit this site without javascript enabled (e.g. you happen to be a web crawler), you just follow the links and you get just regular pages as always.
If however you've javascript enabled, onclick events on each url intercept a click on a link and fetch just the information about the contact you clicked on (using an alternate url, for example /contacts/john.json), cancels the default action and (re)renders the page.
Then it does one of two things: - if pushState is supported it just updates the url - if pushState is not supported it adds '#john' to the url
If someone visits '/contacts/#john' with javascript enabled, /contacts/ is retrieved and then john's data is loaded and displayed.
If someone visits '/contacts/#john' without javascript enabled, he gets the full contact list, with the focus on the link to john's page, which he can then click.
By using this scheme: - search engine and other non-javascript users can fully use the site and see completely normal urls - XHR page loads are supported - XHR loaded pages don't break the backbutton - XHR loaded pages are bookmarkable - Bookmarks to XHR loaded pages are fully shareable if the recipient has javascript enabled or pushState is supported, and at least not totally broken if not.
The only drawback I can see is the 'sharing bookmarks with someone who has no javascript support' issue - is that a real biggie? In addition of course to the 'made error in javascript, now all stops working' issue - but that is something that has not so much do with the #! debate as well as with the 'is loading primary content via XHR a good idea' debate.
To me it seems that current users of the #! technique have just gone overboard a bit by relying only on the #! technique instead of combining it in a progressively enhancing way with regular HTTP requests.
http://searchengineland.com/google-proposes-to-make-ajax-cra...
http://searchengineland.com/googles-proposal-for-crawling-aj...
http://searchengineland.com/its-official-googles-proposal-fo...
Of course, a better solution is some type of progressive enhancement that ensures both that search engines can crawl the URLs and anyone using device without JavaScript support can view all of the content and navigate the site.
For developers of AJAX apps it: 1. Improves productivity 2. Improves user experience 3. Is more efficient on the server as it prevents a lot of initializing code.
I think the old school needs to wake-up a bit!
Just because you don't like using hashbangs does not mean no-one else can.
Sure, use of hashbangs might make seo of your site harder. Yes, it might make it harder for hackers who want to do curls of your site's pages. But maybe this is not your aim with your site.
Maybe you want to give your users a slicker experience by not loading whole new pages but instead grabs bits of new content.
The web is a place for experimentation and we as hackers should encourage such experimentation, rather than condemning it because it does not fit with how we think things should be done.
If the site is ajax enabled for a slicker experience, then as the user browses from here they might get something like this in their address bar:
/shop/shoes#!shop/socks
or even
/shop/shoes#!help/technical
which starts to look really weird. The google hashbang spec at least fixes this problem. The spider understands the normal URLs of the app, and will dispatch users to them.
That's one redirect to the homepage (which you're already doing by 301-redirecting the JavaScript-free URLs anyway), so it's hardly going to be difficult.
I'm puzzled, considering the haphazard redirects already going on for incoming links to hash-banged sites, why this isn't a trivial problem.
Incoming link is to /shop/shoes#!shop/socks JavaScript right at the top of /shop/shoes that window.location to /#!shop/shoes
1) The link is weird and confusing in the first place, /shop/shoes#!shop/socks refers to two different resources
2) The server will already have done work to find the shoes, when the javascript redirects to the socks page.
2.) The site is already paying this price by redirecting _escaped_fragment_ URLs, and the old clean style urls. All inbound links will have this problem, so you're only shifting some of the burden through this door instead of the others.
<a href="realurl.html" onclick="javascript_magic(); return false">
/help/thing#!/something/otherthing
Which is equally confusing and more error prone for the developers as /help/thing loads code specific to that view and then the /something/otherthing stuff is loaded too. Not insolvable and preventable by proper encapsulation, but stuff leaks, so mistakes will happen.
The ideas behind it still stand, though, in the idea of microformats. These are just standardized ways of using existing HTML to structure particular kinds of data, so any program (browser plug-in, web crawler, &c) can scrape through my data and parse it as metadata, more precisely and with greater semantic content than raw text search, but without the tedium that comes with ontologies and RDF.
Now, these ideas are about the structured exchange of information between arbitrary nodes on the internet. If every recipe site used the hRecipe microformat, for example, I could write a recipe search engine which automatically parses the given recipes and supply them in various formats (recipe card, full-page instructions, &c) because I have a recipe schema and arbitrary recipes I've never seen before on sites my crawler just found conform to this. I could write a local client that does the same thing, or a web app which consolidates the recipes from other sites into my own personal recipe book. It turns the internet into much more of a net, and makes pulling together this information in new and interesting ways tenable. In its grandest incarnation, using the whole internet would be like using Wolfram Alpha.
The #! has precisely the opposite effect. If you offer #! urls and nothing else, then you are making your site harder to process except by human beings sitting at full-stack, JS-enabled, HTML5-ready web browsers; you are actively hindering any other kind of data exchange. Using #!-only is a valid choice, I'm not saying it's always the wrong one—web apps definitely benefit from #! much more than they do from awkward backwards compatibility. But using #! without graceful degradation of your pages turns the internet from interconnected-realms-of-information to what amounts to a distribution channel for your webapps. It actively hinders communication between anybody but the server and the client, and closes off lots of ideas about what the internet could be, and those ideas are not just "SEO is harder and people can't use curl anymore."
I don't want to condemn experimentation, either, and I'm as excited as anyone to see what JS can do when it's really unleashed. But framing this debate as an argument between crotchety graybeards and The Daring Future Of The Internet misses a lot of the subtleties involved.
Also, how do you reassemble the hash-bang URL from HTTP Referrer header?
In this context hash-bang urls are broken.
Also, with that wrapper or patch, curl & wget will still not be remotely HTML5 ready, which I hope demonstrates that HTML5 is not a requirement in any way. A single HTML5-non-ready browser that can't handle this doesn't mean therefore that HTML5 is a requirement.
I hate to invoke a slippery slope, but it seems a frightening proposition that $entity can start putting out arbitrary standards and suddenly the entire Internet infrastructure has to follow suit in order to be compatible. It's happened before, e.g. favicon.ico. All of them are noble ideas (personalize bookmarks and site feel, allow Ajax content to be accessible) with troublesome implementation (force thousands of redundant GET /favicon.ico requests instead of using something like <meta>, force existing infrastructure to make changes if they want to continue operations as usual.)
All of this is moot, of course, if you just write your pages to fall back sensibly instead of doing what Gawker did and allowing no backwards-compatible text-only fallback. Have JS rewrite your links from "foo/bar" to "#!foo/bar" and then non-compliant user agents and compliant browsers are happy.
As a specific issue, that seems like a minus, but an exceedingly minor one, as lynx is probably a negligible proportion of Gawker's audience. In principle, backwards-compatibility is a great thing, until it impedes some kind of desirable change, such as doing something new or doing it more economically.
> it seems a frightening proposition that $entity can start putting out arbitrary standards
I generally do want someone putting out new standards, and sometimes it's worth breaking backwards-compatibility to an extent. So it really depends on $entity: if it's WHATWG, great. If it's Google, then more caution is warranted. But there's been plenty of cases of innovations (e.g. canvas) starting with a specific player and going mainstream from there. I do agree that Google's approach feels like an ugly hack in a way that is reminiscent of favicon.ico.
> All of this is moot, of course...
This is good general advice, but it's not always true. At least one webapp I've worked on has many important ajax-loads triggered by non-anchor elements; it's about as useful in lynx as Google maps would be. The devs could go through and convert as much as possible to gracefully-degrading anchors, that would at least partly help with noscript, but it seems like a really bad use of resources, given the goals of that app.
So perhaps the solution is for every #! page to have a meta tag pointing to the canonical API resource which it is drawing data from. Bingo, semantic web!
To me, it seems that Google recommends indicating ajax content in the path in the same way that our government issues concealed weapon permits. Yes it okay to have concealed content that can loads on the fly as long as you are very clear of your intentions. Once again this is a usability issue that wouldn't be an issue if it weren't for spammers.
I also don't see why these web pages (specifically twitter) shouldn't be an ajax application - I think loading a page statically, and then the data via ajax is a Good Thing™.
Actually, you are right with regards to the bang pash of hashbang - that’s just a Google defined way of letting them access the content.
Let's say I'm writing a web based word processor, and a user clicks on a document. I want the URL to be a reference to that specific document. The only way to change the URL to be specific without requiring a whole-page refresh is to use the hashbang syntax.
<a rel="nofollow" href="/#!about/" title="Click here to go to About">About</a>
without the hashbang shit, that's a nice normal link that any normal http client can work with. you can still layer on ajax to bind to that link, and use the fragment for the benefit of browser state and user interaction should you think that is a wise thing.
The hash without the bang. It's only been done for about 10 years. You can put whatever you want after the hash. It's up to your application to decide the meaning of it.
For an example, see the OSX SDK documentations: http://developer.apple.com/library/mac/#documentation/Cocoa/...
Sorry, your other choice was #1.