I think you would never feel this with Pivotal tracker, Gmail or other well designed apps. Its easy to cache rendered htnk elements in js and show them when back button is pressed and then update them in the background (in fact this is what we do).

If the initial page load is a problem, you can use a CDN to distribute the javascript files. I am not sure about the scroll problem though. Thanks for pointing it out