The libraries will be cached but it means that most of the logic is processed and rendered from JavaScript code and not loaded from the server.
If Google wanted to start over they could pre-render these pages on their massive servers and you could then download these pages without any JavaScript at all. But rewriting is often not the way and really this is a trade off. Do you want to see the content fast or do you want the interactions to be fast when it is ready? If you pre-render everything you either have to rely on new request to get new views or you still have to download all of that JavaScript to do view changes.
They are speeding up JavaScript with Chrome, they are making faster protocols with SPDY and they are doing work in the back ground to make their servers faster. If they didn't do anything to the Gmail code base it would still become faster over time because of these other improvements.
Really the approach that they took is the same one that most Flash projects took because they found that interaction was more important than download time. I would assume they have tested this approach and likely most real users where not bothered by the second or two of progress bar.
20 emails in the inbox view (API endpoint 1), directory information (API endpoint 2), Google Chat (API endpoint 3), your most-frequented labels (API endpoint 3) - and especially the email-endpoints are quite traffic-intensive!
So the GMail team decided they'll load with a progress bar instead of creating four or five widgets with "loading..." placeholders (unlike I do at readme.fm). When you got a lot of various and/or traffic-intensive assets, then you have to choose one of those two models.
(Of course, you can take advantage of IDB/WebSQL/LocalStorage/AppCache, but that is only helping a little bit)