You raise some good points. Within the same website, loading a different server-generated full page will typically load many of the same assets like CSS, and header images, which will remain cached by the client for subsequent loads.
It's true that some markup framing the content is duplicated; incidentally in the olden days we used <frame> tags to separate a site-global navigation from the page-specific content, but that went out of style -- it had URL addressability drawbacks and no convention ever developed to fix it.
But ultimately the client needs to render the View, while the server (at some point presumably) originates the Model. Somehow you have to turn the M into the V, and there have been hundreds of different takes over the last couple decades on how exactly to accomplish this. Classically, every HTML page was the complete view, but then the DOM API was introduced to allow this view to be mutated imperatively after the View has been rendered.
The main innovation of client-side scripting was Ajax, which was explicitly about lazy- and/or continuous loading of small snippets of innercontent, like a single Tweet or a couple Google Maps tiles. This emboldened some to push more and more state onto the client, and now we have entire frameworks that do MVC or MVVM in (client-side) Javascript. If we had a way besides the DOM API to morph one state of the View to another; say, a declarative HTML-diff, or, something like XSLT or JSON-patch, I'm sure people would take it. If this sounds a lot like Virtual DOM, it's because they solve the same problem.