This is a valid argument, and you do need to benchmark on underpowered hardware. My gut feeling that a 1 MB blob of JS is not a huge problem even for a cheap smartphone nowadays, and if it is, the HTML/CSS can become the bigger problem.

I remember doing really silly-sounding things (like shipping a web app + the entire database) with excellent results. Even if the initial load takes a while, you can't beat the instant responsiveness of already having the data when the user clicks on another piece of content. Hundreds of KB of JSON on a 2013 smartphone (probably Nexus 4 or 5) was still a pretty good experience IIRC, at least on par with a modern web site on a modern smartphone. On a PC, I've mercilessly thrown hundreds of MB of binary data at JavaScript.

In my experience, aside from long network request chains, the biggest performance killer is e.g. having nested Angular elements that all refresh every time you touch anything on the page, not code size. If you know what you're doing and care (e.g. diligently mark immutable stuff as such), you get excellent performance. It's just that most don't care.