Since .js, .css and assets are immutable between releases (and modern tooling like NextJS appends hashes to the filenames so they can be served with 'Cache-Control: immutable'), the app is always served from browser cache until there is a new release - which is usually weeks, not days. And if a the browser cache should be empty, you would compare waiting 500ms-1s to use an app that you will use for hours that day. If however, every link click, every route change, every interaction triggers a SSR server roundtrip, the app will not feel snappy during usage.
Now, if people chose the wrong tool for the job and use a 1MB SPA to serve a landing page, that is where things go wrong. But for me, metrics that include the download time of the .js/.css assets are pointless, as they occur once - relative to total time of app usage. After initial load, the snappyness of your SPA will mostly depend on your database queries and API performance, which is also the case in a SSR solution. YMMV of course.