What I have seen is that, depending on the number of pages and frequency that each page is accessed, it may be necessary to pre-cache some or all of the server-side rendered pages to get good performance.
If your site is in any way dynamic, you won't be able to cache much anyway
Many of these page renders require calls out to third-party APIs, which is responsible for some of the slowness. There are definitely optimizations that can be performed (e.g. caching the results of the backend APIs). But it's easier for less experienced developers to just cache the whole page with a reasonable TTL.
Also agreed that you won't be able to cache much (or as much) if the user is logged in. Logged in pages matter less for SEO, so we haven't focused too much on it.