That said, user experience (in this case site speed) can be a factor in how Google actually ranks pages, so that could be another motivation.
Knowing a bit about SEO, if you want to invest in your ranking, I wouldn't go without server-side rendering.
Just pre-render the page in headless chromium / puppeteer when a search engine bot requests it and return the resulting html.
I would certainly consider this ahead of SSR driven by "isomorphic" Javascript for frontend-heavy apps.
On the other hand, puppeteer executes your application in virtually the same environment as the user's browser, without polluting your frontend architecture with SSR concepts and limitations.
Yes, you'll need to set up a bit of infra initially. Depending on the use case it may very well pay off.
Example in my profile.