Yes you're right, but keep in mind that with SPA, only the first loading is slower, then you don't need to reload again when changing pages.
That won't always work out or be necessary though.. For an app like Lucidchart or something, I can deal with DL'ing updates every now and then. I spend most of the time in the running app.
Beyond that, unless your users are a captive audience (have to use your software for their job) you should probably be optimizing for a positive first impression.