We use nextjs because we want to optimize the page for SEO and crawlability. If you don’t need your page to be crawlable (eg a dashboard behind a login), then there’s a good chance you don’t need nextjs, and pure React will indeed be simpler.
as an end user, the only downside would be the typical SPA ones, such as the client needs to render the site with JS like you mentioned (except in cases where we use pre-rendering).
here's an example site built with Routerino if you'd like to experience what it is like in usage: https://virtourlimited.netlify.app/
But page load optimization does typically bring better user engagement so it’s not just an SEO strategy.