> I think literally the only stuff on my phone that isn't backed up is local files

And besides downloads, there are many apps that just use local files and that you therefore need to transfert manually. e.g. apps from F-Droid, SMS, Wechat, etc. So even though "many" stuffs are automatically stored in Google servers, you still need a lot of case-by-case manual attention and care when transferring to a new phone.