When available I prefer installing applications via brew as casks, since at least this way if I decide to uninstall it later brew will take care of deleting all of these associated directories.
I remember using an app called AppZapper several years ago which did this but for arbitrary applications. No idea if it’s still around/maintained.