The problem is that whenever you want to do a quick tweak, you have to do a `home-manager switch`, which can take a good 15 seconds or so. It's not the end of the world, but it's a very annoying bit of friction. It's particularly painful if you're trying to quickly test out changes that you're not confident of and iterate.
This is compounded by the fact that home-manager currently doesn't work properly with the new flake based profiles (https://github.com/nix-community/home-manager/issues/2848).