I haven't ever broken macOS doing stuff like that, and I've been doing it for years, with both Macports and Homebrew.

I use the default prefix and make sure the Homebrew stuff is in my $PATH before the built-in stuff and it works fine.