Very bad idea, and one that will cause a lot of breakage. There’s a reason every package management system on Mac uses its own prefix.
I use the default prefix and make sure the Homebrew stuff is in my $PATH before the built-in stuff and it works fine.