Same. Long ago, I finally got fed up with MacPorts when it insisted on building an entire Perl distribution as a dependency of a package I asked it to install. I found Homebrew, and I was very happy to learn that it offered a way install the same package with an optional flag that disabled the Perl dependency. It reminded me of Portage, the powerful package manager from Gentoo, so I migrated from MacPorts to Homebrew. I happily ran it out of ~/usr/brew for years, which allowed me to trivially clone my entire installation and experiment with potentially destructive operations like major database upgrades.
As Homebrew became more popular, it got more opinionated. The system that supports optional flags is effectively (if not officially) deprecated, and running it outside of /usr/local is unsupported. I recently tried to install a fresh Homebrew in a different custom location (/opt/brew). Many packages require building from source in this scenario, but I accepted that trade-off. I even begrudgingly accepting building an entire Python distribution as a dependency, because I really wanted everything to live under /opt/brew. But no matter what I tried, I could not get any of the modern postgresql packages to build. I burned several multi-hour sessions trying different versions of XCode CLT and alternate build flags. I really wanted to inquire how the maintainers actually manage to build the binary packages in the first place, but my impression from reading Github issues and mailing list posts is that I'd basically be told I was running an unsupported configuration, and that my only recourse would be to create my own formula and run it from a tap. But that finally was a bridge too far (been there done that), so I just gave up and let it make its mess in /usr/local.
I do think I understand why Homebrew has evolved this way. They are trying to support a huge userbase, and they're also subject to Apple's whims with regards to their development tools[1]. It's a massive project every year when new versions of macOS and XCode are released. They have to set limits to keep their sanity. It's just a bummer that I'm considering migrating again...
[1]: https://github.com/Homebrew/brew/pull/4334#issuecomment-4029...
EDIT: remove unnecessary bits; add forgotten link