I ended up creating a homebrew user and giving my personal user sudo access to the homebrew user, which has seemed to work well so far (I may have had to give group read/execute access to all of /usr/local though)