The way I've been doing this so far is to let admin be the group of /usr/local/ and then let them have write perms within that directory. For what I do, and how much security I need on my personal machine, it works out pretty well.
I suspect that there are security implications regarding this when done on more of a server-style environment. Perhaps that is what you were saying 'Ouch!' about?
Or maybe you were thinking of another pain point that isn't occurring to me right now?