One option that you sort-of mentioned but missed: go with the static groups, but don’t let the users feel that.

In other words, show the kick/invite options to users when it does happen, but destroy and create a new group behind the scenes.