You'd be better off with passage[0] -- it's a fork of pass but uses age for encryption. You can just make a directory hierarchy with appropriate .age-recipients files at the right levels.

[0]: https://github.com/FiloSottile/passage