I run into this with Dito GAM[1] when handling groups/calendar access. I workaround by just making it try three times. It always succeeds on the second try.
Having run into similar issue to this in the past, this is most likely caused by caching of group data by the backend servers, possibly coupled with multiple servers answering queries.