It becomes a problem when Google decides to keep those security fixes to themselves. I can very much see microG (a GMS reimplementation) provide the same service using their own compiled patches, kind of like Linux distributions provide packages and kernel updates developed by Intel, Microsoft, Red Hat, etc, which end up publicly available.