The biggest difference I've found is which apps you can use from the Google Play Store. Calyx supports microG, which allows you to access the Play store via front-ends like Aurora. It also works with F-Droid, as mentioned below.

I also notice things like push notifications work just fine with Calyx, and not on Graphene, which I think is due to microG and/or Firebase.

I think there is some additional hardening that Graphene does as well.

If your goal is the most possible privacy and security, I'd go with Graphene. For a much-improved privacy, mostly non-Google experience, where you can still use Maps or your bank's app, I'd go with Calyx.